2024-09-16T10:11:46.423 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T10:11:46.439 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T10:11:46.525 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906637 branch: reef description: rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906637' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 seed: 1390 sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 sleep_before_teardown: 0 subset: 5319/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGOb/8SHFRhuz7J1nnA4Tb0Fdtz6uS1aFHJfxDr+hrFJl66qiiYjx49w2h2nqVXH6n9Of3aVMZ9p3WEOfID9keU= tasks: - install: null - ceph: log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - rep_lost_unfound_delete: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T10:11:46.525 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa; will attempt to use it 2024-09-16T10:11:46.657 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa/tasks 2024-09-16T10:11:46.657 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T10:11:46.659 INFO:teuthology.task.internal:Checking packages... 2024-09-16T10:11:46.680 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8' 2024-09-16T10:11:46.680 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T10:11:46.680 INFO:teuthology.packaging:ref: None 2024-09-16T10:11:46.680 INFO:teuthology.packaging:tag: None 2024-09-16T10:11:46.680 INFO:teuthology.packaging:branch: reef 2024-09-16T10:11:46.680 INFO:teuthology.packaging:sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:11:46.680 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-16T10:11:46.935 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-853.gae7f3733 2024-09-16T10:11:46.937 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T10:11:46.944 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T10:11:46.944 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T10:11:46.962 INFO:teuthology.task.internal:Saving configuration 2024-09-16T10:11:46.974 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T10:11:46.981 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T10:11:47.006 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906637', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 10:07:07.042644', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGOb/8SHFRhuz7J1nnA4Tb0Fdtz6uS1aFHJfxDr+hrFJl66qiiYjx49w2h2nqVXH6n9Of3aVMZ9p3WEOfID9keU='} 2024-09-16T10:11:47.006 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T10:11:47.014 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T10:11:47.014 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T10:11:47.045 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f62a4e0fac0>, signals=[15]) 2024-09-16T10:11:47.045 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T10:11:47.051 INFO:teuthology.task.internal:Opening connections... 2024-09-16T10:11:47.052 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-09-16T10:11:47.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:11:47.128 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T10:11:47.134 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-09-16T10:11:47.207 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-09-16T10:11:47.207 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-09-16T10:11:47.264 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="CentOS Stream" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="9" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:ID="centos" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE="rhel fedora" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="9" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:PLATFORM_ID="platform:el9" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:ANSI_COLOR="0;31" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:LOGO="fedora-logo-icon" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://centos.org/" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T10:11:47.265 INFO:teuthology.orchestra.run.smithi050.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T10:11:47.266 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-09-16T10:11:47.285 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T10:11:47.293 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T10:11:47.298 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T10:11:47.298 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-09-16T10:11:47.322 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T10:11:47.327 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T10:11:47.327 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-09-16T10:11:47.382 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T10:11:47.414 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T10:11:47.428 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T10:11:47.428 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T10:11:47.429 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T10:11:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:11:47.446 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-09-16T10:11:47.502 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-503.el9.x86_64 2024-09-16T10:11:47.502 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.14.0-503.el9.x86_64 2024-09-16T10:11:47.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum install -y kernel 2024-09-16T10:11:51.067 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-09-16T10:11:53.118 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-16T10:11:57.877 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.5 MB 00:00 2024-09-16T10:11:59.353 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 19 kB 00:00 2024-09-16T10:12:00.115 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-16T10:12:05.696 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T10:12:07.828 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T10:12:07.828 INFO:teuthology.orchestra.run.smithi050.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T10:12:07.920 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:12:07.928 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:12:07.929 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2024-09-16T10:12:07.930 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:12:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 79 M 2024-09-16T10:12:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 126 M 2024-09-16T10:12:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:12:08.139 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 238 kB/s | 23 kB 00:00 2024-09-16T10:12:10.431 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 6.6 MB/s | 16 MB 00:02 2024-09-16T10:12:12.573 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 7.6 MB/s | 35 MB 00:04 2024-09-16T10:12:12.939 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 5.9 MB/s | 29 MB 00:04 2024-09-16T10:12:12.940 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:12:12.940 INFO:teuthology.orchestra.run.smithi050.stdout:Total 16 MB/s | 79 MB 00:05 2024-09-16T10:12:13.653 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:12:14.035 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:12:14.035 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:12:15.036 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:12:15.037 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:12:17.729 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:12:18.437 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-16T10:12:18.518 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T10:12:21.309 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T10:12:21.446 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T10:12:26.565 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T10:12:26.606 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:12:31.753 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:13:10.635 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:13:10.676 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:13:11.136 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:13:11.136 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-16T10:13:11.136 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-16T10:13:11.136 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-16T10:13:11.944 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-16T10:13:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:13:11.945 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:13:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-16T10:13:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-16T10:13:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-16T10:13:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-16T10:13:11.945 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:13:11.945 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:13:12.270 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T10:13:12.356 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T10:13:12.356 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T10:13:12.356 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-16T10:13:12.356 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T10:13:12.357 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T10:13:12.357 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T10:13:12.357 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T10:13:12.357 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T10:13:12.416 INFO:teuthology.orchestra.run.smithi050.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-16T10:13:12.416 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-16T10:13:12.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo rpm -qi grub2-tools 2024-09-16T10:13:12.449 INFO:teuthology.orchestra.run.smithi050.stdout:Name : grub2-tools 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Epoch : 1 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Version : 2.06 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Release : 82.el9 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: x86_64 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Group : Unspecified 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Size : 8274670 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:License : GPLv3+ 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Packager : builder@centos.org 2024-09-16T10:13:12.450 INFO:teuthology.orchestra.run.smithi050.stdout:Vendor : CentOS 2024-09-16T10:13:12.451 INFO:teuthology.orchestra.run.smithi050.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T10:13:12.451 INFO:teuthology.orchestra.run.smithi050.stdout:Summary : Support tools for GRUB. 2024-09-16T10:13:12.451 INFO:teuthology.orchestra.run.smithi050.stdout:Description : 2024-09-16T10:13:12.451 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:13:12.451 INFO:teuthology.orchestra.run.smithi050.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T10:13:12.451 INFO:teuthology.orchestra.run.smithi050.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T10:13:12.451 INFO:teuthology.orchestra.run.smithi050.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T10:13:12.451 INFO:teuthology.orchestra.run.smithi050.stdout:hardware devices. 2024-09-16T10:13:12.451 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:13:12.451 INFO:teuthology.orchestra.run.smithi050.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T10:13:12.452 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T10:13:12.452 INFO:teuthology.task.kernel:Updating grub on smithi050 to boot 5.14.0-505.el9.x86_64 2024-09-16T10:13:12.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T10:13:13.072 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2024-09-16T10:13:14.464 INFO:teuthology.orchestra.run.smithi050.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T10:13:14.494 INFO:teuthology.orchestra.run.smithi050.stderr:done 2024-09-16T10:13:14.496 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-09-16T10:13:14.511 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.HNxGCzMOzH 2024-09-16T10:13:14.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HNxGCzMOzH 2024-09-16T10:13:14.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.HNxGCzMOzH 2024-09-16T10:13:14.719 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.HNxGCzMOzH is 6KB 2024-09-16T10:13:14.768 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.HNxGCzMOzH 2024-09-16T10:13:14.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T10:13:14.853 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T10:13:14.853 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T10:13:14.853 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T10:13:14.853 INFO:teuthology.orchestra.run.smithi050.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-16T10:13:14.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-16T10:13:15.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2024-09-16T10:13:45.049 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T10:13:45.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-16T10:13:45.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:14:03.581 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-09-16T10:14:12.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-16T10:14:12.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:15:12.654 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T10:15:24.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-16T10:15:24.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:15:24.980 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-09-16T10:15:25.383 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-16T10:15:25.383 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-16T10:15:25.384 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-09-16T10:15:25.400 INFO:teuthology.orchestra.run.smithi050.stdout:5.14.0-505.el9.x86_64 2024-09-16T10:15:25.401 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-16T10:15:25.401 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T10:15:25.401 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T10:15:26.402 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T10:15:26.402 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T10:15:26.468 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2024-09-16T10:15:26.491 DEBUG:teuthology.parallel:result is None 2024-09-16T10:15:26.492 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T10:15:26.502 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T10:15:26.503 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T10:15:26.521 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T10:15:26.527 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T10:15:26.537 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T10:15:26.537 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T10:15:26.589 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T10:15:26.595 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T10:15:26.596 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T10:15:26.662 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:15:26.674 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:15:26.676 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T10:15:26.683 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T10:15:26.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T10:15:26.745 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T10:15:26.758 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T10:15:26.759 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T10:15:26.802 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:15:26.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:15:26.933 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:15:27.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:15:27.059 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:15:27.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T10:15:27.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-09-16T10:15:27.212 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T10:15:27.584 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T10:15:27.593 INFO:teuthology.task.internal:Starting timer... 2024-09-16T10:15:27.593 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T10:15:27.606 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T10:15:27.615 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-16T10:15:27.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo service auditd rotate 2024-09-16T10:15:27.687 INFO:teuthology.orchestra.run.smithi050.stdout:Rotating logs: 2024-09-16T10:15:27.689 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T10:15:27.689 DEBUG:teuthology.orchestra.run.smithi050:> /usr/sbin/getenforce 2024-09-16T10:15:27.727 INFO:teuthology.orchestra.run.smithi050.stdout:Permissive 2024-09-16T10:15:27.727 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi050.front.sepia.ceph.com': 'permissive'} 2024-09-16T10:15:27.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T10:15:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:15:27.794 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T10:15:27.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo /usr/sbin/setenforce permissive 2024-09-16T10:15:27.874 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T10:15:27.887 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T10:15:27.899 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T10:15:27.900 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T10:19:18.397 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com')] 2024-09-16T10:19:18.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-16T10:19:18.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:19:18.481 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-09-16T10:19:18.570 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-09-16T10:19:18.570 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T10:19:18.581 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T10:19:18.581 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T10:19:18.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:19:18.641 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T10:19:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T10:19:18.684 INFO:teuthology.orchestra.run.smithi050.stderr:sudo: ntpd: command not found 2024-09-16T10:19:18.694 INFO:teuthology.orchestra.run.smithi050.stdout:506 Cannot talk to daemon 2024-09-16T10:19:18.709 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T10:19:18.724 INFO:teuthology.orchestra.run.smithi050.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T10:19:18.790 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-09-16T10:19:18.793 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T10:19:18.794 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-09-16T10:19:18.794 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:19:18.794 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:19:18.794 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:19:18.794 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:19:18.795 INFO:teuthology.run_tasks:Running task install... 2024-09-16T10:19:18.803 DEBUG:teuthology.task.install:project ceph 2024-09-16T10:19:18.803 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}} 2024-09-16T10:19:18.803 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'} 2024-09-16T10:19:18.803 INFO:teuthology.task.install:Using flavor: default 2024-09-16T10:19:18.854 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T10:19:18.854 INFO:teuthology.task.install:extra packages: [] 2024-09-16T10:19:18.855 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8', 'tag': None, 'wait_for_package': False} 2024-09-16T10:19:18.855 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:19:19.002 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/ 2024-09-16T10:19:19.002 INFO:teuthology.task.install.rpm:Package version is 18.2.4-845.ge4e227b3 2024-09-16T10:19:19.119 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T10:19:19.119 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:19:19.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T10:19:19.148 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-16T10:19:19.149 DEBUG:teuthology.orchestra.run.smithi050:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T10:19:19.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T10:19:19.291 DEBUG:teuthology.orchestra.run.smithi050:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T10:19:19.356 INFO:teuthology.orchestra.run.smithi050.stdout:check_obsoletes = 1 2024-09-16T10:19:19.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum clean all 2024-09-16T10:19:19.596 INFO:teuthology.orchestra.run.smithi050.stdout:45 files removed 2024-09-16T10:19:19.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-radosgw 2024-09-16T10:19:20.184 INFO:teuthology.orchestra.run.smithi050.stdout:ceph packages for x86_64 295 kB/s | 77 kB 00:00 2024-09-16T10:19:20.468 INFO:teuthology.orchestra.run.smithi050.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-16T10:19:20.744 INFO:teuthology.orchestra.run.smithi050.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-16T10:19:21.210 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.2 MB 00:00 2024-09-16T10:19:23.178 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-16T10:19:31.672 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.5 MB 00:04 2024-09-16T10:19:33.263 INFO:teuthology.orchestra.run.smithi050.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 19 kB 00:00 2024-09-16T10:19:34.709 INFO:teuthology.orchestra.run.smithi050.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-16T10:19:40.398 INFO:teuthology.orchestra.run.smithi050.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-16T10:19:42.852 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:19:42.869 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 7.7 M 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrading: 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: librados2 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 3.3 M 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 3.0 M 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 5.1 M 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 18 M 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 25 k 2024-09-16T10:19:42.870 INFO:teuthology.orchestra.run.smithi050.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 712 k 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 478 k 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2 x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 4.5 M 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T10:19:42.871 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 45 k 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 129 k 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 162 k 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 321 k 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 300 k 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 99 k 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:19:42.872 INFO:teuthology.orchestra.run.smithi050.stdout:Install 28 Packages 2024-09-16T10:19:42.873 INFO:teuthology.orchestra.run.smithi050.stdout:Upgrade 2 Packages 2024-09-16T10:19:42.873 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:19:42.873 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 53 M 2024-09-16T10:19:42.874 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:19:43.549 INFO:teuthology.orchestra.run.smithi050.stdout:(1/30): ceph-base-18.2.4-845.ge4e227b3.el9.x86_ 11 MB/s | 5.1 MB 00:00 2024-09-16T10:19:43.600 INFO:teuthology.orchestra.run.smithi050.stdout:(2/30): ceph-selinux-18.2.4-845.ge4e227b3.el9.x 504 kB/s | 25 kB 00:00 2024-09-16T10:19:43.716 INFO:teuthology.orchestra.run.smithi050.stdout:(3/30): libcephfs2-18.2.4-845.ge4e227b3.el9.x86 6.0 MB/s | 712 kB 00:00 2024-09-16T10:19:44.234 INFO:teuthology.orchestra.run.smithi050.stdout:(4/30): ceph-common-18.2.4-845.ge4e227b3.el9.x8 16 MB/s | 18 MB 00:01 2024-09-16T10:19:44.267 INFO:teuthology.orchestra.run.smithi050.stdout:(5/30): libradosstriper1-18.2.4-845.ge4e227b3.e 869 kB/s | 478 kB 00:00 2024-09-16T10:19:44.426 INFO:teuthology.orchestra.run.smithi050.stdout:(6/30): librgw2-18.2.4-845.ge4e227b3.el9.x86_64 23 MB/s | 4.5 MB 00:00 2024-09-16T10:19:44.451 INFO:teuthology.orchestra.run.smithi050.stdout:(7/30): python3-ceph-argparse-18.2.4-845.ge4e22 245 kB/s | 45 kB 00:00 2024-09-16T10:19:44.484 INFO:teuthology.orchestra.run.smithi050.stdout:(8/30): python3-ceph-common-18.2.4-845.ge4e227b 2.2 MB/s | 129 kB 00:00 2024-09-16T10:19:44.610 INFO:teuthology.orchestra.run.smithi050.stdout:(9/30): ceph-radosgw-18.2.4-845.ge4e227b3.el9.x 5.1 MB/s | 7.7 MB 00:01 2024-09-16T10:19:44.644 INFO:teuthology.orchestra.run.smithi050.stdout:(10/30): python3-cephfs-18.2.4-845.ge4e227b3.el 841 kB/s | 162 kB 00:00 2024-09-16T10:19:44.669 INFO:teuthology.orchestra.run.smithi050.stdout:(11/30): python3-rados-18.2.4-845.ge4e227b3.el9 1.7 MB/s | 321 kB 00:00 2024-09-16T10:19:44.694 INFO:teuthology.orchestra.run.smithi050.stdout:(12/30): python3-rbd-18.2.4-845.ge4e227b3.el9.x 3.5 MB/s | 300 kB 00:00 2024-09-16T10:19:44.736 INFO:teuthology.orchestra.run.smithi050.stdout:(13/30): python3-rgw-18.2.4-845.ge4e227b3.el9.x 1.1 MB/s | 99 kB 00:00 2024-09-16T10:19:44.845 INFO:teuthology.orchestra.run.smithi050.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 418 kB/s | 45 kB 00:00 2024-09-16T10:19:44.870 INFO:teuthology.orchestra.run.smithi050.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 532 kB/s | 107 kB 00:00 2024-09-16T10:19:44.895 INFO:teuthology.orchestra.run.smithi050.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 799 kB/s | 160 kB 00:00 2024-09-16T10:19:44.937 INFO:teuthology.orchestra.run.smithi050.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-16T10:19:44.962 INFO:teuthology.orchestra.run.smithi050.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-16T10:19:45.013 INFO:teuthology.orchestra.run.smithi050.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-09-16T10:19:45.046 INFO:teuthology.orchestra.run.smithi050.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 1.0 MB/s | 111 kB 00:00 2024-09-16T10:19:45.071 INFO:teuthology.orchestra.run.smithi050.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-16T10:19:45.097 INFO:teuthology.orchestra.run.smithi050.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-16T10:19:45.122 INFO:teuthology.orchestra.run.smithi050.stdout:(23/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-16T10:19:45.147 INFO:teuthology.orchestra.run.smithi050.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T10:19:45.181 INFO:teuthology.orchestra.run.smithi050.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-16T10:19:45.223 INFO:teuthology.orchestra.run.smithi050.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-16T10:19:45.281 INFO:teuthology.orchestra.run.smithi050.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-16T10:19:45.390 INFO:teuthology.orchestra.run.smithi050.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-16T10:19:45.490 INFO:teuthology.orchestra.run.smithi050.stdout:(29/30): librados2-18.2.4-845.ge4e227b3.el9.x86 12 MB/s | 3.3 MB 00:00 2024-09-16T10:19:45.574 INFO:teuthology.orchestra.run.smithi050.stdout:(30/30): librbd1-18.2.4-845.ge4e227b3.el9.x86_6 10 MB/s | 3.0 MB 00:00 2024-09-16T10:19:45.579 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:19:45.579 INFO:teuthology.orchestra.run.smithi050.stdout:Total 20 MB/s | 53 MB 00:02 2024-09-16T10:19:45.656 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:19:45.696 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:19:45.696 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:19:46.319 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:19:46.320 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:19:47.747 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:19:47.832 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-16T10:19:47.862 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-16T10:19:47.913 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-16T10:19:48.302 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-16T10:19:48.361 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 5/32 2024-09-16T10:19:48.455 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 5/32 2024-09-16T10:19:48.540 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 6/32 2024-09-16T10:19:48.571 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 7/32 2024-09-16T10:19:48.634 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 7/32 2024-09-16T10:19:48.671 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-16T10:19:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-argparse-2:18.2.4-845.ge4e227b3.el9.x 9/32 2024-09-16T10:19:48.782 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 10/32 2024-09-16T10:19:48.806 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 11/32 2024-09-16T10:19:48.872 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 11/32 2024-09-16T10:19:48.983 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-16T10:19:49.014 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-16T10:19:49.055 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-16T10:19:49.096 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-16T10:19:49.123 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-16T10:19:49.159 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-16T10:19:49.202 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-16T10:19:49.515 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-16T10:19:49.563 INFO:teuthology.orchestra.run.smithi050.stdout: Upgrading : librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 20/32 2024-09-16T10:19:49.654 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 20/32 2024-09-16T10:19:49.684 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 21/32 2024-09-16T10:19:49.720 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-16T10:19:50.148 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-16T10:19:50.173 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 24/32 2024-09-16T10:19:50.245 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 24/32 2024-09-16T10:19:50.324 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 25/32 2024-09-16T10:19:50.359 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ceph-common-2:18.2.4-845.ge4e227b3.el9.x86 26/32 2024-09-16T10:19:52.512 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 27/32 2024-09-16T10:19:52.547 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 27/32 2024-09-16T10:19:53.125 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 27/32 2024-09-16T10:19:53.159 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 28/32 2024-09-16T10:19:53.262 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 28/32 2024-09-16T10:19:53.262 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T10:19:53.262 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T10:19:53.262 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:19:53.302 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 29/32 2024-09-16T10:20:03.879 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 29/32 2024-09-16T10:20:03.879 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /sys 2024-09-16T10:20:03.879 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /proc 2024-09-16T10:20:03.879 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /mnt 2024-09-16T10:20:03.879 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /var/tmp 2024-09-16T10:20:03.879 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /home 2024-09-16T10:20:03.879 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /root 2024-09-16T10:20:03.880 INFO:teuthology.orchestra.run.smithi050.stdout:skipping the directory /tmp 2024-09-16T10:20:03.880 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:04.678 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 30/32 2024-09-16T10:20:04.707 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 30/32 2024-09-16T10:20:04.707 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:20:04.707 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T10:20:04.707 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:20:04.708 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:20:04.708 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:04.709 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-16T10:20:04.781 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-16T10:20:04.781 INFO:teuthology.orchestra.run.smithi050.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-16T10:20:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-16T10:20:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 1/32 2024-09-16T10:20:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 2/32 2024-09-16T10:20:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 3/32 2024-09-16T10:20:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 4/32 2024-09-16T10:20:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 5/32 2024-09-16T10:20:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 6/32 2024-09-16T10:20:05.868 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 7/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-argparse-2:18.2.4-845.ge4e227b3.el9.x 8/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ceph-common-2:18.2.4-845.ge4e227b3.el9.x86 9/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 10/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 11/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 12/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 13/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-16T10:20:05.869 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 29/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-16T10:20:05.870 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 31/32 2024-09-16T10:20:06.874 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-16T10:20:06.874 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:06.874 INFO:teuthology.orchestra.run.smithi050.stdout:Upgraded: 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-radosgw-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-selinux-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T10:20:06.875 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: libradosstriper1-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: librgw2-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.876 INFO:teuthology.orchestra.run.smithi050.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T10:20:06.877 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.877 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.877 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:06.877 INFO:teuthology.orchestra.run.smithi050.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T10:20:06.877 INFO:teuthology.orchestra.run.smithi050.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T10:20:06.877 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:06.877 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:20:07.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-test 2024-09-16T10:20:07.792 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:27 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:20:07.929 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 40 M 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 166 k 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:20:07.930 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:07.931 INFO:teuthology.orchestra.run.smithi050.stdout:Install 4 Packages 2024-09-16T10:20:07.931 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:07.931 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 40 M 2024-09-16T10:20:07.931 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 164 M 2024-09-16T10:20:07.931 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:20:08.285 INFO:teuthology.orchestra.run.smithi050.stdout:(1/4): libcephsqlite-18.2.4-845.ge4e227b3.el9.x 658 kB/s | 166 kB 00:00 2024-09-16T10:20:08.635 INFO:teuthology.orchestra.run.smithi050.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 182 kB/s | 64 kB 00:00 2024-09-16T10:20:08.661 INFO:teuthology.orchestra.run.smithi050.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 484 kB/s | 304 kB 00:00 2024-09-16T10:20:09.878 INFO:teuthology.orchestra.run.smithi050.stdout:(4/4): ceph-test-18.2.4-845.ge4e227b3.el9.x86_6 21 MB/s | 40 MB 00:01 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:20:09.880 INFO:teuthology.orchestra.run.smithi050.stdout:Total 21 MB/s | 40 MB 00:01 2024-09-16T10:20:09.909 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:20:09.930 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:20:09.930 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:20:10.272 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:20:10.273 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:20:10.899 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:20:10.973 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T10:20:11.020 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T10:20:11.046 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 3/4 2024-09-16T10:20:14.964 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 3/4 2024-09-16T10:20:15.047 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 4/4 2024-09-16T10:20:17.007 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 4/4 2024-09-16T10:20:17.008 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 1/4 2024-09-16T10:20:17.008 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 2/4 2024-09-16T10:20:17.008 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T10:20:17.330 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T10:20:17.330 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:17.330 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:20:17.330 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-test-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:17.330 INFO:teuthology.orchestra.run.smithi050.stdout: libcephsqlite-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:17.330 INFO:teuthology.orchestra.run.smithi050.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T10:20:17.330 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T10:20:17.330 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:17.330 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:20:17.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph 2024-09-16T10:20:18.229 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:38 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:20:18.380 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:20:18.382 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:18.382 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-16T10:20:18.382 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:18.382 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: ceph x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 6.5 k 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 2.1 M 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 1.5 M 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 246 k 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 4.7 M 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 17 M 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T10:20:18.383 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T10:20:18.384 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:18.385 INFO:teuthology.orchestra.run.smithi050.stdout:Install 33 Packages 2024-09-16T10:20:18.386 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:18.386 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 29 M 2024-09-16T10:20:18.386 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 104 M 2024-09-16T10:20:18.386 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:20:18.711 INFO:teuthology.orchestra.run.smithi050.stdout:(1/33): ceph-18.2.4-845.ge4e227b3.el9.x86_64.rp 44 kB/s | 6.5 kB 00:00 2024-09-16T10:20:19.362 INFO:teuthology.orchestra.run.smithi050.stdout:(2/33): ceph-mds-18.2.4-845.ge4e227b3.el9.x86_6 2.6 MB/s | 2.1 MB 00:00 2024-09-16T10:20:19.420 INFO:teuthology.orchestra.run.smithi050.stdout:(3/33): ceph-mgr-18.2.4-845.ge4e227b3.el9.x86_6 1.7 MB/s | 1.5 MB 00:00 2024-09-16T10:20:19.504 INFO:teuthology.orchestra.run.smithi050.stdout:(4/33): ceph-mgr-modules-core-18.2.4-845.ge4e22 2.9 MB/s | 246 kB 00:00 2024-09-16T10:20:19.621 INFO:teuthology.orchestra.run.smithi050.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-16T10:20:19.705 INFO:teuthology.orchestra.run.smithi050.stdout:(6/33): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-09-16T10:20:19.747 INFO:teuthology.orchestra.run.smithi050.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T10:20:19.788 INFO:teuthology.orchestra.run.smithi050.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-16T10:20:19.830 INFO:teuthology.orchestra.run.smithi050.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-16T10:20:19.872 INFO:teuthology.orchestra.run.smithi050.stdout:(10/33): python3-urllib3-1.26.5-6.el9.noarch.rp 5.0 MB/s | 215 kB 00:00 2024-09-16T10:20:19.947 INFO:teuthology.orchestra.run.smithi050.stdout:(11/33): ceph-mon-18.2.4-845.ge4e227b3.el9.x86_ 3.8 MB/s | 4.7 MB 00:01 2024-09-16T10:20:19.973 INFO:teuthology.orchestra.run.smithi050.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-09-16T10:20:20.006 INFO:teuthology.orchestra.run.smithi050.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.5 MB/s | 51 kB 00:00 2024-09-16T10:20:20.023 INFO:teuthology.orchestra.run.smithi050.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 460 kB/s | 35 kB 00:00 2024-09-16T10:20:20.040 INFO:teuthology.orchestra.run.smithi050.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-09-16T10:20:20.074 INFO:teuthology.orchestra.run.smithi050.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 864 kB/s | 43 kB 00:00 2024-09-16T10:20:20.216 INFO:teuthology.orchestra.run.smithi050.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2024-09-16T10:20:20.233 INFO:teuthology.orchestra.run.smithi050.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 645 kB/s | 11 kB 00:00 2024-09-16T10:20:20.258 INFO:teuthology.orchestra.run.smithi050.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 789 kB/s | 172 kB 00:00 2024-09-16T10:20:20.283 INFO:teuthology.orchestra.run.smithi050.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-16T10:20:20.309 INFO:teuthology.orchestra.run.smithi050.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-09-16T10:20:20.334 INFO:teuthology.orchestra.run.smithi050.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-16T10:20:20.359 INFO:teuthology.orchestra.run.smithi050.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-16T10:20:20.385 INFO:teuthology.orchestra.run.smithi050.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 918 kB/s | 46 kB 00:00 2024-09-16T10:20:20.410 INFO:teuthology.orchestra.run.smithi050.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T10:20:20.435 INFO:teuthology.orchestra.run.smithi050.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-09-16T10:20:20.461 INFO:teuthology.orchestra.run.smithi050.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-16T10:20:20.486 INFO:teuthology.orchestra.run.smithi050.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-16T10:20:20.511 INFO:teuthology.orchestra.run.smithi050.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-16T10:20:20.537 INFO:teuthology.orchestra.run.smithi050.stdout:(30/33): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-16T10:20:20.562 INFO:teuthology.orchestra.run.smithi050.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-09-16T10:20:20.604 INFO:teuthology.orchestra.run.smithi050.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-16T10:20:20.812 INFO:teuthology.orchestra.run.smithi050.stdout:(33/33): ceph-osd-18.2.4-845.ge4e227b3.el9.x86_ 11 MB/s | 17 MB 00:01 2024-09-16T10:20:20.817 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:20:20.817 INFO:teuthology.orchestra.run.smithi050.stdout:Total 12 MB/s | 29 MB 00:02 2024-09-16T10:20:20.938 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:20:20.979 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:20:20.979 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:20:21.427 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:20:21.428 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:20:22.670 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:20:22.732 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-16T10:20:22.765 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-16T10:20:22.789 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-16T10:20:22.886 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-16T10:20:22.962 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-16T10:20:23.009 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-16T10:20:23.048 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-16T10:20:23.077 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-16T10:20:23.103 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-16T10:20:23.136 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-16T10:20:23.186 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-16T10:20:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-16T10:20:23.263 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-16T10:20:23.301 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-16T10:20:23.379 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-16T10:20:23.530 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-16T10:20:23.621 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-16T10:20:23.680 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-16T10:20:23.719 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-16T10:20:23.777 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-16T10:20:23.840 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-16T10:20:24.037 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-16T10:20:24.105 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-16T10:20:24.186 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-16T10:20:24.286 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-16T10:20:24.329 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-16T10:20:24.454 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-16T10:20:24.619 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-845.ge4e227b3.el9.n 28/33 2024-09-16T10:20:24.647 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 29/33 2024-09-16T10:20:24.674 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 29/33 2024-09-16T10:20:24.674 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:20:24.674 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T10:20:24.674 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T10:20:24.674 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T10:20:24.674 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:26.185 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 30/33 2024-09-16T10:20:26.215 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 30/33 2024-09-16T10:20:26.215 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:20:26.215 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T10:20:26.216 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T10:20:26.216 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T10:20:26.216 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:26.677 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 31/33 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 31/33 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T10:20:26.703 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T10:20:26.704 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:26.912 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 32/33 2024-09-16T10:20:26.938 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 32/33 2024-09-16T10:20:26.938 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:20:26.938 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T10:20:26.938 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T10:20:26.938 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T10:20:26.938 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:27.003 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 33/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 33/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 1/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 2/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 3/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 4/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 5/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-845.ge4e227b3.el9.n 6/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-16T10:20:28.651 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-16T10:20:28.652 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-16T10:20:28.653 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-16T10:20:28.653 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-16T10:20:28.653 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-16T10:20:28.653 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-16T10:20:28.653 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-16T10:20:28.653 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-16T10:20:28.653 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-16T10:20:29.681 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-16T10:20:29.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:29.681 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:20:29.681 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:29.681 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mds-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mon-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-osd-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T10:20:29.682 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T10:20:29.683 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T10:20:29.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T10:20:29.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T10:20:29.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T10:20:29.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T10:20:29.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T10:20:29.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T10:20:29.684 INFO:teuthology.orchestra.run.smithi050.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T10:20:29.684 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:29.684 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:20:30.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-base 2024-09-16T10:20:30.649 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:50 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:20:30.743 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-base-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:20:30.789 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:20:30.790 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-16T10:20:30.790 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:20:30.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install cephadm 2024-09-16T10:20:31.441 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:51 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:20:31.579 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:20:31.580 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:31.580 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-16T10:20:31.580 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:31.580 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:20:31.581 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 226 k 2024-09-16T10:20:31.581 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:31.581 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:20:31.581 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:31.581 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-16T10:20:31.581 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:31.581 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 226 k 2024-09-16T10:20:31.581 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 217 k 2024-09-16T10:20:31.581 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:20:31.818 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm-18.2.4-845.ge4e227b3.el9.noarch.rpm 953 kB/s | 226 kB 00:00 2024-09-16T10:20:31.819 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:20:31.819 INFO:teuthology.orchestra.run.smithi050.stdout:Total 946 kB/s | 226 kB 00:00 2024-09-16T10:20:31.819 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:20:31.825 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:20:31.825 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:20:31.828 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:20:31.828 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:20:31.899 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:20:32.325 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:20:32.416 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:20:32.787 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:20:33.045 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/1 2024-09-16T10:20:33.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:33.045 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:20:33.045 INFO:teuthology.orchestra.run.smithi050.stdout: cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:20:33.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:33.046 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:20:33.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T10:20:33.838 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:20:33.975 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:20:33.976 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:33.976 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-16T10:20:33.976 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:33.977 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:20:33.977 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 143 k 2024-09-16T10:20:33.977 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:33.977 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:20:33.977 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:33.977 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-16T10:20:33.978 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:33.978 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 143 k 2024-09-16T10:20:33.978 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 395 k 2024-09-16T10:20:33.978 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:20:34.208 INFO:teuthology.orchestra.run.smithi050.stdout:ceph-immutable-object-cache-18.2.4-845.ge4e227b 617 kB/s | 143 kB 00:00 2024-09-16T10:20:34.208 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:20:34.209 INFO:teuthology.orchestra.run.smithi050.stdout:Total 612 kB/s | 143 kB 00:00 2024-09-16T10:20:34.209 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:20:34.217 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:20:34.217 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:20:34.278 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:20:34.279 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:20:34.438 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:20:34.517 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.e 1/1 2024-09-16T10:20:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.e 1/1 2024-09-16T10:20:34.544 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:20:34.544 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T10:20:34.544 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:35.477 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.e 1/1 2024-09-16T10:20:35.477 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:35.477 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:20:35.477 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-immutable-object-cache-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:20:35.478 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:35.478 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:20:35.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr 2024-09-16T10:20:36.292 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:20:36.387 INFO:teuthology.orchestra.run.smithi050.stdout:Package ceph-mgr-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:20:36.432 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:20:36.433 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-16T10:20:36.433 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:20:36.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T10:20:37.079 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:00:57 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:20:37.217 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:20:37.217 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-16T10:20:37.218 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:37.218 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:20:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 3.5 M 2024-09-16T10:20:37.218 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-16T10:20:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 24 k 2024-09-16T10:20:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 16 k 2024-09-16T10:20:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T10:20:37.218 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T10:20:37.219 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:37.219 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:20:37.219 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:37.219 INFO:teuthology.orchestra.run.smithi050.stdout:Install 5 Packages 2024-09-16T10:20:37.219 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:37.219 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.8 M 2024-09-16T10:20:37.219 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 87 M 2024-09-16T10:20:37.220 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:20:37.595 INFO:teuthology.orchestra.run.smithi050.stdout:(1/5): ceph-grafana-dashboards-18.2.4-845.ge4e2 155 kB/s | 24 kB 00:00 2024-09-16T10:20:37.620 INFO:teuthology.orchestra.run.smithi050.stdout:(2/5): ceph-prometheus-alerts-18.2.4-845.ge4e22 90 kB/s | 16 kB 00:00 2024-09-16T10:20:37.662 INFO:teuthology.orchestra.run.smithi050.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 462 kB/s | 31 kB 00:00 2024-09-16T10:20:37.687 INFO:teuthology.orchestra.run.smithi050.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-16T10:20:37.904 INFO:teuthology.orchestra.run.smithi050.stdout:(5/5): ceph-mgr-dashboard-18.2.4-845.ge4e227b3. 7.6 MB/s | 3.5 MB 00:00 2024-09-16T10:20:37.908 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:20:37.908 INFO:teuthology.orchestra.run.smithi050.stdout:Total 5.5 MB/s | 3.8 MB 00:00 2024-09-16T10:20:37.908 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:20:37.915 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:20:37.915 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:20:38.006 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:20:38.006 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:20:38.267 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:20:38.379 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-16T10:20:38.409 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-16T10:20:38.438 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-845.ge4e227b3.el9.no 3/5 2024-09-16T10:20:39.552 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-845.ge4e227b3.el9.n 4/5 2024-09-16T10:20:39.591 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 5/5 2024-09-16T10:20:40.678 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 5/5 2024-09-16T10:20:40.678 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-845.ge4e227b3.el9.n 1/5 2024-09-16T10:20:40.678 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 2/5 2024-09-16T10:20:40.678 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-845.ge4e227b3.el9.no 3/5 2024-09-16T10:20:40.678 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-16T10:20:40.945 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-16T10:20:40.945 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:40.945 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:20:40.945 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-grafana-dashboards-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:20:40.945 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:20:40.946 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-prometheus-alerts-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:20:40.946 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T10:20:40.946 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T10:20:40.946 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:40.946 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:20:41.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T10:20:41.732 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:01 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:20:41.871 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:20:41.872 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================== 2024-09-16T10:20:41.872 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-16T10:20:41.872 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================== 2024-09-16T10:20:41.872 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:20:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 7.4 M 2024-09-16T10:20:41.873 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-16T10:20:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T10:20:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T10:20:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T10:20:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T10:20:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T10:20:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T10:20:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T10:20:41.873 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T10:20:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T10:20:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T10:20:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T10:20:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:41.874 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:20:41.874 INFO:teuthology.orchestra.run.smithi050.stdout:====================================================================================== 2024-09-16T10:20:41.874 INFO:teuthology.orchestra.run.smithi050.stdout:Install 12 Packages 2024-09-16T10:20:41.874 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:41.875 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 38 M 2024-09-16T10:20:41.875 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 212 M 2024-09-16T10:20:41.875 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:20:42.394 INFO:teuthology.orchestra.run.smithi050.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-16T10:20:42.436 INFO:teuthology.orchestra.run.smithi050.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.4 MB/s | 801 kB 00:00 2024-09-16T10:20:42.520 INFO:teuthology.orchestra.run.smithi050.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-09-16T10:20:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-16T10:20:42.654 INFO:teuthology.orchestra.run.smithi050.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 461 kB/s | 38 kB 00:00 2024-09-16T10:20:42.863 INFO:teuthology.orchestra.run.smithi050.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.4-84 12 MB/s | 7.4 MB 00:00 2024-09-16T10:20:43.063 INFO:teuthology.orchestra.run.smithi050.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-16T10:20:43.580 INFO:teuthology.orchestra.run.smithi050.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-16T10:20:43.722 INFO:teuthology.orchestra.run.smithi050.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 250 kB 00:00 2024-09-16T10:20:44.630 INFO:teuthology.orchestra.run.smithi050.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.5 MB/s | 4.9 MB 00:01 2024-09-16T10:20:44.814 INFO:teuthology.orchestra.run.smithi050.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.9 MB/s | 5.1 MB 00:01 2024-09-16T10:20:49.989 INFO:teuthology.orchestra.run.smithi050.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.6 MB/s | 16 MB 00:06 2024-09-16T10:20:49.989 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:20:49.990 INFO:teuthology.orchestra.run.smithi050.stdout:Total 4.7 MB/s | 38 MB 00:08 2024-09-16T10:20:50.362 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:20:50.390 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:20:50.390 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:20:50.891 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:20:50.892 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:20:51.684 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:20:51.759 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T10:20:51.796 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T10:20:51.910 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T10:20:51.955 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T10:20:52.140 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T10:20:52.170 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T10:20:52.250 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T10:20:52.964 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T10:20:53.096 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T10:20:54.433 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T10:20:55.509 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T10:20:55.536 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227 12/12 2024-09-16T10:20:58.116 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227 12/12 2024-09-16T10:20:58.116 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227 1/12 2024-09-16T10:20:58.116 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T10:20:58.116 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T10:20:58.117 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T10:20:58.117 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T10:20:58.117 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T10:20:58.117 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T10:20:58.117 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T10:20:58.117 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T10:20:58.117 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T10:20:58.117 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T10:20:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T10:20:58.570 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-diskprediction-local-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T10:20:58.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T10:20:58.572 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:58.572 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:20:58.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-rook 2024-09-16T10:20:59.578 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:19 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:20:59.718 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:20:59.719 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:59.719 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-16T10:20:59.719 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:59.719 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:20:59.719 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 49 k 2024-09-16T10:20:59.719 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-16T10:20:59.719 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:59.720 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:20:59.721 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:20:59.721 INFO:teuthology.orchestra.run.smithi050.stdout:Install 13 Packages 2024-09-16T10:20:59.721 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:20:59.721 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 2.2 M 2024-09-16T10:20:59.721 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 26 M 2024-09-16T10:20:59.721 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:21:00.045 INFO:teuthology.orchestra.run.smithi050.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 244 kB/s | 19 kB 00:00 2024-09-16T10:21:00.071 INFO:teuthology.orchestra.run.smithi050.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 247 kB/s | 26 kB 00:00 2024-09-16T10:21:00.138 INFO:teuthology.orchestra.run.smithi050.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-09-16T10:21:00.163 INFO:teuthology.orchestra.run.smithi050.stdout:(4/13): ceph-mgr-rook-18.2.4-845.ge4e227b3.el9. 252 kB/s | 49 kB 00:00 2024-09-16T10:21:00.205 INFO:teuthology.orchestra.run.smithi050.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-16T10:21:00.230 INFO:teuthology.orchestra.run.smithi050.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.0 MB/s | 279 kB 00:00 2024-09-16T10:21:00.264 INFO:teuthology.orchestra.run.smithi050.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 548 kB/s | 32 kB 00:00 2024-09-16T10:21:00.281 INFO:teuthology.orchestra.run.smithi050.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 456 kB/s | 54 kB 00:00 2024-09-16T10:21:00.298 INFO:teuthology.orchestra.run.smithi050.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 210 kB/s | 14 kB 00:00 2024-09-16T10:21:00.323 INFO:teuthology.orchestra.run.smithi050.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 3.2 MB/s | 59 kB 00:00 2024-09-16T10:21:00.348 INFO:teuthology.orchestra.run.smithi050.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-16T10:21:00.373 INFO:teuthology.orchestra.run.smithi050.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-16T10:21:00.409 INFO:teuthology.orchestra.run.smithi050.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-09-16T10:21:00.410 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:21:00.410 INFO:teuthology.orchestra.run.smithi050.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-16T10:21:00.476 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:21:00.493 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:21:00.493 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:21:00.589 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:21:00.591 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:21:01.116 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:21:01.296 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T10:21:01.351 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T10:21:01.407 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T10:21:01.443 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T10:21:01.485 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T10:21:01.591 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T10:21:01.709 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T10:21:01.762 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T10:21:02.521 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T10:21:02.574 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T10:21:02.609 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T10:21:02.658 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T10:21:02.686 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 13/13 2024-09-16T10:21:03.505 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 13/13 2024-09-16T10:21:03.505 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 1/13 2024-09-16T10:21:03.505 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T10:21:03.505 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T10:21:03.505 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T10:21:03.505 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T10:21:03.505 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T10:21:03.506 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T10:21:03.506 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T10:21:03.506 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T10:21:03.506 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T10:21:03.506 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T10:21:03.506 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T10:21:03.850 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-rook-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T10:21:03.851 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T10:21:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T10:21:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T10:21:03.852 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:03.852 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:04.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T10:21:04.695 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:24 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:04.834 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:04.835 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:04.835 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-16T10:21:04.835 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:04.835 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:21:04.835 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 139 k 2024-09-16T10:21:04.835 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-16T10:21:04.835 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T10:21:04.835 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T10:21:04.835 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T10:21:04.835 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T10:21:04.836 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T10:21:04.836 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:04.836 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:21:04.836 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:04.836 INFO:teuthology.orchestra.run.smithi050.stdout:Install 6 Packages 2024-09-16T10:21:04.836 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:04.836 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 7.0 M 2024-09-16T10:21:04.836 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 33 M 2024-09-16T10:21:04.837 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:21:05.709 INFO:teuthology.orchestra.run.smithi050.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-09-16T10:21:05.767 INFO:teuthology.orchestra.run.smithi050.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-16T10:21:05.784 INFO:teuthology.orchestra.run.smithi050.stdout:(3/6): ceph-mgr-cephadm-18.2.4-845.ge4e227b3.el 570 kB/s | 139 kB 00:00 2024-09-16T10:21:05.818 INFO:teuthology.orchestra.run.smithi050.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-16T10:21:05.860 INFO:teuthology.orchestra.run.smithi050.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 992 kB/s | 74 kB 00:00 2024-09-16T10:21:05.951 INFO:teuthology.orchestra.run.smithi050.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-16T10:21:05.954 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:21:05.954 INFO:teuthology.orchestra.run.smithi050.stdout:Total 6.3 MB/s | 7.0 MB 00:01 2024-09-16T10:21:06.030 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:21:06.042 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:21:06.042 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:21:06.147 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:21:06.149 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:21:06.464 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:21:06.649 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T10:21:06.697 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T10:21:07.249 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T10:21:07.334 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T10:21:07.404 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T10:21:07.441 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 6/6 2024-09-16T10:21:08.240 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 6/6 2024-09-16T10:21:08.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 1/6 2024-09-16T10:21:08.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T10:21:08.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T10:21:08.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T10:21:08.240 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-cephadm-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:08.516 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:08.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-fuse 2024-09-16T10:21:09.321 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:29 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:09.459 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:09.460 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:09.460 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-16T10:21:09.460 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:09.460 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:21:09.460 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 852 k 2024-09-16T10:21:09.460 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-16T10:21:09.460 INFO:teuthology.orchestra.run.smithi050.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T10:21:09.460 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:09.460 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:21:09.461 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:09.461 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-09-16T10:21:09.461 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:09.461 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 932 k 2024-09-16T10:21:09.461 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 2.7 M 2024-09-16T10:21:09.461 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:21:09.778 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): ceph-fuse-18.2.4-845.ge4e227b3.el9.x86_6 2.8 MB/s | 852 kB 00:00 2024-09-16T10:21:10.120 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 125 kB/s | 80 kB 00:00 2024-09-16T10:21:10.121 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:21:10.121 INFO:teuthology.orchestra.run.smithi050.stdout:Total 1.4 MB/s | 932 kB 00:00 2024-09-16T10:21:10.134 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:21:10.159 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:21:10.159 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:21:10.246 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:21:10.246 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:21:10.456 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:21:10.606 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T10:21:10.671 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 2/2 2024-09-16T10:21:11.381 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 2/2 2024-09-16T10:21:11.381 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/2 2024-09-16T10:21:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T10:21:11.683 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:11.683 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:21:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T10:21:11.684 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:11.684 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:11.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install ceph-volume 2024-09-16T10:21:12.549 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:12.688 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:12.689 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:12.689 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repository Size 2024-09-16T10:21:12.689 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:12.689 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:21:12.689 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume noarch 2:18.2.4-845.ge4e227b3.el9 ceph-noarch 264 k 2024-09-16T10:21:12.689 INFO:teuthology.orchestra.run.smithi050.stdout:Installing dependencies: 2024-09-16T10:21:12.689 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T10:21:12.689 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:12.689 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:21:12.690 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:12.690 INFO:teuthology.orchestra.run.smithi050.stdout:Install 2 Packages 2024-09-16T10:21:12.690 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:12.690 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 342 k 2024-09-16T10:21:12.690 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 1.5 M 2024-09-16T10:21:12.690 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:21:13.041 INFO:teuthology.orchestra.run.smithi050.stdout:(1/2): ceph-volume-18.2.4-845.ge4e227b3.el9.noa 1.2 MB/s | 264 kB 00:00 2024-09-16T10:21:13.074 INFO:teuthology.orchestra.run.smithi050.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 307 kB/s | 77 kB 00:00 2024-09-16T10:21:13.075 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:21:13.075 INFO:teuthology.orchestra.run.smithi050.stdout:Total 886 kB/s | 342 kB 00:00 2024-09-16T10:21:13.087 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:21:13.098 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:21:13.098 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:21:13.137 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:21:13.137 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:21:13.290 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:21:13.485 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-16T10:21:13.519 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 2/2 2024-09-16T10:21:13.535 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 2/2 2024-09-16T10:21:13.535 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:21:13.535 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T10:21:13.535 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:14.237 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 1/2 2024-09-16T10:21:14.557 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-16T10:21:14.557 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:14.557 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:21:14.557 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-volume-2:18.2.4-845.ge4e227b3.el9.noarch 2024-09-16T10:21:14.557 INFO:teuthology.orchestra.run.smithi050.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T10:21:14.557 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:14.557 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:14.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados-devel 2024-09-16T10:21:15.434 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:15.571 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:15.572 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:15.572 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-16T10:21:15.572 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:15.572 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:21:15.572 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 127 k 2024-09-16T10:21:15.572 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:15.573 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:21:15.573 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:15.573 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-16T10:21:15.573 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:15.573 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 127 k 2024-09-16T10:21:15.573 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 456 k 2024-09-16T10:21:15.573 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:21:15.780 INFO:teuthology.orchestra.run.smithi050.stdout:librados-devel-18.2.4-845.ge4e227b3.el9.x86_64. 613 kB/s | 127 kB 00:00 2024-09-16T10:21:15.781 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:21:15.781 INFO:teuthology.orchestra.run.smithi050.stdout:Total 608 kB/s | 127 kB 00:00 2024-09-16T10:21:15.781 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:21:15.787 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:21:15.787 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:21:15.854 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:21:15.854 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:21:16.027 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:21:16.131 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:16.547 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:16.806 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:16.806 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:16.806 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:21:16.806 INFO:teuthology.orchestra.run.smithi050.stdout: librados-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:21:16.806 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:16.806 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:16.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs2 2024-09-16T10:21:17.596 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:37 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:17.691 INFO:teuthology.orchestra.run.smithi050.stdout:Package libcephfs2-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:21:17.737 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:17.738 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-16T10:21:17.738 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:17.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install libcephfs-devel 2024-09-16T10:21:18.385 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:18.523 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:18.524 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-16T10:21:18.524 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:18.524 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:21:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 31 k 2024-09-16T10:21:18.524 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:18.524 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:21:18.524 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:18.525 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-16T10:21:18.525 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:18.525 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 31 k 2024-09-16T10:21:18.525 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 139 k 2024-09-16T10:21:18.525 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:21:18.683 INFO:teuthology.orchestra.run.smithi050.stdout:libcephfs-devel-18.2.4-845.ge4e227b3.el9.x86_64 196 kB/s | 31 kB 00:00 2024-09-16T10:21:18.684 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:21:18.684 INFO:teuthology.orchestra.run.smithi050.stdout:Total 194 kB/s | 31 kB 00:00 2024-09-16T10:21:18.684 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:21:18.688 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:21:18.688 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:21:18.711 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:21:18.711 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:21:18.815 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:21:18.950 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:19.319 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:19.563 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:19.563 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:19.564 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:21:19.564 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-devel-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:21:19.564 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:19.564 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:19.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librados2 2024-09-16T10:21:20.324 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:20.418 INFO:teuthology.orchestra.run.smithi050.stdout:Package librados2-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:21:20.463 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:20.464 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-16T10:21:20.465 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:20.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install librbd1 2024-09-16T10:21:21.116 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:21.212 INFO:teuthology.orchestra.run.smithi050.stdout:Package librbd1-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:21:21.257 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:21.258 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-16T10:21:21.258 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:21.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rados 2024-09-16T10:21:21.912 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:22.006 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rados-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:21:22.052 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:22.053 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-16T10:21:22.053 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:22.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rgw 2024-09-16T10:21:22.708 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:22.800 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rgw-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:21:22.845 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:22.846 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-16T10:21:22.846 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:22.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-cephfs 2024-09-16T10:21:23.503 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:23.597 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-cephfs-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:21:23.643 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:23.644 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-16T10:21:23.644 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:23.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install python3-rbd 2024-09-16T10:21:24.289 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:24.382 INFO:teuthology.orchestra.run.smithi050.stdout:Package python3-rbd-2:18.2.4-845.ge4e227b3.el9.x86_64 is already installed. 2024-09-16T10:21:24.426 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:24.427 INFO:teuthology.orchestra.run.smithi050.stdout:Nothing to do. 2024-09-16T10:21:24.427 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:24.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-fuse 2024-09-16T10:21:25.081 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:25.222 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:25.223 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:25.223 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-16T10:21:25.223 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:25.223 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:21:25.223 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 87 k 2024-09-16T10:21:25.223 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:25.223 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:21:25.223 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:25.224 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-16T10:21:25.224 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:25.224 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 87 k 2024-09-16T10:21:25.224 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 227 k 2024-09-16T10:21:25.224 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:21:25.420 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-fuse-18.2.4-845.ge4e227b3.el9.x86_64.rpm 443 kB/s | 87 kB 00:00 2024-09-16T10:21:25.421 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:21:25.421 INFO:teuthology.orchestra.run.smithi050.stdout:Total 440 kB/s | 87 kB 00:00 2024-09-16T10:21:25.421 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:21:25.428 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:21:25.428 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:21:25.500 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:21:25.500 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:21:25.660 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:21:25.754 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:26.131 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:26.405 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:26.405 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:26.405 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:21:26.405 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-fuse-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:21:26.406 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:26.406 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:26.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-mirror 2024-09-16T10:21:27.259 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:27.403 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:27.403 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout: Package Arch Version Repo Size 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 3.0 M 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 3.0 M 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 12 M 2024-09-16T10:21:27.404 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:21:27.765 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-mirror-18.2.4-845.ge4e227b3.el9.x86_64.rpm 8.3 MB/s | 3.0 MB 00:00 2024-09-16T10:21:27.766 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:21:27.766 INFO:teuthology.orchestra.run.smithi050.stdout:Total 8.3 MB/s | 3.0 MB 00:00 2024-09-16T10:21:27.766 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:21:27.776 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:21:27.776 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:21:27.861 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:21:27.861 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:21:28.304 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:21:28.374 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:28.401 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:28.401 INFO:teuthology.orchestra.run.smithi050.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T10:21:28.401 INFO:teuthology.orchestra.run.smithi050.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T10:21:28.401 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T10:21:28.402 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T10:21:28.402 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:29.522 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:29.522 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:29.522 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:21:29.522 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-mirror-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:21:29.522 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:29.522 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:29.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo yum -y install rbd-nbd 2024-09-16T10:21:30.330 INFO:teuthology.orchestra.run.smithi050.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 10:19:40 AM UTC. 2024-09-16T10:21:30.471 INFO:teuthology.orchestra.run.smithi050.stdout:Dependencies resolved. 2024-09-16T10:21:30.471 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:30.471 INFO:teuthology.orchestra.run.smithi050.stdout: Package Architecture Version Repository Size 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout:Installing: 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd x86_64 2:18.2.4-845.ge4e227b3.el9 ceph 172 k 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction Summary 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout:================================================================================ 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout:Install 1 Package 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout:Total download size: 172 k 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout:Installed size: 494 k 2024-09-16T10:21:30.472 INFO:teuthology.orchestra.run.smithi050.stdout:Downloading Packages: 2024-09-16T10:21:30.694 INFO:teuthology.orchestra.run.smithi050.stdout:rbd-nbd-18.2.4-845.ge4e227b3.el9.x86_64.rpm 779 kB/s | 172 kB 00:00 2024-09-16T10:21:30.694 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-09-16T10:21:30.694 INFO:teuthology.orchestra.run.smithi050.stdout:Total 773 kB/s | 172 kB 00:00 2024-09-16T10:21:30.694 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction check 2024-09-16T10:21:30.703 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction check succeeded. 2024-09-16T10:21:30.703 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction test 2024-09-16T10:21:30.777 INFO:teuthology.orchestra.run.smithi050.stdout:Transaction test succeeded. 2024-09-16T10:21:30.777 INFO:teuthology.orchestra.run.smithi050.stdout:Running transaction 2024-09-16T10:21:30.960 INFO:teuthology.orchestra.run.smithi050.stdout: Preparing : 1/1 2024-09-16T10:21:31.069 INFO:teuthology.orchestra.run.smithi050.stdout: Installing : rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:31.476 INFO:teuthology.orchestra.run.smithi050.stdout: Running scriptlet: rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:31.762 INFO:teuthology.orchestra.run.smithi050.stdout: Verifying : rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 1/1 2024-09-16T10:21:31.762 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:31.762 INFO:teuthology.orchestra.run.smithi050.stdout:Installed: 2024-09-16T10:21:31.762 INFO:teuthology.orchestra.run.smithi050.stdout: rbd-nbd-2:18.2.4-845.ge4e227b3.el9.x86_64 2024-09-16T10:21:31.762 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:31.762 INFO:teuthology.orchestra.run.smithi050.stdout:Complete! 2024-09-16T10:21:31.965 DEBUG:teuthology.parallel:result is None 2024-09-16T10:21:31.966 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:21:32.114 DEBUG:teuthology.orchestra.run.smithi050:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T10:21:32.136 INFO:teuthology.orchestra.run.smithi050.stdout:18.2.4-845.ge4e227b3.el9 2024-09-16T10:21:32.136 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845.ge4e227b3.el9 2024-09-16T10:21:32.136 INFO:teuthology.task.install:The correct ceph version 18.2.4-845.ge4e227b3 is installed. 2024-09-16T10:21:32.138 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T10:21:32.161 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:32.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T10:21:32.202 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T10:21:32.203 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:32.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T10:21:32.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T10:21:32.333 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T10:21:32.334 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:32.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T10:21:32.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T10:21:32.465 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T10:21:32.465 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:32.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T10:21:32.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T10:21:32.597 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T10:21:32.945 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T10:21:32.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /var/log/ceph 2024-09-16T10:21:32.972 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T10:21:32.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T10:21:33.038 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T10:21:33.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T10:21:33.111 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T10:21:33.111 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T10:21:33.111 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906637', 'branch': 'reef', 'description': 'rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7906637', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 931, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2']], 'seed': 1390, 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8', 'sleep_before_teardown': 0, 'subset': '5319/100000', 'suite': 'rados', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6', 'targets': {'smithi050.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGOb/8SHFRhuz7J1nnA4Tb0Fdtz6uS1aFHJfxDr+hrFJl66qiiYjx49w2h2nqVXH6n9Of3aVMZ9p3WEOfID9keU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8', 'cluster': 'ceph'}}, {'rep_lost_unfound_delete': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_22:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T10:21:33.112 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T10:21:33.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T10:21:33.226 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:33.226 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T10:21:33.282 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T10:21:33.283 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-09-16T10:21:33.338 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T10:21:33.338 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:21:33.338 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-16T10:21:33.338 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:21:33.338 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:21:33.338 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-16 10:21:28.676812345 +0000 2024-09-16T10:21:33.338 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-16 10:18:58.248047146 +0000 2024-09-16T10:21:33.339 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-16 10:18:58.248047146 +0000 2024-09-16T10:21:33.339 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-16 10:18:58.248047146 +0000 2024-09-16T10:21:33.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T10:21:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-16T10:21:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-16T10:21:33.405 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.0889e-05 s, 5.6 MB/s 2024-09-16T10:21:33.405 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T10:21:33.463 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-09-16T10:21:33.519 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T10:21:33.519 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:21:33.519 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-16T10:21:33.519 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:21:33.520 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:21:33.520 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-16 10:21:28.677812320 +0000 2024-09-16T10:21:33.520 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-16 10:18:58.575038884 +0000 2024-09-16T10:21:33.520 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-16 10:18:58.575038884 +0000 2024-09-16T10:21:33.520 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-16 10:18:58.575038884 +0000 2024-09-16T10:21:33.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T10:21:33.583 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-16T10:21:33.583 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-16T10:21:33.583 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000101743 s, 5.0 MB/s 2024-09-16T10:21:33.584 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T10:21:33.641 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-09-16T10:21:33.697 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T10:21:33.697 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:21:33.697 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-16T10:21:33.697 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:21:33.697 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:21:33.697 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-16 10:21:28.677812320 +0000 2024-09-16T10:21:33.697 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-16 10:18:58.896030775 +0000 2024-09-16T10:21:33.698 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-16 10:18:58.896030775 +0000 2024-09-16T10:21:33.698 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-16 10:18:58.896030775 +0000 2024-09-16T10:21:33.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T10:21:33.761 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-16T10:21:33.761 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-16T10:21:33.761 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 9.735e-05 s, 5.3 MB/s 2024-09-16T10:21:33.762 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T10:21:33.819 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-09-16T10:21:33.875 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T10:21:33.875 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T10:21:33.875 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-16T10:21:33.875 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T10:21:33.875 INFO:teuthology.orchestra.run.smithi050.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T10:21:33.875 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-09-16 10:21:28.677812320 +0000 2024-09-16T10:21:33.876 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-09-16 10:18:59.215022715 +0000 2024-09-16T10:21:33.876 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-09-16 10:18:59.215022715 +0000 2024-09-16T10:21:33.876 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: 2024-09-16 10:18:59.215022715 +0000 2024-09-16T10:21:33.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T10:21:33.939 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-09-16T10:21:33.939 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-09-16T10:21:33.939 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 8.8726e-05 s, 5.8 MB/s 2024-09-16T10:21:33.940 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T10:21:33.997 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:21:33.997 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T10:21:33.997 INFO:tasks.ceph:Generating config... 2024-09-16T10:21:34.018 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T10:21:34.018 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T10:21:34.018 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-16T10:21:34.019 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-16T10:21:34.019 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T10:21:34.019 INFO:tasks.ceph:[global] ms type = async 2024-09-16T10:21:34.019 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T10:21:34.019 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T10:21:34.019 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T10:21:34.019 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T10:21:34.019 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T10:21:34.019 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T10:21:34.019 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-16T10:21:34.019 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-16T10:21:34.019 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T10:21:34.019 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T10:21:34.019 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T10:21:34.020 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T10:21:34.021 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T10:21:34.021 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T10:21:34.021 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T10:21:34.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T10:21:34.083 INFO:teuthology.orchestra.run.smithi050.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T10:21:34.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T10:21:34.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T10:21:34.194 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.50'), ('mon.b', '[v2:172.21.15.50:3301,v1:172.21.15.50:6790]'), ('mon.c', '[v2:172.21.15.50:3302,v1:172.21.15.50:6791]')] 2024-09-16T10:21:34.194 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.50,[v2:172.21.15.50:3301,v1:172.21.15.50:6790],[v2:172.21.15.50:3302,v1:172.21.15.50:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-16T10:21:34.195 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:34.195 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T10:21:34.250 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.50 --addv b '[v2:172.21.15.50:3301,v1:172.21.15.50:6790]' --addv c '[v2:172.21.15.50:3302,v1:172.21.15.50:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:21:34.336 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:21:34.336 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: generated fsid 3f765c65-edd5-4e6d-930a-33ce25cc666f 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = pacific 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 0 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 3f765c65-edd5-4e6d-930a-33ce25cc666f 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:last_changed 2024-09-16T10:21:34.335898+0000 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-09-16T10:21:34.335898+0000 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:min_mon_release 16 (pacific) 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:election_strategy: 1 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:1: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.b 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:2: [v2:172.21.15.50:3302/0,v1:172.21.15.50:6791/0] mon.c 2024-09-16T10:21:34.337 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T10:21:34.339 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T10:21:34.354 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3f765c65-edd5-4e6d-930a-33ce25cc666f... 2024-09-16T10:21:34.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T10:21:34.439 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2024-09-16T10:21:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: chdir = "" 2024-09-16T10:21:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T10:21:34.439 INFO:teuthology.orchestra.run.smithi050.stdout: auth supported = cephx 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: filestore xattr use omap = true 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: mon clock drift allowed = 1.000 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: osd crush chooseleaf type = 0 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: auth debug = true 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on old message = true 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: ms die on bug = true 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.440 INFO:teuthology.orchestra.run.smithi050.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: mon pg warn max object skew = 0 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default size = 2 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow primary affinity = true 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd allow pg remap = true 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on legacy crush tunables = false 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on crush straw calc version zero = false 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on no sortbitwise = false 2024-09-16T10:21:34.441 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on osd down out interval zero = false 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: mon warn on too few osds = false 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: mon_allow_pool_size_one = true 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: osd default data pool replay window = 5 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: mon allow pool delete = true 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: mon cluster log file level = debug 2024-09-16T10:21:34.442 INFO:teuthology.orchestra.run.smithi050.stdout: debug asserts on shutdown = true 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: mon health detail to clog = false 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: mon host = "172.21.15.50,[v2:172.21.15.50:3301,v1:172.21.15.50:6790],[v2:172.21.15.50:3302,v1:172.21.15.50:6791]" 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: mon client directed command retry = 5 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: mon election default strategy = 1 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: ms bind msgr1 = False 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: ms bind msgr2 = True 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: ms inject socket failures = 5000 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: ms type = async 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = 3f765c65-edd5-4e6d-930a-33ce25cc666f 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout:[osd] 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: osd journal size = 100 2024-09-16T10:21:34.443 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub load threshold = 5.0 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: osd scrub max interval = 600 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock profile = high_recovery_ops 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: osd recover clone overlap = true 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: osd recovery max chunk = 1048576 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug shutdown = true 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug op order = true 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify stray on activate = true 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: osd open classes on start = true 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug pg log writeout = true 2024-09-16T10:21:34.444 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: osd deep scrub update digest min age = 30 2024-09-16T10:21:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: osd map max advance = 10 2024-09-16T10:21:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: journal zero on create = true 2024-09-16T10:21:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: filestore ondisk finisher threads = 3 2024-09-16T10:21:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: filestore apply finisher threads = 3 2024-09-16T10:21:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.445 INFO:teuthology.orchestra.run.smithi050.stdout: bdev debug aio = true 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug misdirected ops = true 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: bluefs allocator = hybrid 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore allocator = hybrid 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore block size = 96636764160 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore fsck on mount = True 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: bluestore zero block detection = True 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluefs = 20 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: debug bluestore = 20 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: debug osd = 20 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: debug rocksdb = 10 2024-09-16T10:21:34.446 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd full ratio = 0.9 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd nearfull ratio = 0.8 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify cached snaps = True 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: osd debug verify missing on start = True 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: osd failsafe full ratio = 0.95 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: osd mclock override recovery settings = True 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: osd objectstore = bluestore 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue = debug_random 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: osd op queue cut off = debug_random 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr] 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-09-16T10:21:34.447 INFO:teuthology.orchestra.run.smithi050.stdout: debug mgr = 20 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: mgr/telemetry/nag = false 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout:[mon] 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: debug ms = 1 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: debug mon = 20 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: debug paxos = 20 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: debug auth = 20 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: mon data avail warn = 5 2024-09-16T10:21:34.448 INFO:teuthology.orchestra.run.smithi050.stdout: mon mgr mkfs grace = 240 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min pgs per osd = 4 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd reporter subtree level = osd 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: mon osd prime pg temp = true 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: mon reweight min bytes per osd = 10 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T10:21:34.449 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: # 1m isn't quite enough 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: mon_down_mkfs_grace = 2m 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: mon_warn_on_filestore_osds = false 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: mon scrub interval = 300 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout:[client] 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: rgw cache enabled = true 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable ops log = true 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: rgw enable usage log = true 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.a] 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.b] 2024-09-16T10:21:34.450 INFO:teuthology.orchestra.run.smithi050.stdout:[mon.c] 2024-09-16T10:21:34.451 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T10:21:34.452 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T10:21:34.546 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T10:21:34.546 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:34.546 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T10:21:34.562 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:34.562 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T10:21:34.618 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi050.front.sepia.ceph.com 2024-09-16T10:21:34.618 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:34.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T10:21:34.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T10:21:34.693 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:34.694 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:21:34.749 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T10:21:34.749 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T10:21:34.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T10:21:34.841 INFO:teuthology.orchestra.run.smithi050.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T10:21:34.843 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T10:21:34.844 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T10:21:34.844 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T10:21:34.844 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi050.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T10:21:34.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T10:21:34.869 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:21:34.870 INFO:tasks.ceph:role: osd.0 2024-09-16T10:21:34.870 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-16T10:21:34.870 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T10:21:34.935 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:21:34.935 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:21:34.935 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:21:34.935 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T10:21:34.935 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:21:34.936 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:21:34.936 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:21:34.936 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T10:21:34.936 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:21:34.936 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:21:34.969 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-09-16T10:21:34.971 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-09-16T10:21:34.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T10:21:35.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T10:21:35.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T10:21:35.178 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:21:35.178 INFO:tasks.ceph:role: osd.1 2024-09-16T10:21:35.178 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-16T10:21:35.179 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T10:21:35.244 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:21:35.245 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:21:35.245 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:21:35.245 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T10:21:35.245 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:21:35.245 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:21:35.245 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:21:35.246 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T10:21:35.246 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:21:35.246 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:21:35.276 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-09-16T10:21:35.279 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-09-16T10:21:35.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T10:21:35.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T10:21:35.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T10:21:35.485 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T10:21:35.486 INFO:tasks.ceph:role: osd.2 2024-09-16T10:21:35.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-16T10:21:35.486 DEBUG:teuthology.orchestra.run.smithi050:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T10:21:35.550 INFO:teuthology.orchestra.run.smithi050.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T10:21:35.550 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T10:21:35.550 INFO:teuthology.orchestra.run.smithi050.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T10:21:35.550 INFO:teuthology.orchestra.run.smithi050.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T10:21:35.550 INFO:teuthology.orchestra.run.smithi050.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T10:21:35.550 INFO:teuthology.orchestra.run.smithi050.stdout: = sunit=0 swidth=0 blks 2024-09-16T10:21:35.550 INFO:teuthology.orchestra.run.smithi050.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T10:21:35.550 INFO:teuthology.orchestra.run.smithi050.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T10:21:35.551 INFO:teuthology.orchestra.run.smithi050.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T10:21:35.551 INFO:teuthology.orchestra.run.smithi050.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T10:21:35.586 INFO:teuthology.orchestra.run.smithi050.stdout:Discarding blocks...Done. 2024-09-16T10:21:35.588 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi050.front.sepia.ceph.com -o noatime 2024-09-16T10:21:35.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T10:21:35.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T10:21:35.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:21:35.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:35.835+0000 7fdb4262c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T10:21:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:35.835+0000 7fdb4262c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T10:21:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:35.835+0000 7fdb4262c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T10:21:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:35.835+0000 7fdb4262c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T10:21:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:35.835+0000 7fdb4262c640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-16T10:21:35.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:35.836+0000 7fdb4262c640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T10:21:38.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T10:21:38.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:21:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:38.814+0000 7ff32e9b2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T10:21:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:38.814+0000 7ff32e9b2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T10:21:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:38.814+0000 7ff32e9b2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T10:21:38.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:38.814+0000 7ff32e9b2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T10:21:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:38.814+0000 7ff32e9b2640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-16T10:21:38.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:38.815+0000 7ff32e9b2640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T10:21:41.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T10:21:41.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:21:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:41.789+0000 7f962fb1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T10:21:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:41.790+0000 7f962fb1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T10:21:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:41.790+0000 7f962fb1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T10:21:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:41.790+0000 7f962fb1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T10:21:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:41.790+0000 7f962fb1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-16T10:21:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:21:41.790+0000 7f962fb1e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T10:21:44.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T10:21:44.683 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T10:21:44.683 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:44.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T10:21:44.753 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:44.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T10:21:44.821 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:44.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T10:21:44.889 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:44.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T10:21:44.958 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T10:21:44.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T10:21:45.025 INFO:teuthology.orchestra.run.smithi050.stdout:[mgr.x] 2024-09-16T10:21:45.026 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCuBuhmHw8ZMhAAwhrsl9C8/wEajRSwt/+hfQ== 2024-09-16T10:21:45.026 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.0] 2024-09-16T10:21:45.026 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCvBuhmPZDcMRAAGHppHQZPClIi8CJUUXQWzQ== 2024-09-16T10:21:45.026 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.1] 2024-09-16T10:21:45.026 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQCyBuhm3DeYMBAA8FDTx7NrgR91/sfLzthl0w== 2024-09-16T10:21:45.026 INFO:teuthology.orchestra.run.smithi050.stdout:[osd.2] 2024-09-16T10:21:45.026 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQC1BuhmSWAgLxAALdWgeDpjmS61LATyKiqyMQ== 2024-09-16T10:21:45.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T10:21:45.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:21:45.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:21:45.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T10:21:45.282 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T10:21:45.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T10:21:45.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:21:45.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T10:21:45.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T10:21:45.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:21:45.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T10:21:45.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T10:21:45.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T10:21:45.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T10:21:45.810 DEBUG:teuthology.orchestra.run.smithi050:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T10:21:45.866 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T10:21:45.866 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T10:21:45.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T10:21:45.909 INFO:tasks.ceph.mon.a:Started 2024-09-16T10:21:45.909 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T10:21:45.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T10:21:45.911 INFO:tasks.ceph.mon.b:Started 2024-09-16T10:21:45.911 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T10:21:45.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T10:21:45.913 INFO:tasks.ceph.mon.c:Started 2024-09-16T10:21:45.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph config set mgr mgr_pool false --force 2024-09-16T10:21:52.108 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T10:21:52.109 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T10:21:52.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T10:21:52.111 INFO:tasks.ceph.mgr.x:Started 2024-09-16T10:21:52.111 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T10:21:52.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T10:21:52.291 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to default 2024-09-16T10:21:52.291 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:52.289+0000 7f5f4e523040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T10:21:52.302 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T10:21:52.302 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T10:21:52.302 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:52.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T10:21:52.327 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:52.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T10:21:52.335 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:52.334+0000 7f5f4e523040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T10:21:52.360 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-09-16T10:21:52.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T10:21:52.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new 346f1b3d-875c-427b-9fb3-a906c6ea62f8 0 2024-09-16T10:21:52.489 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:52.485+0000 7f5f4e523040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T10:21:52.540 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:52.539+0000 7f5f4e523040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T10:21:52.648 INFO:teuthology.orchestra.run.smithi050.stdout:0 2024-09-16T10:21:52.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new c991e2ab-7bd4-4499-bd8f-a2f7c5581928 1 2024-09-16T10:21:52.708 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:52.706+0000 7f5f4e523040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T10:21:52.826 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:52.824+0000 7f5f4e523040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T10:21:52.880 INFO:teuthology.orchestra.run.smithi050.stdout:1 2024-09-16T10:21:52.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd new af14e6f0-c6ad-4545-ae6c-69130d4ec765 2 2024-09-16T10:21:53.109 INFO:teuthology.orchestra.run.smithi050.stdout:2 2024-09-16T10:21:53.120 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T10:21:53.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T10:21:53.122 INFO:tasks.ceph.osd.0:Started 2024-09-16T10:21:53.122 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T10:21:53.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T10:21:53.124 INFO:tasks.ceph.osd.1:Started 2024-09-16T10:21:53.124 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T10:21:53.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T10:21:53.126 INFO:tasks.ceph.osd.2:Started 2024-09-16T10:21:53.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:21:53.310 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:21:53.311 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":5,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:21:53.106207+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T10:21:53.106207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:21:53.321 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T10:21:53.322 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T10:21:53.333 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:53.332+0000 7f5f4e523040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T10:21:53.391 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:53.389+0000 7f5f4e523040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T10:21:53.546 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:53.544+0000 7f5f4e523040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T10:21:53.624 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T10:21:53.721 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-16T10:21:53.719+0000 7fe739325640 -1 Falling back to public interface 2024-09-16T10:21:53.722 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:21:53.720+0000 7f919ba0d640 -1 Falling back to public interface 2024-09-16T10:21:53.758 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-09-16T10:21:53.757+0000 7f2d5ccb6640 -1 Falling back to public interface 2024-09-16T10:21:53.784 INFO:teuthology.misc.health.smithi050.stdout: 2024-09-16T10:21:53.784 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":5,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:21:53.106207+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T10:21:53.106207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:21:53.793 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T10:21:54.529 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:54.528+0000 7f5f4e523040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T10:21:54.583 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:54.582+0000 7f5f4e523040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T10:21:54.971 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:54.969+0000 7f5f4e523040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-16T10:21:55.059 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.057+0000 7f5f4e523040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T10:21:55.169 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.168+0000 7f5f4e523040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T10:21:55.218 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.216+0000 7f5f4e523040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T10:21:55.334 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.332+0000 7f5f4e523040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-16T10:21:55.390 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.388+0000 7f5f4e523040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T10:21:55.437 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.435+0000 7f5f4e523040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T10:21:55.552 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.551+0000 7f5f4e523040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T10:21:55.604 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.603+0000 7f5f4e523040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T10:21:55.658 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.657+0000 7f5f4e523040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T10:21:55.729 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.727+0000 7f5f4e523040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T10:21:55.775 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.773+0000 7f5f4e523040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T10:21:55.821 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:55.819+0000 7f5f4e523040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T10:21:56.539 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:56.537+0000 7f5f4e523040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T10:21:57.194 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:21:57.193+0000 7f919ba0d640 -1 osd.0 0 log_to_monitors true 2024-09-16T10:21:57.230 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-16T10:21:57.228+0000 7fe739325640 -1 osd.1 0 log_to_monitors true 2024-09-16T10:21:57.257 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-09-16T10:21:57.256+0000 7f2d5ccb6640 -1 osd.2 0 log_to_monitors true 2024-09-16T10:21:57.551 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:57.549+0000 7f5eeeed8640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T10:21:57.551 INFO:tasks.ceph.mgr.x.smithi050.stderr:2024-09-16T10:21:57.549+0000 7f5eeeed8640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T10:21:58.609 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-16T10:21:58.607+0000 7fe7352b2640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T10:21:58.609 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-09-16T10:21:58.607+0000 7f2d58c43640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T10:21:58.614 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-16T10:21:58.613+0000 7fe7310db640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:21:58.615 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-09-16T10:21:58.613+0000 7f2d54a6c640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:21:59.213 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:21:59.211+0000 7f9197438640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T10:21:59.217 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:21:59.216+0000 7f918f259640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:22:00.096 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T10:22:00.382 INFO:teuthology.misc.health.smithi050.stdout: 2024-09-16T10:22:00.382 INFO:teuthology.misc.health.smithi050.stdout:{"epoch":8,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:21:59.601797+0000","last_up_change":"2024-09-16T10:21:59.601797+0000","last_in_change":"2024-09-16T10:21:53.106207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3990881301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":3990881301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":3990881301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3990881301}]},"public_addr":"172.21.15.50:6804/3990881301","cluster_addr":"172.21.15.50:6805/3990881301","heartbeat_back_addr":"172.21.15.50:6807/3990881301","heartbeat_front_addr":"172.21.15.50:6806/3990881301","state":["exists","up"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":202723672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":202723672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":202723672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":202723672}]},"public_addr":"172.21.15.50:6800/202723672","cluster_addr":"172.21.15.50:6801/202723672","heartbeat_back_addr":"172.21.15.50:6803/202723672","heartbeat_front_addr":"172.21.15.50:6802/202723672","state":["exists","up"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":2584589368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":2584589368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":2584589368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":2584589368}]},"public_addr":"172.21.15.50:6808/2584589368","cluster_addr":"172.21.15.50:6809/2584589368","heartbeat_back_addr":"172.21.15.50:6811/2584589368","heartbeat_front_addr":"172.21.15.50:6810/2584589368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:22:00.393 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T10:22:00.393 INFO:tasks.ceph:Creating RBD pool 2024-09-16T10:22:00.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T10:22:01.616 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'rbd' created 2024-09-16T10:22:01.633 DEBUG:teuthology.orchestra.run.smithi050:> rbd --cluster ceph pool init rbd 2024-09-16T10:22:04.637 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T10:22:04.637 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T10:22:04.637 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T10:22:04.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T10:22:04.638 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T10:22:04.977 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:04.988 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":4,"flags":0,"active_gid":4109,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":1087142618}]},"active_addr":"172.21.15.50:6812/1087142618","active_change":"2024-09-16T10:21:56.540225+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2339119243}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":1905399656}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2667506789}]}]} 2024-09-16T10:22:04.990 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T10:22:04.990 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T10:22:04.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:22:05.270 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:05.270 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":13,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:22:04.620914+0000","last_up_change":"2024-09-16T10:21:59.601797+0000","last_in_change":"2024-09-16T10:21:53.106207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T10:22:00.707491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3990881301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":3990881301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":3990881301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3990881301}]},"public_addr":"172.21.15.50:6804/3990881301","cluster_addr":"172.21.15.50:6805/3990881301","heartbeat_back_addr":"172.21.15.50:6807/3990881301","heartbeat_front_addr":"172.21.15.50:6806/3990881301","state":["exists","up"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":202723672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":202723672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":202723672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":202723672}]},"public_addr":"172.21.15.50:6800/202723672","cluster_addr":"172.21.15.50:6801/202723672","heartbeat_back_addr":"172.21.15.50:6803/202723672","heartbeat_front_addr":"172.21.15.50:6802/202723672","state":["exists","up"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":2584589368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":2584589368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":2584589368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":2584589368}]},"public_addr":"172.21.15.50:6808/2584589368","cluster_addr":"172.21.15.50:6809/2584589368","heartbeat_back_addr":"172.21.15.50:6811/2584589368","heartbeat_front_addr":"172.21.15.50:6810/2584589368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.244822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.213317+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.228331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:22:05.282 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T10:22:05.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:22:05.566 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:05.566 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":13,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:22:04.620914+0000","last_up_change":"2024-09-16T10:21:59.601797+0000","last_in_change":"2024-09-16T10:21:53.106207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T10:22:00.707491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3990881301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":3990881301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":3990881301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3990881301}]},"public_addr":"172.21.15.50:6804/3990881301","cluster_addr":"172.21.15.50:6805/3990881301","heartbeat_back_addr":"172.21.15.50:6807/3990881301","heartbeat_front_addr":"172.21.15.50:6806/3990881301","state":["exists","up"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":202723672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":202723672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":202723672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":202723672}]},"public_addr":"172.21.15.50:6800/202723672","cluster_addr":"172.21.15.50:6801/202723672","heartbeat_back_addr":"172.21.15.50:6803/202723672","heartbeat_front_addr":"172.21.15.50:6802/202723672","state":["exists","up"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":2584589368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":2584589368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":2584589368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":2584589368}]},"public_addr":"172.21.15.50:6808/2584589368","cluster_addr":"172.21.15.50:6809/2584589368","heartbeat_back_addr":"172.21.15.50:6811/2584589368","heartbeat_front_addr":"172.21.15.50:6810/2584589368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.244822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.213317+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.228331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:22:05.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T10:22:05.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T10:22:05.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T10:22:05.720 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2024-09-16T10:22:05.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:22:05.738 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2024-09-16T10:22:05.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:22:05.739 INFO:teuthology.orchestra.run.smithi050.stdout:34359738371 2024-09-16T10:22:05.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:22:06.021 INFO:teuthology.orchestra.run.smithi050.stdout:34359738370 2024-09-16T10:22:06.034 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738370 for osd.1 2024-09-16T10:22:06.086 INFO:teuthology.orchestra.run.smithi050.stdout:34359738370 2024-09-16T10:22:06.094 INFO:teuthology.orchestra.run.smithi050.stdout:34359738369 2024-09-16T10:22:06.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738370 for osd.2 2024-09-16T10:22:06.107 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738369 for osd.0 2024-09-16T10:22:07.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:22:07.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:22:07.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:22:07.350 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2024-09-16T10:22:07.367 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2024-09-16T10:22:07.367 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:07.399 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2024-09-16T10:22:07.413 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2024-09-16T10:22:07.413 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:07.447 INFO:teuthology.orchestra.run.smithi050.stdout:34359738371 2024-09-16T10:22:07.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738371 got 34359738371 for osd.0 2024-09-16T10:22:07.460 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:07.460 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T10:22:07.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:07.783 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:07.783 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:07.795 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-16T10:22:06.546810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":85516,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":85115,"kb_avail":283030004,"statfs":{"total":289910292480,"available":289822724096,"internally_reserved":0,"allocated":352256,"data_stored":80387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":87158099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628454+0000","last_change":"2024-09-16T10:22:04.628454+0000","last_active":"2024-09-16T10:22:04.628454+0000","last_peered":"2024-09-16T10:22:04.628454+0000","last_clean":"2024-09-16T10:22:04.628454+0000","last_became_active":"2024-09-16T10:22:02.623835+0000","last_became_peered":"2024-09-16T10:22:02.623835+0000","last_unstale":"2024-09-16T10:22:04.628454+0000","last_undegraded":"2024-09-16T10:22:04.628454+0000","last_fullsized":"2024-09-16T10:22:04.628454+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:10.601195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.627934+0000","last_change":"2024-09-16T10:22:04.627934+0000","last_active":"2024-09-16T10:22:04.627934+0000","last_peered":"2024-09-16T10:22:04.627934+0000","last_clean":"2024-09-16T10:22:04.627934+0000","last_became_active":"2024-09-16T10:22:02.623453+0000","last_became_peered":"2024-09-16T10:22:02.623453+0000","last_unstale":"2024-09-16T10:22:04.627934+0000","last_undegraded":"2024-09-16T10:22:04.627934+0000","last_fullsized":"2024-09-16T10:22:04.627934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:59.318421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628838+0000","last_change":"2024-09-16T10:22:04.628838+0000","last_active":"2024-09-16T10:22:04.628838+0000","last_peered":"2024-09-16T10:22:04.628838+0000","last_clean":"2024-09-16T10:22:04.628838+0000","last_became_active":"2024-09-16T10:22:02.623394+0000","last_became_peered":"2024-09-16T10:22:02.623394+0000","last_unstale":"2024-09-16T10:22:04.628838+0000","last_undegraded":"2024-09-16T10:22:04.628838+0000","last_fullsized":"2024-09-16T10:22:04.628838+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:06.472472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011940130000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628932+0000","last_change":"2024-09-16T10:22:04.628932+0000","last_active":"2024-09-16T10:22:04.628932+0000","last_peered":"2024-09-16T10:22:04.628932+0000","last_clean":"2024-09-16T10:22:04.628932+0000","last_became_active":"2024-09-16T10:22:02.624545+0000","last_became_peered":"2024-09-16T10:22:02.624545+0000","last_unstale":"2024-09-16T10:22:04.628932+0000","last_undegraded":"2024-09-16T10:22:04.628932+0000","last_fullsized":"2024-09-16T10:22:04.628932+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:02.516293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"11'1","reported_seq":16,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.627907+0000","last_change":"2024-09-16T10:22:04.627907+0000","last_active":"2024-09-16T10:22:04.627907+0000","last_peered":"2024-09-16T10:22:04.627907+0000","last_clean":"2024-09-16T10:22:04.627907+0000","last_became_active":"2024-09-16T10:22:02.624518+0000","last_became_peered":"2024-09-16T10:22:02.624518+0000","last_unstale":"2024-09-16T10:22:04.627907+0000","last_undegraded":"2024-09-16T10:22:04.627907+0000","last_fullsized":"2024-09-16T10:22:04.627907+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:10.458645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"13'2","reported_seq":17,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.631208+0000","last_change":"2024-09-16T10:22:04.629301+0000","last_active":"2024-09-16T10:22:04.631208+0000","last_peered":"2024-09-16T10:22:04.631208+0000","last_clean":"2024-09-16T10:22:04.631208+0000","last_became_active":"2024-09-16T10:22:02.621803+0000","last_became_peered":"2024-09-16T10:22:02.621803+0000","last_unstale":"2024-09-16T10:22:04.631208+0000","last_undegraded":"2024-09-16T10:22:04.631208+0000","last_fullsized":"2024-09-16T10:22:04.631208+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:11.319895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628883+0000","last_change":"2024-09-16T10:22:04.628883+0000","last_active":"2024-09-16T10:22:04.628883+0000","last_peered":"2024-09-16T10:22:04.628883+0000","last_clean":"2024-09-16T10:22:04.628883+0000","last_became_active":"2024-09-16T10:22:02.625120+0000","last_became_peered":"2024-09-16T10:22:02.625120+0000","last_unstale":"2024-09-16T10:22:04.628883+0000","last_undegraded":"2024-09-16T10:22:04.628883+0000","last_fullsized":"2024-09-16T10:22:04.628883+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:48:42.724379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001146357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628522+0000","last_change":"2024-09-16T10:22:04.628522+0000","last_active":"2024-09-16T10:22:04.628522+0000","last_peered":"2024-09-16T10:22:04.628522+0000","last_clean":"2024-09-16T10:22:04.628522+0000","last_became_active":"2024-09-16T10:22:02.622331+0000","last_became_peered":"2024-09-16T10:22:02.622331+0000","last_unstale":"2024-09-16T10:22:04.628522+0000","last_undegraded":"2024-09-16T10:22:04.628522+0000","last_fullsized":"2024-09-16T10:22:04.628522+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.499939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":26783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":118784,"data_stored":26802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31240,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340600,"statfs":{"total":96636764160,"available":96604774400,"internally_reserved":0,"allocated":118784,"data_stored":26802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:07.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:08.073 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:08.073 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:08.085 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-16T10:22:06.546810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":85516,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":85115,"kb_avail":283030004,"statfs":{"total":289910292480,"available":289822724096,"internally_reserved":0,"allocated":352256,"data_stored":80387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":87158099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628454+0000","last_change":"2024-09-16T10:22:04.628454+0000","last_active":"2024-09-16T10:22:04.628454+0000","last_peered":"2024-09-16T10:22:04.628454+0000","last_clean":"2024-09-16T10:22:04.628454+0000","last_became_active":"2024-09-16T10:22:02.623835+0000","last_became_peered":"2024-09-16T10:22:02.623835+0000","last_unstale":"2024-09-16T10:22:04.628454+0000","last_undegraded":"2024-09-16T10:22:04.628454+0000","last_fullsized":"2024-09-16T10:22:04.628454+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:10.601195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.627934+0000","last_change":"2024-09-16T10:22:04.627934+0000","last_active":"2024-09-16T10:22:04.627934+0000","last_peered":"2024-09-16T10:22:04.627934+0000","last_clean":"2024-09-16T10:22:04.627934+0000","last_became_active":"2024-09-16T10:22:02.623453+0000","last_became_peered":"2024-09-16T10:22:02.623453+0000","last_unstale":"2024-09-16T10:22:04.627934+0000","last_undegraded":"2024-09-16T10:22:04.627934+0000","last_fullsized":"2024-09-16T10:22:04.627934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:59.318421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628838+0000","last_change":"2024-09-16T10:22:04.628838+0000","last_active":"2024-09-16T10:22:04.628838+0000","last_peered":"2024-09-16T10:22:04.628838+0000","last_clean":"2024-09-16T10:22:04.628838+0000","last_became_active":"2024-09-16T10:22:02.623394+0000","last_became_peered":"2024-09-16T10:22:02.623394+0000","last_unstale":"2024-09-16T10:22:04.628838+0000","last_undegraded":"2024-09-16T10:22:04.628838+0000","last_fullsized":"2024-09-16T10:22:04.628838+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:06.472472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011940130000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628932+0000","last_change":"2024-09-16T10:22:04.628932+0000","last_active":"2024-09-16T10:22:04.628932+0000","last_peered":"2024-09-16T10:22:04.628932+0000","last_clean":"2024-09-16T10:22:04.628932+0000","last_became_active":"2024-09-16T10:22:02.624545+0000","last_became_peered":"2024-09-16T10:22:02.624545+0000","last_unstale":"2024-09-16T10:22:04.628932+0000","last_undegraded":"2024-09-16T10:22:04.628932+0000","last_fullsized":"2024-09-16T10:22:04.628932+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:02.516293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"11'1","reported_seq":16,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.627907+0000","last_change":"2024-09-16T10:22:04.627907+0000","last_active":"2024-09-16T10:22:04.627907+0000","last_peered":"2024-09-16T10:22:04.627907+0000","last_clean":"2024-09-16T10:22:04.627907+0000","last_became_active":"2024-09-16T10:22:02.624518+0000","last_became_peered":"2024-09-16T10:22:02.624518+0000","last_unstale":"2024-09-16T10:22:04.627907+0000","last_undegraded":"2024-09-16T10:22:04.627907+0000","last_fullsized":"2024-09-16T10:22:04.627907+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:10.458645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"13'2","reported_seq":17,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.631208+0000","last_change":"2024-09-16T10:22:04.629301+0000","last_active":"2024-09-16T10:22:04.631208+0000","last_peered":"2024-09-16T10:22:04.631208+0000","last_clean":"2024-09-16T10:22:04.631208+0000","last_became_active":"2024-09-16T10:22:02.621803+0000","last_became_peered":"2024-09-16T10:22:02.621803+0000","last_unstale":"2024-09-16T10:22:04.631208+0000","last_undegraded":"2024-09-16T10:22:04.631208+0000","last_fullsized":"2024-09-16T10:22:04.631208+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:11.319895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628883+0000","last_change":"2024-09-16T10:22:04.628883+0000","last_active":"2024-09-16T10:22:04.628883+0000","last_peered":"2024-09-16T10:22:04.628883+0000","last_clean":"2024-09-16T10:22:04.628883+0000","last_became_active":"2024-09-16T10:22:02.625120+0000","last_became_peered":"2024-09-16T10:22:02.625120+0000","last_unstale":"2024-09-16T10:22:04.628883+0000","last_undegraded":"2024-09-16T10:22:04.628883+0000","last_fullsized":"2024-09-16T10:22:04.628883+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:48:42.724379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001146357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628522+0000","last_change":"2024-09-16T10:22:04.628522+0000","last_active":"2024-09-16T10:22:04.628522+0000","last_peered":"2024-09-16T10:22:04.628522+0000","last_clean":"2024-09-16T10:22:04.628522+0000","last_became_active":"2024-09-16T10:22:02.622331+0000","last_became_peered":"2024-09-16T10:22:02.622331+0000","last_unstale":"2024-09-16T10:22:04.628522+0000","last_undegraded":"2024-09-16T10:22:04.628522+0000","last_fullsized":"2024-09-16T10:22:04.628522+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.499939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":26783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":118784,"data_stored":26802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.318}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31240,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340600,"statfs":{"total":96636764160,"available":96604774400,"internally_reserved":0,"allocated":118784,"data_stored":26802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:08.086 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T10:22:08.086 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T10:22:08.086 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T10:22:08.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T10:22:08.404 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:08.405 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T10:22:08.417 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T10:22:08.417 INFO:teuthology.run_tasks:Running task rep_lost_unfound_delete... 2024-09-16T10:22:08.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:22:08.758 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:08.759 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":13,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:22:04.620914+0000","last_up_change":"2024-09-16T10:21:59.601797+0000","last_in_change":"2024-09-16T10:21:53.106207+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T10:22:00.707491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":3990881301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":3990881301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":3990881301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":3990881301}]},"public_addr":"172.21.15.50:6804/3990881301","cluster_addr":"172.21.15.50:6805/3990881301","heartbeat_back_addr":"172.21.15.50:6807/3990881301","heartbeat_front_addr":"172.21.15.50:6806/3990881301","state":["exists","up"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":202723672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":202723672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":202723672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":202723672}]},"public_addr":"172.21.15.50:6800/202723672","cluster_addr":"172.21.15.50:6801/202723672","heartbeat_back_addr":"172.21.15.50:6803/202723672","heartbeat_front_addr":"172.21.15.50:6802/202723672","state":["exists","up"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":2584589368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":2584589368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":2584589368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":2584589368}]},"public_addr":"172.21.15.50:6808/2584589368","cluster_addr":"172.21.15.50:6809/2584589368","heartbeat_back_addr":"172.21.15.50:6811/2584589368","heartbeat_front_addr":"172.21.15.50:6810/2584589368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.244822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.213317+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.228331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:22:08.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-16T10:22:00.707491+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 13, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T10:22:08.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T10:22:09.054 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 8 2024-09-16T10:22:09.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T10:22:09.348 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 13 2024-09-16T10:22:09.348 INFO:teuthology.orchestra.run.smithi050.stdout:fsid 3f765c65-edd5-4e6d-930a-33ce25cc666f 2024-09-16T10:22:09.348 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-09-16T10:21:51.212707+0000 2024-09-16T10:22:09.349 INFO:teuthology.orchestra.run.smithi050.stdout:modified 2024-09-16T10:22:04.620914+0000 2024-09-16T10:22:09.349 INFO:teuthology.orchestra.run.smithi050.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T10:22:09.349 INFO:teuthology.orchestra.run.smithi050.stdout:crush_version 4 2024-09-16T10:22:09.349 INFO:teuthology.orchestra.run.smithi050.stdout:full_ratio 0.95 2024-09-16T10:22:09.349 INFO:teuthology.orchestra.run.smithi050.stdout:backfillfull_ratio 0.9 2024-09-16T10:22:09.349 INFO:teuthology.orchestra.run.smithi050.stdout:nearfull_ratio 0.85 2024-09-16T10:22:09.349 INFO:teuthology.orchestra.run.smithi050.stdout:require_min_compat_client luminous 2024-09-16T10:22:09.350 INFO:teuthology.orchestra.run.smithi050.stdout:min_compat_client jewel 2024-09-16T10:22:09.350 INFO:teuthology.orchestra.run.smithi050.stdout:require_osd_release reef 2024-09-16T10:22:09.350 INFO:teuthology.orchestra.run.smithi050.stdout:stretch_mode_enabled false 2024-09-16T10:22:09.350 INFO:teuthology.orchestra.run.smithi050.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 13 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.88 2024-09-16T10:22:09.350 INFO:teuthology.orchestra.run.smithi050.stdout: removed_snaps_queue [2~1] 2024-09-16T10:22:09.350 INFO:teuthology.orchestra.run.smithi050.stdout:max_osd 3 2024-09-16T10:22:09.351 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6804/3990881301 v2:172.21.15.50:6805/3990881301 exists,up 346f1b3d-875c-427b-9fb3-a906c6ea62f8 2024-09-16T10:22:09.351 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6800/202723672 v2:172.21.15.50:6801/202723672 exists,up c991e2ab-7bd4-4499-bd8f-a2f7c5581928 2024-09-16T10:22:09.351 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6808/2584589368 v2:172.21.15.50:6809/2584589368 exists,up af14e6f0-c6ad-4545-ae6c-69130d4ec765 2024-09-16T10:22:09.360 INFO:tasks.rep_lost_unfound_delete.ceph_manager:['osd.0 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6804/3990881301 v2:172.21.15.50:6805/3990881301 exists,up 346f1b3d-875c-427b-9fb3-a906c6ea62f8', 'osd.1 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6800/202723672 v2:172.21.15.50:6801/202723672 exists,up c991e2ab-7bd4-4499-bd8f-a2f7c5581928', 'osd.2 up in weight 1 up_from 8 up_thru 10 down_at 0 last_clean_interval [0,0) v2:172.21.15.50:6808/2584589368 v2:172.21.15.50:6809/2584589368 exists,up af14e6f0-c6ad-4545-ae6c-69130d4ec765'] 2024-09-16T10:22:09.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T10:22:09.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T10:22:09.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T10:22:09.501 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2024-09-16T10:22:09.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:22:09.520 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2024-09-16T10:22:09.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:22:09.521 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2024-09-16T10:22:09.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:22:09.794 INFO:teuthology.orchestra.run.smithi050.stdout:34359738371 2024-09-16T10:22:09.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738372 got 34359738371 for osd.0 2024-09-16T10:22:09.850 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2024-09-16T10:22:09.852 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2024-09-16T10:22:09.864 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738372 for osd.1 2024-09-16T10:22:09.865 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738372 for osd.2 2024-09-16T10:22:10.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:22:10.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:22:10.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:22:11.116 INFO:teuthology.orchestra.run.smithi050.stdout:34359738372 2024-09-16T10:22:11.130 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738372 got 34359738372 for osd.0 2024-09-16T10:22:11.130 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:11.166 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2024-09-16T10:22:11.167 INFO:teuthology.orchestra.run.smithi050.stdout:34359738373 2024-09-16T10:22:11.179 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738373 for osd.2 2024-09-16T10:22:11.179 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:11.180 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738373 got 34359738373 for osd.1 2024-09-16T10:22:11.180 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:11.180 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for clean 2024-09-16T10:22:11.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:11.500 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:11.512 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-16T10:22:10.997026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":85516,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":85115,"kb_avail":283030004,"statfs":{"total":289910292480,"available":289822724096,"internally_reserved":0,"allocated":352256,"data_stored":80387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":87158099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.450215"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628454+0000","last_change":"2024-09-16T10:22:04.628454+0000","last_active":"2024-09-16T10:22:04.628454+0000","last_peered":"2024-09-16T10:22:04.628454+0000","last_clean":"2024-09-16T10:22:04.628454+0000","last_became_active":"2024-09-16T10:22:02.623835+0000","last_became_peered":"2024-09-16T10:22:02.623835+0000","last_unstale":"2024-09-16T10:22:04.628454+0000","last_undegraded":"2024-09-16T10:22:04.628454+0000","last_fullsized":"2024-09-16T10:22:04.628454+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:10.601195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.627934+0000","last_change":"2024-09-16T10:22:04.627934+0000","last_active":"2024-09-16T10:22:04.627934+0000","last_peered":"2024-09-16T10:22:04.627934+0000","last_clean":"2024-09-16T10:22:04.627934+0000","last_became_active":"2024-09-16T10:22:02.623453+0000","last_became_peered":"2024-09-16T10:22:02.623453+0000","last_unstale":"2024-09-16T10:22:04.627934+0000","last_undegraded":"2024-09-16T10:22:04.627934+0000","last_fullsized":"2024-09-16T10:22:04.627934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:59.318421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628838+0000","last_change":"2024-09-16T10:22:04.628838+0000","last_active":"2024-09-16T10:22:04.628838+0000","last_peered":"2024-09-16T10:22:04.628838+0000","last_clean":"2024-09-16T10:22:04.628838+0000","last_became_active":"2024-09-16T10:22:02.623394+0000","last_became_peered":"2024-09-16T10:22:02.623394+0000","last_unstale":"2024-09-16T10:22:04.628838+0000","last_undegraded":"2024-09-16T10:22:04.628838+0000","last_fullsized":"2024-09-16T10:22:04.628838+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:06.472472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011940130000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628932+0000","last_change":"2024-09-16T10:22:04.628932+0000","last_active":"2024-09-16T10:22:04.628932+0000","last_peered":"2024-09-16T10:22:04.628932+0000","last_clean":"2024-09-16T10:22:04.628932+0000","last_became_active":"2024-09-16T10:22:02.624545+0000","last_became_peered":"2024-09-16T10:22:02.624545+0000","last_unstale":"2024-09-16T10:22:04.628932+0000","last_undegraded":"2024-09-16T10:22:04.628932+0000","last_fullsized":"2024-09-16T10:22:04.628932+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:02.516293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"11'1","reported_seq":16,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.627907+0000","last_change":"2024-09-16T10:22:04.627907+0000","last_active":"2024-09-16T10:22:04.627907+0000","last_peered":"2024-09-16T10:22:04.627907+0000","last_clean":"2024-09-16T10:22:04.627907+0000","last_became_active":"2024-09-16T10:22:02.624518+0000","last_became_peered":"2024-09-16T10:22:02.624518+0000","last_unstale":"2024-09-16T10:22:04.627907+0000","last_undegraded":"2024-09-16T10:22:04.627907+0000","last_fullsized":"2024-09-16T10:22:04.627907+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:10.458645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"13'2","reported_seq":17,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.631208+0000","last_change":"2024-09-16T10:22:04.629301+0000","last_active":"2024-09-16T10:22:04.631208+0000","last_peered":"2024-09-16T10:22:04.631208+0000","last_clean":"2024-09-16T10:22:04.631208+0000","last_became_active":"2024-09-16T10:22:02.621803+0000","last_became_peered":"2024-09-16T10:22:02.621803+0000","last_unstale":"2024-09-16T10:22:04.631208+0000","last_undegraded":"2024-09-16T10:22:04.631208+0000","last_fullsized":"2024-09-16T10:22:04.631208+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:11.319895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628883+0000","last_change":"2024-09-16T10:22:04.628883+0000","last_active":"2024-09-16T10:22:04.628883+0000","last_peered":"2024-09-16T10:22:04.628883+0000","last_clean":"2024-09-16T10:22:04.628883+0000","last_became_active":"2024-09-16T10:22:02.625120+0000","last_became_peered":"2024-09-16T10:22:02.625120+0000","last_unstale":"2024-09-16T10:22:04.628883+0000","last_undegraded":"2024-09-16T10:22:04.628883+0000","last_fullsized":"2024-09-16T10:22:04.628883+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:48:42.724379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001146357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628522+0000","last_change":"2024-09-16T10:22:04.628522+0000","last_active":"2024-09-16T10:22:04.628522+0000","last_peered":"2024-09-16T10:22:04.628522+0000","last_clean":"2024-09-16T10:22:04.628522+0000","last_became_active":"2024-09-16T10:22:02.622331+0000","last_became_peered":"2024-09-16T10:22:02.622331+0000","last_unstale":"2024-09-16T10:22:04.628522+0000","last_undegraded":"2024-09-16T10:22:04.628522+0000","last_fullsized":"2024-09-16T10:22:04.628522+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.499939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":26783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":118784,"data_stored":26802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31240,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340600,"statfs":{"total":96636764160,"available":96604774400,"internally_reserved":0,"allocated":118784,"data_stored":26802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:11.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:11.794 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:11.795 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:11.807 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-16T10:22:10.997026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":85516,"kb_used_data":344,"kb_used_omap":4,"kb_used_meta":85115,"kb_avail":283030004,"statfs":{"total":289910292480,"available":289822724096,"internally_reserved":0,"allocated":352256,"data_stored":80387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":87158099},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.450215"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628454+0000","last_change":"2024-09-16T10:22:04.628454+0000","last_active":"2024-09-16T10:22:04.628454+0000","last_peered":"2024-09-16T10:22:04.628454+0000","last_clean":"2024-09-16T10:22:04.628454+0000","last_became_active":"2024-09-16T10:22:02.623835+0000","last_became_peered":"2024-09-16T10:22:02.623835+0000","last_unstale":"2024-09-16T10:22:04.628454+0000","last_undegraded":"2024-09-16T10:22:04.628454+0000","last_fullsized":"2024-09-16T10:22:04.628454+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:10.601195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.627934+0000","last_change":"2024-09-16T10:22:04.627934+0000","last_active":"2024-09-16T10:22:04.627934+0000","last_peered":"2024-09-16T10:22:04.627934+0000","last_clean":"2024-09-16T10:22:04.627934+0000","last_became_active":"2024-09-16T10:22:02.623453+0000","last_became_peered":"2024-09-16T10:22:02.623453+0000","last_unstale":"2024-09-16T10:22:04.627934+0000","last_undegraded":"2024-09-16T10:22:04.627934+0000","last_fullsized":"2024-09-16T10:22:04.627934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:59.318421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628838+0000","last_change":"2024-09-16T10:22:04.628838+0000","last_active":"2024-09-16T10:22:04.628838+0000","last_peered":"2024-09-16T10:22:04.628838+0000","last_clean":"2024-09-16T10:22:04.628838+0000","last_became_active":"2024-09-16T10:22:02.623394+0000","last_became_peered":"2024-09-16T10:22:02.623394+0000","last_unstale":"2024-09-16T10:22:04.628838+0000","last_undegraded":"2024-09-16T10:22:04.628838+0000","last_fullsized":"2024-09-16T10:22:04.628838+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:06.472472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011940130000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628932+0000","last_change":"2024-09-16T10:22:04.628932+0000","last_active":"2024-09-16T10:22:04.628932+0000","last_peered":"2024-09-16T10:22:04.628932+0000","last_clean":"2024-09-16T10:22:04.628932+0000","last_became_active":"2024-09-16T10:22:02.624545+0000","last_became_peered":"2024-09-16T10:22:02.624545+0000","last_unstale":"2024-09-16T10:22:04.628932+0000","last_undegraded":"2024-09-16T10:22:04.628932+0000","last_fullsized":"2024-09-16T10:22:04.628932+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:02.516293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"11'1","reported_seq":16,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.627907+0000","last_change":"2024-09-16T10:22:04.627907+0000","last_active":"2024-09-16T10:22:04.627907+0000","last_peered":"2024-09-16T10:22:04.627907+0000","last_clean":"2024-09-16T10:22:04.627907+0000","last_became_active":"2024-09-16T10:22:02.624518+0000","last_became_peered":"2024-09-16T10:22:02.624518+0000","last_unstale":"2024-09-16T10:22:04.627907+0000","last_undegraded":"2024-09-16T10:22:04.627907+0000","last_fullsized":"2024-09-16T10:22:04.627907+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:58:10.458645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"13'2","reported_seq":17,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.631208+0000","last_change":"2024-09-16T10:22:04.629301+0000","last_active":"2024-09-16T10:22:04.631208+0000","last_peered":"2024-09-16T10:22:04.631208+0000","last_clean":"2024-09-16T10:22:04.631208+0000","last_became_active":"2024-09-16T10:22:02.621803+0000","last_became_peered":"2024-09-16T10:22:02.621803+0000","last_unstale":"2024-09-16T10:22:04.631208+0000","last_undegraded":"2024-09-16T10:22:04.631208+0000","last_fullsized":"2024-09-16T10:22:04.631208+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:11.319895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628883+0000","last_change":"2024-09-16T10:22:04.628883+0000","last_active":"2024-09-16T10:22:04.628883+0000","last_peered":"2024-09-16T10:22:04.628883+0000","last_clean":"2024-09-16T10:22:04.628883+0000","last_became_active":"2024-09-16T10:22:02.625120+0000","last_became_peered":"2024-09-16T10:22:02.625120+0000","last_unstale":"2024-09-16T10:22:04.628883+0000","last_undegraded":"2024-09-16T10:22:04.628883+0000","last_fullsized":"2024-09-16T10:22:04.628883+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:48:42.724379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001146357,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-09-16T10:22:04.628522+0000","last_change":"2024-09-16T10:22:04.628522+0000","last_active":"2024-09-16T10:22:04.628522+0000","last_peered":"2024-09-16T10:22:04.628522+0000","last_clean":"2024-09-16T10:22:04.628522+0000","last_became_active":"2024-09-16T10:22:02.622331+0000","last_became_peered":"2024-09-16T10:22:02.622331+0000","last_unstale":"2024-09-16T10:22:04.628522+0000","last_undegraded":"2024-09-16T10:22:04.628522+0000","last_fullsized":"2024-09-16T10:22:04.628522+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.499939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":26783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":118784,"data_stored":26802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31240,"kb_used_data":116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340600,"statfs":{"total":96636764160,"available":96604774400,"internally_reserved":0,"allocated":118784,"data_stored":26802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:11.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:clean! 2024-09-16T10:22:11.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:creating pool_name unfounddel_pool 2024-09-16T10:22:11.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unfounddel_pool 16 2024-09-16T10:22:13.007 INFO:teuthology.orchestra.run.smithi050.stderr:pool 'unfounddel_pool' created 2024-09-16T10:22:13.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unfounddel_pool rados --yes-i-really-mean-it || true 2024-09-16T10:22:14.007 INFO:teuthology.orchestra.run.smithi050.stderr:enabled application 'rados' on pool 'unfounddel_pool' 2024-09-16T10:22:15.034 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T10:22:15.034 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:22:15.061 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T10:22:15.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2 2024-09-16T10:22:15.402 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-16T10:22:15.400+0000 7f0322f76640 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-16T10:22:16.021 INFO:teuthology.orchestra.run.smithi050.stderr:marked down osd.2. 2024-09-16T10:22:16.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-09-16T10:22:16.341 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-16T10:22:16.339+0000 7f0322f76640 -1 mon.a@0(leader).osd e18 definitely_dead 0 2024-09-16T10:22:17.022 INFO:teuthology.orchestra.run.smithi050.stderr:marked out osd.2. 2024-09-16T10:22:17.040 INFO:tasks.util.rados:rados -p unfounddel_pool put dummy /etc/fstab 2024-09-16T10:22:17.041 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put dummy /etc/fstab 2024-09-16T10:22:17.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T10:22:17.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T10:22:17.234 INFO:teuthology.orchestra.run.smithi050.stdout:34359738376 2024-09-16T10:22:17.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:22:17.235 INFO:teuthology.orchestra.run.smithi050.stdout:34359738375 2024-09-16T10:22:17.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:22:17.561 INFO:teuthology.orchestra.run.smithi050.stdout:34359738375 2024-09-16T10:22:17.566 INFO:teuthology.orchestra.run.smithi050.stdout:34359738374 2024-09-16T10:22:17.574 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738376 got 34359738375 for osd.1 2024-09-16T10:22:17.579 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738375 got 34359738374 for osd.0 2024-09-16T10:22:18.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:22:18.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:22:18.882 INFO:teuthology.orchestra.run.smithi050.stdout:34359738375 2024-09-16T10:22:18.895 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738376 got 34359738375 for osd.1 2024-09-16T10:22:18.915 INFO:teuthology.orchestra.run.smithi050.stdout:34359738374 2024-09-16T10:22:18.928 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738375 got 34359738374 for osd.0 2024-09-16T10:22:19.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:22:19.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:22:20.217 INFO:teuthology.orchestra.run.smithi050.stdout:34359738376 2024-09-16T10:22:20.231 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738376 got 34359738376 for osd.1 2024-09-16T10:22:20.231 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:20.248 INFO:teuthology.orchestra.run.smithi050.stdout:34359738375 2024-09-16T10:22:20.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738375 got 34359738375 for osd.0 2024-09-16T10:22:20.261 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:20.261 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for recovery to complete 2024-09-16T10:22:20.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:20.581 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:20.581 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:20.593 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-16T10:22:20.549112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58464,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":58108,"kb_avail":188685216,"statfs":{"total":193273528320,"available":193213661184,"internally_reserved":0,"allocated":327680,"data_stored":85517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":59503500},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.534311"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018733+0000","last_change":"2024-09-16T10:22:14.012743+0000","last_active":"2024-09-16T10:22:16.018733+0000","last_peered":"2024-09-16T10:22:16.018733+0000","last_clean":"2024-09-16T10:22:16.018733+0000","last_became_active":"2024-09-16T10:22:14.012441+0000","last_became_peered":"2024-09-16T10:22:14.012441+0000","last_unstale":"2024-09-16T10:22:16.018733+0000","last_undegraded":"2024-09-16T10:22:16.018733+0000","last_fullsized":"2024-09-16T10:22:16.018733+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:22:30.278434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019745+0000","last_change":"2024-09-16T10:22:17.019745+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.019745+0000","last_undegraded":"2024-09-16T10:22:17.019745+0000","last_fullsized":"2024-09-16T10:22:17.019745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:26:46.341891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.022611+0000","last_change":"2024-09-16T10:22:17.022723+0000","last_active":"2024-09-16T10:22:16.018763+0000","last_peered":"2024-09-16T10:22:15.604722+0000","last_clean":"2024-09-16T10:22:15.604722+0000","last_became_active":"2024-09-16T10:22:14.011425+0000","last_became_peered":"2024-09-16T10:22:14.011425+0000","last_unstale":"2024-09-16T10:22:17.022611+0000","last_undegraded":"2024-09-16T10:22:17.022611+0000","last_fullsized":"2024-09-16T10:22:17.022611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:55.807352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.020220+0000","last_change":"2024-09-16T10:22:17.020220+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.020220+0000","last_undegraded":"2024-09-16T10:22:17.020220+0000","last_fullsized":"2024-09-16T10:22:17.020220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:13:10.798122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019451+0000","last_change":"2024-09-16T10:22:17.019589+0000","last_active":"2024-09-16T10:22:16.020412+0000","last_peered":"2024-09-16T10:22:15.604561+0000","last_clean":"2024-09-16T10:22:15.604561+0000","last_became_active":"2024-09-16T10:22:14.016123+0000","last_became_peered":"2024-09-16T10:22:14.016123+0000","last_unstale":"2024-09-16T10:22:17.019451+0000","last_undegraded":"2024-09-16T10:22:17.019451+0000","last_fullsized":"2024-09-16T10:22:17.019451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:00.901249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.017723+0000","last_change":"2024-09-16T10:22:14.017879+0000","last_active":"2024-09-16T10:22:16.017723+0000","last_peered":"2024-09-16T10:22:16.017723+0000","last_clean":"2024-09-16T10:22:16.017723+0000","last_became_active":"2024-09-16T10:22:14.016590+0000","last_became_peered":"2024-09-16T10:22:14.016590+0000","last_unstale":"2024-09-16T10:22:16.017723+0000","last_undegraded":"2024-09-16T10:22:16.017723+0000","last_fullsized":"2024-09-16T10:22:16.017723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:05:29.446766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018439+0000","last_change":"2024-09-16T10:22:14.013439+0000","last_active":"2024-09-16T10:22:16.018439+0000","last_peered":"2024-09-16T10:22:16.018439+0000","last_clean":"2024-09-16T10:22:16.018439+0000","last_became_active":"2024-09-16T10:22:14.013167+0000","last_became_peered":"2024-09-16T10:22:14.013167+0000","last_unstale":"2024-09-16T10:22:16.018439+0000","last_undegraded":"2024-09-16T10:22:16.018439+0000","last_fullsized":"2024-09-16T10:22:16.018439+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:03:22.706300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019069+0000","last_change":"2024-09-16T10:22:17.019896+0000","last_active":"2024-09-16T10:22:16.018695+0000","last_peered":"2024-09-16T10:22:15.604528+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:22:14.016413+0000","last_became_peered":"2024-09-16T10:22:14.016413+0000","last_unstale":"2024-09-16T10:22:17.019069+0000","last_undegraded":"2024-09-16T10:22:17.019069+0000","last_fullsized":"2024-09-16T10:22:17.019069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:24.656917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.019485+0000","last_change":"2024-09-16T10:22:04.628522+0000","last_active":"2024-09-16T10:22:16.019485+0000","last_peered":"2024-09-16T10:22:16.019485+0000","last_clean":"2024-09-16T10:22:16.019485+0000","last_became_active":"2024-09-16T10:22:02.622331+0000","last_became_peered":"2024-09-16T10:22:02.622331+0000","last_unstale":"2024-09-16T10:22:16.019485+0000","last_undegraded":"2024-09-16T10:22:16.019485+0000","last_fullsized":"2024-09-16T10:22:16.019485+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.499939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.022392+0000","last_change":"2024-09-16T10:22:17.022505+0000","last_active":"2024-09-16T10:22:16.018394+0000","last_peered":"2024-09-16T10:22:15.604668+0000","last_clean":"2024-09-16T10:22:15.604668+0000","last_became_active":"2024-09-16T10:22:14.017312+0000","last_became_peered":"2024-09-16T10:22:14.017312+0000","last_unstale":"2024-09-16T10:22:17.022392+0000","last_undegraded":"2024-09-16T10:22:17.022392+0000","last_fullsized":"2024-09-16T10:22:17.022392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:34.558297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019619+0000","last_change":"2024-09-16T10:22:17.019619+0000","last_active":"2024-09-16T10:22:01.610622+0000","last_peered":"2024-09-16T10:22:01.610622+0000","last_clean":"2024-09-16T10:22:01.610622+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.019619+0000","last_undegraded":"2024-09-16T10:22:17.019619+0000","last_fullsized":"2024-09-16T10:22:17.019619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:07:30.261928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018952+0000","last_change":"2024-09-16T10:22:14.012931+0000","last_active":"2024-09-16T10:22:16.018952+0000","last_peered":"2024-09-16T10:22:16.018952+0000","last_clean":"2024-09-16T10:22:16.018952+0000","last_became_active":"2024-09-16T10:22:14.012491+0000","last_became_peered":"2024-09-16T10:22:14.012491+0000","last_unstale":"2024-09-16T10:22:16.018952+0000","last_undegraded":"2024-09-16T10:22:16.018952+0000","last_fullsized":"2024-09-16T10:22:16.018952+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:25.655709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018978+0000","last_change":"2024-09-16T10:22:04.629301+0000","last_active":"2024-09-16T10:22:16.018978+0000","last_peered":"2024-09-16T10:22:16.018978+0000","last_clean":"2024-09-16T10:22:16.018978+0000","last_became_active":"2024-09-16T10:22:02.621803+0000","last_became_peered":"2024-09-16T10:22:02.621803+0000","last_unstale":"2024-09-16T10:22:16.018978+0000","last_undegraded":"2024-09-16T10:22:16.018978+0000","last_fullsized":"2024-09-16T10:22:16.018978+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:11.319895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.021116+0000","last_change":"2024-09-16T10:22:17.021116+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.021116+0000","last_undegraded":"2024-09-16T10:22:17.021116+0000","last_fullsized":"2024-09-16T10:22:17.021116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:38.804693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":25,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.022128+0000","last_change":"2024-09-16T10:22:17.022277+0000","last_active":"2024-09-16T10:22:16.019501+0000","last_peered":"2024-09-16T10:22:15.017989+0000","last_clean":"2024-09-16T10:22:15.017989+0000","last_became_active":"2024-09-16T10:22:02.624518+0000","last_became_peered":"2024-09-16T10:22:02.624518+0000","last_unstale":"2024-09-16T10:22:17.022128+0000","last_undegraded":"2024-09-16T10:22:17.022128+0000","last_fullsized":"2024-09-16T10:22:17.022128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:53.404588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.021083+0000","last_change":"2024-09-16T10:22:17.021083+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.021083+0000","last_undegraded":"2024-09-16T10:22:17.021083+0000","last_fullsized":"2024-09-16T10:22:17.021083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:39.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018567+0000","last_change":"2024-09-16T10:22:14.013058+0000","last_active":"2024-09-16T10:22:16.018567+0000","last_peered":"2024-09-16T10:22:16.018567+0000","last_clean":"2024-09-16T10:22:16.018567+0000","last_became_active":"2024-09-16T10:22:14.012790+0000","last_became_peered":"2024-09-16T10:22:14.012790+0000","last_unstale":"2024-09-16T10:22:16.018567+0000","last_undegraded":"2024-09-16T10:22:16.018567+0000","last_fullsized":"2024-09-16T10:22:16.018567+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:29.964382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.017934+0000","last_change":"2024-09-16T10:22:04.628932+0000","last_active":"2024-09-16T10:22:16.017934+0000","last_peered":"2024-09-16T10:22:16.017934+0000","last_clean":"2024-09-16T10:22:16.017934+0000","last_became_active":"2024-09-16T10:22:02.624545+0000","last_became_peered":"2024-09-16T10:22:02.624545+0000","last_unstale":"2024-09-16T10:22:16.017934+0000","last_undegraded":"2024-09-16T10:22:16.017934+0000","last_fullsized":"2024-09-16T10:22:16.017934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:02.516293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.019421+0000","last_change":"2024-09-16T10:22:14.016687+0000","last_active":"2024-09-16T10:22:16.019421+0000","last_peered":"2024-09-16T10:22:16.019421+0000","last_clean":"2024-09-16T10:22:16.019421+0000","last_became_active":"2024-09-16T10:22:14.016431+0000","last_became_peered":"2024-09-16T10:22:14.016431+0000","last_unstale":"2024-09-16T10:22:16.019421+0000","last_undegraded":"2024-09-16T10:22:16.019421+0000","last_fullsized":"2024-09-16T10:22:16.019421+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:01.014731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019926+0000","last_change":"2024-09-16T10:22:17.019926+0000","last_active":"2024-09-16T10:22:01.610622+0000","last_peered":"2024-09-16T10:22:01.610622+0000","last_clean":"2024-09-16T10:22:01.610622+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.019926+0000","last_undegraded":"2024-09-16T10:22:17.019926+0000","last_fullsized":"2024-09-16T10:22:17.019926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:16:55.693372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018593+0000","last_change":"2024-09-16T10:22:14.012517+0000","last_active":"2024-09-16T10:22:16.018593+0000","last_peered":"2024-09-16T10:22:16.018593+0000","last_clean":"2024-09-16T10:22:16.018593+0000","last_became_active":"2024-09-16T10:22:14.012267+0000","last_became_peered":"2024-09-16T10:22:14.012267+0000","last_unstale":"2024-09-16T10:22:16.018593+0000","last_undegraded":"2024-09-16T10:22:16.018593+0000","last_fullsized":"2024-09-16T10:22:16.018593+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:01:20.793597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.019415+0000","last_change":"2024-09-16T10:22:04.627934+0000","last_active":"2024-09-16T10:22:16.019415+0000","last_peered":"2024-09-16T10:22:16.019415+0000","last_clean":"2024-09-16T10:22:16.019415+0000","last_became_active":"2024-09-16T10:22:02.623453+0000","last_became_peered":"2024-09-16T10:22:02.623453+0000","last_unstale":"2024-09-16T10:22:16.019415+0000","last_undegraded":"2024-09-16T10:22:16.019415+0000","last_fullsized":"2024-09-16T10:22:16.019415+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:59.318421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":24,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.023125+0000","last_change":"2024-09-16T10:22:17.023241+0000","last_active":"2024-09-16T10:22:16.019318+0000","last_peered":"2024-09-16T10:22:15.017867+0000","last_clean":"2024-09-16T10:22:15.017867+0000","last_became_active":"2024-09-16T10:22:02.623835+0000","last_became_peered":"2024-09-16T10:22:02.623835+0000","last_unstale":"2024-09-16T10:22:17.023125+0000","last_undegraded":"2024-09-16T10:22:17.023125+0000","last_fullsized":"2024-09-16T10:22:17.023125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:13.467485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.022088+0000","last_change":"2024-09-16T10:22:14.016018+0000","last_active":"2024-09-16T10:22:17.022088+0000","last_peered":"2024-09-16T10:22:17.022088+0000","last_clean":"2024-09-16T10:22:17.022088+0000","last_became_active":"2024-09-16T10:22:14.015752+0000","last_became_peered":"2024-09-16T10:22:14.015752+0000","last_unstale":"2024-09-16T10:22:17.022088+0000","last_undegraded":"2024-09-16T10:22:17.022088+0000","last_fullsized":"2024-09-16T10:22:17.022088+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:51:06.878869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":159744,"data_stored":41371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31288,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340552,"statfs":{"total":96636764160,"available":96604725248,"internally_reserved":0,"allocated":167936,"data_stored":44146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:20.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:20.877 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:20.878 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:20.890 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-16T10:22:20.549112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58464,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":58108,"kb_avail":188685216,"statfs":{"total":193273528320,"available":193213661184,"internally_reserved":0,"allocated":327680,"data_stored":85517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":59503500},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.534311"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018733+0000","last_change":"2024-09-16T10:22:14.012743+0000","last_active":"2024-09-16T10:22:16.018733+0000","last_peered":"2024-09-16T10:22:16.018733+0000","last_clean":"2024-09-16T10:22:16.018733+0000","last_became_active":"2024-09-16T10:22:14.012441+0000","last_became_peered":"2024-09-16T10:22:14.012441+0000","last_unstale":"2024-09-16T10:22:16.018733+0000","last_undegraded":"2024-09-16T10:22:16.018733+0000","last_fullsized":"2024-09-16T10:22:16.018733+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:22:30.278434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019745+0000","last_change":"2024-09-16T10:22:17.019745+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.019745+0000","last_undegraded":"2024-09-16T10:22:17.019745+0000","last_fullsized":"2024-09-16T10:22:17.019745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:26:46.341891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.022611+0000","last_change":"2024-09-16T10:22:17.022723+0000","last_active":"2024-09-16T10:22:16.018763+0000","last_peered":"2024-09-16T10:22:15.604722+0000","last_clean":"2024-09-16T10:22:15.604722+0000","last_became_active":"2024-09-16T10:22:14.011425+0000","last_became_peered":"2024-09-16T10:22:14.011425+0000","last_unstale":"2024-09-16T10:22:17.022611+0000","last_undegraded":"2024-09-16T10:22:17.022611+0000","last_fullsized":"2024-09-16T10:22:17.022611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:55.807352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.020220+0000","last_change":"2024-09-16T10:22:17.020220+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.020220+0000","last_undegraded":"2024-09-16T10:22:17.020220+0000","last_fullsized":"2024-09-16T10:22:17.020220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:13:10.798122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019451+0000","last_change":"2024-09-16T10:22:17.019589+0000","last_active":"2024-09-16T10:22:16.020412+0000","last_peered":"2024-09-16T10:22:15.604561+0000","last_clean":"2024-09-16T10:22:15.604561+0000","last_became_active":"2024-09-16T10:22:14.016123+0000","last_became_peered":"2024-09-16T10:22:14.016123+0000","last_unstale":"2024-09-16T10:22:17.019451+0000","last_undegraded":"2024-09-16T10:22:17.019451+0000","last_fullsized":"2024-09-16T10:22:17.019451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:00.901249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.017723+0000","last_change":"2024-09-16T10:22:14.017879+0000","last_active":"2024-09-16T10:22:16.017723+0000","last_peered":"2024-09-16T10:22:16.017723+0000","last_clean":"2024-09-16T10:22:16.017723+0000","last_became_active":"2024-09-16T10:22:14.016590+0000","last_became_peered":"2024-09-16T10:22:14.016590+0000","last_unstale":"2024-09-16T10:22:16.017723+0000","last_undegraded":"2024-09-16T10:22:16.017723+0000","last_fullsized":"2024-09-16T10:22:16.017723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:05:29.446766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018439+0000","last_change":"2024-09-16T10:22:14.013439+0000","last_active":"2024-09-16T10:22:16.018439+0000","last_peered":"2024-09-16T10:22:16.018439+0000","last_clean":"2024-09-16T10:22:16.018439+0000","last_became_active":"2024-09-16T10:22:14.013167+0000","last_became_peered":"2024-09-16T10:22:14.013167+0000","last_unstale":"2024-09-16T10:22:16.018439+0000","last_undegraded":"2024-09-16T10:22:16.018439+0000","last_fullsized":"2024-09-16T10:22:16.018439+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:03:22.706300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019069+0000","last_change":"2024-09-16T10:22:17.019896+0000","last_active":"2024-09-16T10:22:16.018695+0000","last_peered":"2024-09-16T10:22:15.604528+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:22:14.016413+0000","last_became_peered":"2024-09-16T10:22:14.016413+0000","last_unstale":"2024-09-16T10:22:17.019069+0000","last_undegraded":"2024-09-16T10:22:17.019069+0000","last_fullsized":"2024-09-16T10:22:17.019069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:24.656917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.019485+0000","last_change":"2024-09-16T10:22:04.628522+0000","last_active":"2024-09-16T10:22:16.019485+0000","last_peered":"2024-09-16T10:22:16.019485+0000","last_clean":"2024-09-16T10:22:16.019485+0000","last_became_active":"2024-09-16T10:22:02.622331+0000","last_became_peered":"2024-09-16T10:22:02.622331+0000","last_unstale":"2024-09-16T10:22:16.019485+0000","last_undegraded":"2024-09-16T10:22:16.019485+0000","last_fullsized":"2024-09-16T10:22:16.019485+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.499939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.022392+0000","last_change":"2024-09-16T10:22:17.022505+0000","last_active":"2024-09-16T10:22:16.018394+0000","last_peered":"2024-09-16T10:22:15.604668+0000","last_clean":"2024-09-16T10:22:15.604668+0000","last_became_active":"2024-09-16T10:22:14.017312+0000","last_became_peered":"2024-09-16T10:22:14.017312+0000","last_unstale":"2024-09-16T10:22:17.022392+0000","last_undegraded":"2024-09-16T10:22:17.022392+0000","last_fullsized":"2024-09-16T10:22:17.022392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:34.558297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019619+0000","last_change":"2024-09-16T10:22:17.019619+0000","last_active":"2024-09-16T10:22:01.610622+0000","last_peered":"2024-09-16T10:22:01.610622+0000","last_clean":"2024-09-16T10:22:01.610622+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.019619+0000","last_undegraded":"2024-09-16T10:22:17.019619+0000","last_fullsized":"2024-09-16T10:22:17.019619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:07:30.261928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018952+0000","last_change":"2024-09-16T10:22:14.012931+0000","last_active":"2024-09-16T10:22:16.018952+0000","last_peered":"2024-09-16T10:22:16.018952+0000","last_clean":"2024-09-16T10:22:16.018952+0000","last_became_active":"2024-09-16T10:22:14.012491+0000","last_became_peered":"2024-09-16T10:22:14.012491+0000","last_unstale":"2024-09-16T10:22:16.018952+0000","last_undegraded":"2024-09-16T10:22:16.018952+0000","last_fullsized":"2024-09-16T10:22:16.018952+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:25.655709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018978+0000","last_change":"2024-09-16T10:22:04.629301+0000","last_active":"2024-09-16T10:22:16.018978+0000","last_peered":"2024-09-16T10:22:16.018978+0000","last_clean":"2024-09-16T10:22:16.018978+0000","last_became_active":"2024-09-16T10:22:02.621803+0000","last_became_peered":"2024-09-16T10:22:02.621803+0000","last_unstale":"2024-09-16T10:22:16.018978+0000","last_undegraded":"2024-09-16T10:22:16.018978+0000","last_fullsized":"2024-09-16T10:22:16.018978+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:11.319895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.021116+0000","last_change":"2024-09-16T10:22:17.021116+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.021116+0000","last_undegraded":"2024-09-16T10:22:17.021116+0000","last_fullsized":"2024-09-16T10:22:17.021116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:38.804693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":25,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.022128+0000","last_change":"2024-09-16T10:22:17.022277+0000","last_active":"2024-09-16T10:22:16.019501+0000","last_peered":"2024-09-16T10:22:15.017989+0000","last_clean":"2024-09-16T10:22:15.017989+0000","last_became_active":"2024-09-16T10:22:02.624518+0000","last_became_peered":"2024-09-16T10:22:02.624518+0000","last_unstale":"2024-09-16T10:22:17.022128+0000","last_undegraded":"2024-09-16T10:22:17.022128+0000","last_fullsized":"2024-09-16T10:22:17.022128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:53.404588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.021083+0000","last_change":"2024-09-16T10:22:17.021083+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.021083+0000","last_undegraded":"2024-09-16T10:22:17.021083+0000","last_fullsized":"2024-09-16T10:22:17.021083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:39.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018567+0000","last_change":"2024-09-16T10:22:14.013058+0000","last_active":"2024-09-16T10:22:16.018567+0000","last_peered":"2024-09-16T10:22:16.018567+0000","last_clean":"2024-09-16T10:22:16.018567+0000","last_became_active":"2024-09-16T10:22:14.012790+0000","last_became_peered":"2024-09-16T10:22:14.012790+0000","last_unstale":"2024-09-16T10:22:16.018567+0000","last_undegraded":"2024-09-16T10:22:16.018567+0000","last_fullsized":"2024-09-16T10:22:16.018567+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:29.964382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.017934+0000","last_change":"2024-09-16T10:22:04.628932+0000","last_active":"2024-09-16T10:22:16.017934+0000","last_peered":"2024-09-16T10:22:16.017934+0000","last_clean":"2024-09-16T10:22:16.017934+0000","last_became_active":"2024-09-16T10:22:02.624545+0000","last_became_peered":"2024-09-16T10:22:02.624545+0000","last_unstale":"2024-09-16T10:22:16.017934+0000","last_undegraded":"2024-09-16T10:22:16.017934+0000","last_fullsized":"2024-09-16T10:22:16.017934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:02.516293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.019421+0000","last_change":"2024-09-16T10:22:14.016687+0000","last_active":"2024-09-16T10:22:16.019421+0000","last_peered":"2024-09-16T10:22:16.019421+0000","last_clean":"2024-09-16T10:22:16.019421+0000","last_became_active":"2024-09-16T10:22:14.016431+0000","last_became_peered":"2024-09-16T10:22:14.016431+0000","last_unstale":"2024-09-16T10:22:16.019421+0000","last_undegraded":"2024-09-16T10:22:16.019421+0000","last_fullsized":"2024-09-16T10:22:16.019421+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:01.014731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019926+0000","last_change":"2024-09-16T10:22:17.019926+0000","last_active":"2024-09-16T10:22:01.610622+0000","last_peered":"2024-09-16T10:22:01.610622+0000","last_clean":"2024-09-16T10:22:01.610622+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.019926+0000","last_undegraded":"2024-09-16T10:22:17.019926+0000","last_fullsized":"2024-09-16T10:22:17.019926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:16:55.693372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018593+0000","last_change":"2024-09-16T10:22:14.012517+0000","last_active":"2024-09-16T10:22:16.018593+0000","last_peered":"2024-09-16T10:22:16.018593+0000","last_clean":"2024-09-16T10:22:16.018593+0000","last_became_active":"2024-09-16T10:22:14.012267+0000","last_became_peered":"2024-09-16T10:22:14.012267+0000","last_unstale":"2024-09-16T10:22:16.018593+0000","last_undegraded":"2024-09-16T10:22:16.018593+0000","last_fullsized":"2024-09-16T10:22:16.018593+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:01:20.793597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.019415+0000","last_change":"2024-09-16T10:22:04.627934+0000","last_active":"2024-09-16T10:22:16.019415+0000","last_peered":"2024-09-16T10:22:16.019415+0000","last_clean":"2024-09-16T10:22:16.019415+0000","last_became_active":"2024-09-16T10:22:02.623453+0000","last_became_peered":"2024-09-16T10:22:02.623453+0000","last_unstale":"2024-09-16T10:22:16.019415+0000","last_undegraded":"2024-09-16T10:22:16.019415+0000","last_fullsized":"2024-09-16T10:22:16.019415+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:59.318421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":24,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.023125+0000","last_change":"2024-09-16T10:22:17.023241+0000","last_active":"2024-09-16T10:22:16.019318+0000","last_peered":"2024-09-16T10:22:15.017867+0000","last_clean":"2024-09-16T10:22:15.017867+0000","last_became_active":"2024-09-16T10:22:02.623835+0000","last_became_peered":"2024-09-16T10:22:02.623835+0000","last_unstale":"2024-09-16T10:22:17.023125+0000","last_undegraded":"2024-09-16T10:22:17.023125+0000","last_fullsized":"2024-09-16T10:22:17.023125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:13.467485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.022088+0000","last_change":"2024-09-16T10:22:14.016018+0000","last_active":"2024-09-16T10:22:17.022088+0000","last_peered":"2024-09-16T10:22:17.022088+0000","last_clean":"2024-09-16T10:22:17.022088+0000","last_became_active":"2024-09-16T10:22:14.015752+0000","last_became_peered":"2024-09-16T10:22:14.015752+0000","last_unstale":"2024-09-16T10:22:17.022088+0000","last_undegraded":"2024-09-16T10:22:17.022088+0000","last_fullsized":"2024-09-16T10:22:17.022088+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:51:06.878869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":159744,"data_stored":41371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31288,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340552,"statfs":{"total":96636764160,"available":96604725248,"internally_reserved":0,"allocated":167936,"data_stored":44146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:20.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:21.170 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:21.171 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:21.182 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-16T10:22:20.549112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":28,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58464,"kb_used_data":320,"kb_used_omap":3,"kb_used_meta":58108,"kb_avail":188685216,"statfs":{"total":193273528320,"available":193213661184,"internally_reserved":0,"allocated":327680,"data_stored":85517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":59503500},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.534311"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018733+0000","last_change":"2024-09-16T10:22:14.012743+0000","last_active":"2024-09-16T10:22:16.018733+0000","last_peered":"2024-09-16T10:22:16.018733+0000","last_clean":"2024-09-16T10:22:16.018733+0000","last_became_active":"2024-09-16T10:22:14.012441+0000","last_became_peered":"2024-09-16T10:22:14.012441+0000","last_unstale":"2024-09-16T10:22:16.018733+0000","last_undegraded":"2024-09-16T10:22:16.018733+0000","last_fullsized":"2024-09-16T10:22:16.018733+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:22:30.278434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019745+0000","last_change":"2024-09-16T10:22:17.019745+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.019745+0000","last_undegraded":"2024-09-16T10:22:17.019745+0000","last_fullsized":"2024-09-16T10:22:17.019745+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:26:46.341891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.022611+0000","last_change":"2024-09-16T10:22:17.022723+0000","last_active":"2024-09-16T10:22:16.018763+0000","last_peered":"2024-09-16T10:22:15.604722+0000","last_clean":"2024-09-16T10:22:15.604722+0000","last_became_active":"2024-09-16T10:22:14.011425+0000","last_became_peered":"2024-09-16T10:22:14.011425+0000","last_unstale":"2024-09-16T10:22:17.022611+0000","last_undegraded":"2024-09-16T10:22:17.022611+0000","last_fullsized":"2024-09-16T10:22:17.022611+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:55.807352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.020220+0000","last_change":"2024-09-16T10:22:17.020220+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.020220+0000","last_undegraded":"2024-09-16T10:22:17.020220+0000","last_fullsized":"2024-09-16T10:22:17.020220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:13:10.798122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019451+0000","last_change":"2024-09-16T10:22:17.019589+0000","last_active":"2024-09-16T10:22:16.020412+0000","last_peered":"2024-09-16T10:22:15.604561+0000","last_clean":"2024-09-16T10:22:15.604561+0000","last_became_active":"2024-09-16T10:22:14.016123+0000","last_became_peered":"2024-09-16T10:22:14.016123+0000","last_unstale":"2024-09-16T10:22:17.019451+0000","last_undegraded":"2024-09-16T10:22:17.019451+0000","last_fullsized":"2024-09-16T10:22:17.019451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:53:00.901249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.017723+0000","last_change":"2024-09-16T10:22:14.017879+0000","last_active":"2024-09-16T10:22:16.017723+0000","last_peered":"2024-09-16T10:22:16.017723+0000","last_clean":"2024-09-16T10:22:16.017723+0000","last_became_active":"2024-09-16T10:22:14.016590+0000","last_became_peered":"2024-09-16T10:22:14.016590+0000","last_unstale":"2024-09-16T10:22:16.017723+0000","last_undegraded":"2024-09-16T10:22:16.017723+0000","last_fullsized":"2024-09-16T10:22:16.017723+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:05:29.446766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018439+0000","last_change":"2024-09-16T10:22:14.013439+0000","last_active":"2024-09-16T10:22:16.018439+0000","last_peered":"2024-09-16T10:22:16.018439+0000","last_clean":"2024-09-16T10:22:16.018439+0000","last_became_active":"2024-09-16T10:22:14.013167+0000","last_became_peered":"2024-09-16T10:22:14.013167+0000","last_unstale":"2024-09-16T10:22:16.018439+0000","last_undegraded":"2024-09-16T10:22:16.018439+0000","last_fullsized":"2024-09-16T10:22:16.018439+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:03:22.706300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019069+0000","last_change":"2024-09-16T10:22:17.019896+0000","last_active":"2024-09-16T10:22:16.018695+0000","last_peered":"2024-09-16T10:22:15.604528+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:22:14.016413+0000","last_became_peered":"2024-09-16T10:22:14.016413+0000","last_unstale":"2024-09-16T10:22:17.019069+0000","last_undegraded":"2024-09-16T10:22:17.019069+0000","last_fullsized":"2024-09-16T10:22:17.019069+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:50:24.656917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.019485+0000","last_change":"2024-09-16T10:22:04.628522+0000","last_active":"2024-09-16T10:22:16.019485+0000","last_peered":"2024-09-16T10:22:16.019485+0000","last_clean":"2024-09-16T10:22:16.019485+0000","last_became_active":"2024-09-16T10:22:02.622331+0000","last_became_peered":"2024-09-16T10:22:02.622331+0000","last_unstale":"2024-09-16T10:22:16.019485+0000","last_undegraded":"2024-09-16T10:22:16.019485+0000","last_fullsized":"2024-09-16T10:22:16.019485+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.499939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.022392+0000","last_change":"2024-09-16T10:22:17.022505+0000","last_active":"2024-09-16T10:22:16.018394+0000","last_peered":"2024-09-16T10:22:15.604668+0000","last_clean":"2024-09-16T10:22:15.604668+0000","last_became_active":"2024-09-16T10:22:14.017312+0000","last_became_peered":"2024-09-16T10:22:14.017312+0000","last_unstale":"2024-09-16T10:22:17.022392+0000","last_undegraded":"2024-09-16T10:22:17.022392+0000","last_fullsized":"2024-09-16T10:22:17.022392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:24:34.558297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019619+0000","last_change":"2024-09-16T10:22:17.019619+0000","last_active":"2024-09-16T10:22:01.610622+0000","last_peered":"2024-09-16T10:22:01.610622+0000","last_clean":"2024-09-16T10:22:01.610622+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.019619+0000","last_undegraded":"2024-09-16T10:22:17.019619+0000","last_fullsized":"2024-09-16T10:22:17.019619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:07:30.261928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018952+0000","last_change":"2024-09-16T10:22:14.012931+0000","last_active":"2024-09-16T10:22:16.018952+0000","last_peered":"2024-09-16T10:22:16.018952+0000","last_clean":"2024-09-16T10:22:16.018952+0000","last_became_active":"2024-09-16T10:22:14.012491+0000","last_became_peered":"2024-09-16T10:22:14.012491+0000","last_unstale":"2024-09-16T10:22:16.018952+0000","last_undegraded":"2024-09-16T10:22:16.018952+0000","last_fullsized":"2024-09-16T10:22:16.018952+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:25.655709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":24,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018978+0000","last_change":"2024-09-16T10:22:04.629301+0000","last_active":"2024-09-16T10:22:16.018978+0000","last_peered":"2024-09-16T10:22:16.018978+0000","last_clean":"2024-09-16T10:22:16.018978+0000","last_became_active":"2024-09-16T10:22:02.621803+0000","last_became_peered":"2024-09-16T10:22:02.621803+0000","last_unstale":"2024-09-16T10:22:16.018978+0000","last_undegraded":"2024-09-16T10:22:16.018978+0000","last_fullsized":"2024-09-16T10:22:16.018978+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:11.319895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.021116+0000","last_change":"2024-09-16T10:22:17.021116+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.021116+0000","last_undegraded":"2024-09-16T10:22:17.021116+0000","last_fullsized":"2024-09-16T10:22:17.021116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:38.804693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":25,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.022128+0000","last_change":"2024-09-16T10:22:17.022277+0000","last_active":"2024-09-16T10:22:16.019501+0000","last_peered":"2024-09-16T10:22:15.017989+0000","last_clean":"2024-09-16T10:22:15.017989+0000","last_became_active":"2024-09-16T10:22:02.624518+0000","last_became_peered":"2024-09-16T10:22:02.624518+0000","last_unstale":"2024-09-16T10:22:17.022128+0000","last_undegraded":"2024-09-16T10:22:17.022128+0000","last_fullsized":"2024-09-16T10:22:17.022128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:42:53.404588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.021083+0000","last_change":"2024-09-16T10:22:17.021083+0000","last_active":"2024-09-16T10:22:12.997533+0000","last_peered":"2024-09-16T10:22:12.997533+0000","last_clean":"2024-09-16T10:22:12.997533+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.021083+0000","last_undegraded":"2024-09-16T10:22:17.021083+0000","last_fullsized":"2024-09-16T10:22:17.021083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:39.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018567+0000","last_change":"2024-09-16T10:22:14.013058+0000","last_active":"2024-09-16T10:22:16.018567+0000","last_peered":"2024-09-16T10:22:16.018567+0000","last_clean":"2024-09-16T10:22:16.018567+0000","last_became_active":"2024-09-16T10:22:14.012790+0000","last_became_peered":"2024-09-16T10:22:14.012790+0000","last_unstale":"2024-09-16T10:22:16.018567+0000","last_undegraded":"2024-09-16T10:22:16.018567+0000","last_fullsized":"2024-09-16T10:22:16.018567+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:29.964382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.017934+0000","last_change":"2024-09-16T10:22:04.628932+0000","last_active":"2024-09-16T10:22:16.017934+0000","last_peered":"2024-09-16T10:22:16.017934+0000","last_clean":"2024-09-16T10:22:16.017934+0000","last_became_active":"2024-09-16T10:22:02.624545+0000","last_became_peered":"2024-09-16T10:22:02.624545+0000","last_unstale":"2024-09-16T10:22:16.017934+0000","last_undegraded":"2024-09-16T10:22:16.017934+0000","last_fullsized":"2024-09-16T10:22:16.017934+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:02.516293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.019421+0000","last_change":"2024-09-16T10:22:14.016687+0000","last_active":"2024-09-16T10:22:16.019421+0000","last_peered":"2024-09-16T10:22:16.019421+0000","last_clean":"2024-09-16T10:22:16.019421+0000","last_became_active":"2024-09-16T10:22:14.016431+0000","last_became_peered":"2024-09-16T10:22:14.016431+0000","last_unstale":"2024-09-16T10:22:16.019421+0000","last_undegraded":"2024-09-16T10:22:16.019421+0000","last_fullsized":"2024-09-16T10:22:16.019421+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:01.014731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":5,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.019926+0000","last_change":"2024-09-16T10:22:17.019926+0000","last_active":"2024-09-16T10:22:01.610622+0000","last_peered":"2024-09-16T10:22:01.610622+0000","last_clean":"2024-09-16T10:22:01.610622+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-16T10:22:17.019926+0000","last_undegraded":"2024-09-16T10:22:17.019926+0000","last_fullsized":"2024-09-16T10:22:17.019926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:16:55.693372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.018593+0000","last_change":"2024-09-16T10:22:14.012517+0000","last_active":"2024-09-16T10:22:16.018593+0000","last_peered":"2024-09-16T10:22:16.018593+0000","last_clean":"2024-09-16T10:22:16.018593+0000","last_became_active":"2024-09-16T10:22:14.012267+0000","last_became_peered":"2024-09-16T10:22:14.012267+0000","last_unstale":"2024-09-16T10:22:16.018593+0000","last_undegraded":"2024-09-16T10:22:16.018593+0000","last_fullsized":"2024-09-16T10:22:16.018593+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:01:20.793597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T10:22:16.019415+0000","last_change":"2024-09-16T10:22:04.627934+0000","last_active":"2024-09-16T10:22:16.019415+0000","last_peered":"2024-09-16T10:22:16.019415+0000","last_clean":"2024-09-16T10:22:16.019415+0000","last_became_active":"2024-09-16T10:22:02.623453+0000","last_became_peered":"2024-09-16T10:22:02.623453+0000","last_unstale":"2024-09-16T10:22:16.019415+0000","last_undegraded":"2024-09-16T10:22:16.019415+0000","last_fullsized":"2024-09-16T10:22:16.019415+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:59.318421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":24,"reported_epoch":19,"state":"peering","last_fresh":"2024-09-16T10:22:17.023125+0000","last_change":"2024-09-16T10:22:17.023241+0000","last_active":"2024-09-16T10:22:16.019318+0000","last_peered":"2024-09-16T10:22:15.017867+0000","last_clean":"2024-09-16T10:22:15.017867+0000","last_became_active":"2024-09-16T10:22:02.623835+0000","last_became_peered":"2024-09-16T10:22:02.623835+0000","last_unstale":"2024-09-16T10:22:17.023125+0000","last_undegraded":"2024-09-16T10:22:17.023125+0000","last_fullsized":"2024-09-16T10:22:17.023125+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:13.467485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.022088+0000","last_change":"2024-09-16T10:22:14.016018+0000","last_active":"2024-09-16T10:22:17.022088+0000","last_peered":"2024-09-16T10:22:17.022088+0000","last_clean":"2024-09-16T10:22:17.022088+0000","last_became_active":"2024-09-16T10:22:14.015752+0000","last_became_peered":"2024-09-16T10:22:14.015752+0000","last_unstale":"2024-09-16T10:22:17.022088+0000","last_undegraded":"2024-09-16T10:22:17.022088+0000","last_fullsized":"2024-09-16T10:22:17.022088+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:51:06.878869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":159744,"data_stored":41371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31288,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340552,"statfs":{"total":96636764160,"available":96604725248,"internally_reserved":0,"allocated":167936,"data_stored":44146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:24.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:24.482 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:24.482 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:24.494 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-09-16T10:22:22.549414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":58496,"kb_used_data":352,"kb_used_omap":3,"kb_used_meta":58108,"kb_avail":188685184,"statfs":{"total":193273528320,"available":193213628416,"internally_reserved":0,"allocated":360448,"data_stored":94692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3188,"internal_metadata":59503500},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.533378"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.019264+0000","last_change":"2024-09-16T10:22:14.012743+0000","last_active":"2024-09-16T10:22:17.019264+0000","last_peered":"2024-09-16T10:22:17.019264+0000","last_clean":"2024-09-16T10:22:17.019264+0000","last_became_active":"2024-09-16T10:22:14.012441+0000","last_became_peered":"2024-09-16T10:22:14.012441+0000","last_unstale":"2024-09-16T10:22:17.019264+0000","last_undegraded":"2024-09-16T10:22:17.019264+0000","last_fullsized":"2024-09-16T10:22:17.019264+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:22:30.278434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.025234+0000","last_change":"2024-09-16T10:22:18.025234+0000","last_active":"2024-09-16T10:22:18.025234+0000","last_peered":"2024-09-16T10:22:18.025234+0000","last_clean":"2024-09-16T10:22:18.025234+0000","last_became_active":"2024-09-16T10:22:18.024876+0000","last_became_peered":"2024-09-16T10:22:18.024876+0000","last_unstale":"2024-09-16T10:22:18.025234+0000","last_undegraded":"2024-09-16T10:22:18.025234+0000","last_fullsized":"2024-09-16T10:22:18.025234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:26:46.341891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.027241+0000","last_change":"2024-09-16T10:22:18.027241+0000","last_active":"2024-09-16T10:22:18.027241+0000","last_peered":"2024-09-16T10:22:18.027241+0000","last_clean":"2024-09-16T10:22:18.027241+0000","last_became_active":"2024-09-16T10:22:18.026855+0000","last_became_peered":"2024-09-16T10:22:18.026855+0000","last_unstale":"2024-09-16T10:22:18.027241+0000","last_undegraded":"2024-09-16T10:22:18.027241+0000","last_fullsized":"2024-09-16T10:22:18.027241+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:05.028645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.025058+0000","last_change":"2024-09-16T10:22:18.025058+0000","last_active":"2024-09-16T10:22:18.025058+0000","last_peered":"2024-09-16T10:22:18.025058+0000","last_clean":"2024-09-16T10:22:18.025058+0000","last_became_active":"2024-09-16T10:22:18.024675+0000","last_became_peered":"2024-09-16T10:22:18.024675+0000","last_unstale":"2024-09-16T10:22:18.025058+0000","last_undegraded":"2024-09-16T10:22:18.025058+0000","last_fullsized":"2024-09-16T10:22:18.025058+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:13:10.798122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.024810+0000","last_change":"2024-09-16T10:22:18.024810+0000","last_active":"2024-09-16T10:22:18.024810+0000","last_peered":"2024-09-16T10:22:18.024810+0000","last_clean":"2024-09-16T10:22:18.024810+0000","last_became_active":"2024-09-16T10:22:18.024458+0000","last_became_peered":"2024-09-16T10:22:18.024458+0000","last_unstale":"2024-09-16T10:22:18.024810+0000","last_undegraded":"2024-09-16T10:22:18.024810+0000","last_fullsized":"2024-09-16T10:22:18.024810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:07:47.395793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.021328+0000","last_change":"2024-09-16T10:22:14.017879+0000","last_active":"2024-09-16T10:22:17.021328+0000","last_peered":"2024-09-16T10:22:17.021328+0000","last_clean":"2024-09-16T10:22:17.021328+0000","last_became_active":"2024-09-16T10:22:14.016590+0000","last_became_peered":"2024-09-16T10:22:14.016590+0000","last_unstale":"2024-09-16T10:22:17.021328+0000","last_undegraded":"2024-09-16T10:22:17.021328+0000","last_fullsized":"2024-09-16T10:22:17.021328+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:05:29.446766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.021354+0000","last_change":"2024-09-16T10:22:14.013439+0000","last_active":"2024-09-16T10:22:17.021354+0000","last_peered":"2024-09-16T10:22:17.021354+0000","last_clean":"2024-09-16T10:22:17.021354+0000","last_became_active":"2024-09-16T10:22:14.013167+0000","last_became_peered":"2024-09-16T10:22:14.013167+0000","last_unstale":"2024-09-16T10:22:17.021354+0000","last_undegraded":"2024-09-16T10:22:17.021354+0000","last_fullsized":"2024-09-16T10:22:17.021354+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:03:22.706300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.024006+0000","last_change":"2024-09-16T10:22:18.024006+0000","last_active":"2024-09-16T10:22:18.024006+0000","last_peered":"2024-09-16T10:22:18.024006+0000","last_clean":"2024-09-16T10:22:18.024006+0000","last_became_active":"2024-09-16T10:22:18.023698+0000","last_became_peered":"2024-09-16T10:22:18.023698+0000","last_unstale":"2024-09-16T10:22:18.024006+0000","last_undegraded":"2024-09-16T10:22:18.024006+0000","last_fullsized":"2024-09-16T10:22:18.024006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:16:23.194778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":23,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.020654+0000","last_change":"2024-09-16T10:22:04.628522+0000","last_active":"2024-09-16T10:22:17.020654+0000","last_peered":"2024-09-16T10:22:17.020654+0000","last_clean":"2024-09-16T10:22:17.020654+0000","last_became_active":"2024-09-16T10:22:02.622331+0000","last_became_peered":"2024-09-16T10:22:02.622331+0000","last_unstale":"2024-09-16T10:22:17.020654+0000","last_undegraded":"2024-09-16T10:22:17.020654+0000","last_fullsized":"2024-09-16T10:22:17.020654+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:32.499939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.026812+0000","last_change":"2024-09-16T10:22:18.026812+0000","last_active":"2024-09-16T10:22:18.026812+0000","last_peered":"2024-09-16T10:22:18.026812+0000","last_clean":"2024-09-16T10:22:18.026812+0000","last_became_active":"2024-09-16T10:22:18.026501+0000","last_became_peered":"2024-09-16T10:22:18.026501+0000","last_unstale":"2024-09-16T10:22:18.026812+0000","last_undegraded":"2024-09-16T10:22:18.026812+0000","last_fullsized":"2024-09-16T10:22:18.026812+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:04:44.835466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.024836+0000","last_change":"2024-09-16T10:22:18.024836+0000","last_active":"2024-09-16T10:22:18.024836+0000","last_peered":"2024-09-16T10:22:18.024836+0000","last_clean":"2024-09-16T10:22:18.024836+0000","last_became_active":"2024-09-16T10:22:18.024529+0000","last_became_peered":"2024-09-16T10:22:18.024529+0000","last_unstale":"2024-09-16T10:22:18.024836+0000","last_undegraded":"2024-09-16T10:22:18.024836+0000","last_fullsized":"2024-09-16T10:22:18.024836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:07:30.261928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.018962+0000","last_change":"2024-09-16T10:22:14.012931+0000","last_active":"2024-09-16T10:22:17.018962+0000","last_peered":"2024-09-16T10:22:17.018962+0000","last_clean":"2024-09-16T10:22:17.018962+0000","last_became_active":"2024-09-16T10:22:14.012491+0000","last_became_peered":"2024-09-16T10:22:14.012491+0000","last_unstale":"2024-09-16T10:22:17.018962+0000","last_undegraded":"2024-09-16T10:22:17.018962+0000","last_fullsized":"2024-09-16T10:22:17.018962+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:37:25.655709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":25,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.018828+0000","last_change":"2024-09-16T10:22:04.629301+0000","last_active":"2024-09-16T10:22:17.018828+0000","last_peered":"2024-09-16T10:22:17.018828+0000","last_clean":"2024-09-16T10:22:17.018828+0000","last_became_active":"2024-09-16T10:22:02.621803+0000","last_became_peered":"2024-09-16T10:22:02.621803+0000","last_unstale":"2024-09-16T10:22:17.018828+0000","last_undegraded":"2024-09-16T10:22:17.018828+0000","last_fullsized":"2024-09-16T10:22:17.018828+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:53:11.319895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.026711+0000","last_change":"2024-09-16T10:22:18.026711+0000","last_active":"2024-09-16T10:22:18.026711+0000","last_peered":"2024-09-16T10:22:18.026711+0000","last_clean":"2024-09-16T10:22:18.026711+0000","last_became_active":"2024-09-16T10:22:18.026355+0000","last_became_peered":"2024-09-16T10:22:18.026355+0000","last_unstale":"2024-09-16T10:22:18.026711+0000","last_undegraded":"2024-09-16T10:22:18.026711+0000","last_fullsized":"2024-09-16T10:22:18.026711+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:38.804693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":37,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.031510+0000","last_change":"2024-09-16T10:22:18.031510+0000","last_active":"2024-09-16T10:22:18.031510+0000","last_peered":"2024-09-16T10:22:18.031510+0000","last_clean":"2024-09-16T10:22:18.031510+0000","last_became_active":"2024-09-16T10:22:18.027268+0000","last_became_peered":"2024-09-16T10:22:18.027268+0000","last_unstale":"2024-09-16T10:22:18.031510+0000","last_undegraded":"2024-09-16T10:22:18.031510+0000","last_fullsized":"2024-09-16T10:22:18.031510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:34:27.261861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.027032+0000","last_change":"2024-09-16T10:22:18.027032+0000","last_active":"2024-09-16T10:22:18.027032+0000","last_peered":"2024-09-16T10:22:18.027032+0000","last_clean":"2024-09-16T10:22:18.027032+0000","last_became_active":"2024-09-16T10:22:18.026693+0000","last_became_peered":"2024-09-16T10:22:18.026693+0000","last_unstale":"2024-09-16T10:22:18.027032+0000","last_undegraded":"2024-09-16T10:22:18.027032+0000","last_fullsized":"2024-09-16T10:22:18.027032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:33:39.377427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.022754+0000","last_change":"2024-09-16T10:22:14.013058+0000","last_active":"2024-09-16T10:22:17.022754+0000","last_peered":"2024-09-16T10:22:17.022754+0000","last_clean":"2024-09-16T10:22:17.022754+0000","last_became_active":"2024-09-16T10:22:14.012790+0000","last_became_peered":"2024-09-16T10:22:14.012790+0000","last_unstale":"2024-09-16T10:22:17.022754+0000","last_undegraded":"2024-09-16T10:22:17.022754+0000","last_fullsized":"2024-09-16T10:22:17.022754+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:29.964382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":23,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.022177+0000","last_change":"2024-09-16T10:22:04.628932+0000","last_active":"2024-09-16T10:22:17.022177+0000","last_peered":"2024-09-16T10:22:17.022177+0000","last_clean":"2024-09-16T10:22:17.022177+0000","last_became_active":"2024-09-16T10:22:02.624545+0000","last_became_peered":"2024-09-16T10:22:02.624545+0000","last_unstale":"2024-09-16T10:22:17.022177+0000","last_undegraded":"2024-09-16T10:22:17.022177+0000","last_fullsized":"2024-09-16T10:22:17.022177+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:02.516293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.022504+0000","last_change":"2024-09-16T10:22:14.016687+0000","last_active":"2024-09-16T10:22:17.022504+0000","last_peered":"2024-09-16T10:22:17.022504+0000","last_clean":"2024-09-16T10:22:17.022504+0000","last_became_active":"2024-09-16T10:22:14.016431+0000","last_became_peered":"2024-09-16T10:22:14.016431+0000","last_unstale":"2024-09-16T10:22:17.022504+0000","last_undegraded":"2024-09-16T10:22:17.022504+0000","last_fullsized":"2024-09-16T10:22:17.022504+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:53:01.014731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.025021+0000","last_change":"2024-09-16T10:22:18.025021+0000","last_active":"2024-09-16T10:22:18.025021+0000","last_peered":"2024-09-16T10:22:18.025021+0000","last_clean":"2024-09-16T10:22:18.025021+0000","last_became_active":"2024-09-16T10:22:18.024594+0000","last_became_peered":"2024-09-16T10:22:18.024594+0000","last_unstale":"2024-09-16T10:22:18.025021+0000","last_undegraded":"2024-09-16T10:22:18.025021+0000","last_fullsized":"2024-09-16T10:22:18.025021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:16:55.693372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.022842+0000","last_change":"2024-09-16T10:22:14.012517+0000","last_active":"2024-09-16T10:22:17.022842+0000","last_peered":"2024-09-16T10:22:17.022842+0000","last_clean":"2024-09-16T10:22:17.022842+0000","last_became_active":"2024-09-16T10:22:14.012267+0000","last_became_peered":"2024-09-16T10:22:14.012267+0000","last_unstale":"2024-09-16T10:22:17.022842+0000","last_undegraded":"2024-09-16T10:22:17.022842+0000","last_fullsized":"2024-09-16T10:22:17.022842+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:01:20.793597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":23,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.022593+0000","last_change":"2024-09-16T10:22:04.627934+0000","last_active":"2024-09-16T10:22:17.022593+0000","last_peered":"2024-09-16T10:22:17.022593+0000","last_clean":"2024-09-16T10:22:17.022593+0000","last_became_active":"2024-09-16T10:22:02.623453+0000","last_became_peered":"2024-09-16T10:22:02.623453+0000","last_unstale":"2024-09-16T10:22:17.022593+0000","last_undegraded":"2024-09-16T10:22:17.022593+0000","last_fullsized":"2024-09-16T10:22:17.022593+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:09:59.318421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":32,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-16T10:22:18.027391+0000","last_change":"2024-09-16T10:22:18.027391+0000","last_active":"2024-09-16T10:22:18.027391+0000","last_peered":"2024-09-16T10:22:18.027391+0000","last_clean":"2024-09-16T10:22:18.027391+0000","last_became_active":"2024-09-16T10:22:18.027067+0000","last_became_peered":"2024-09-16T10:22:18.027067+0000","last_unstale":"2024-09-16T10:22:18.027391+0000","last_undegraded":"2024-09-16T10:22:18.027391+0000","last_fullsized":"2024-09-16T10:22:18.027391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:46:30.930956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"19'1","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T10:22:17.092699+0000","last_change":"2024-09-16T10:22:14.016018+0000","last_active":"2024-09-16T10:22:17.092699+0000","last_peered":"2024-09-16T10:22:17.092699+0000","last_clean":"2024-09-16T10:22:17.092699+0000","last_became_active":"2024-09-16T10:22:14.015752+0000","last_became_peered":"2024-09-16T10:22:14.015752+0000","last_unstale":"2024-09-16T10:22:17.092699+0000","last_undegraded":"2024-09-16T10:22:17.092699+0000","last_fullsized":"2024-09-16T10:22:17.092699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:51:06.878869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":47346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738376,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31292,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340548,"statfs":{"total":96636764160,"available":96604721152,"internally_reserved":0,"allocated":180224,"data_stored":47346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:24.496 INFO:tasks.rep_lost_unfound_delete.ceph_manager:recovered! 2024-09-16T10:22:24.496 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_1 /etc/fstab 2024-09-16T10:22:24.496 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_1 /etc/fstab 2024-09-16T10:22:24.547 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_1 /etc/fstab 2024-09-16T10:22:24.547 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_1 /etc/fstab 2024-09-16T10:22:24.598 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_1 2024-09-16T10:22:24.598 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_1 2024-09-16T10:22:24.647 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_2 /etc/fstab 2024-09-16T10:22:24.648 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_2 /etc/fstab 2024-09-16T10:22:24.699 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_2 /etc/fstab 2024-09-16T10:22:24.699 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_2 /etc/fstab 2024-09-16T10:22:24.749 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_2 2024-09-16T10:22:24.750 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_2 2024-09-16T10:22:24.799 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_3 /etc/fstab 2024-09-16T10:22:24.799 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_3 /etc/fstab 2024-09-16T10:22:24.850 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_3 /etc/fstab 2024-09-16T10:22:24.885 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_3 /etc/fstab 2024-09-16T10:22:24.936 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_3 2024-09-16T10:22:24.937 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_3 2024-09-16T10:22:24.986 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_4 /etc/fstab 2024-09-16T10:22:24.986 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_4 /etc/fstab 2024-09-16T10:22:25.036 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_4 /etc/fstab 2024-09-16T10:22:25.037 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_4 /etc/fstab 2024-09-16T10:22:25.088 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_4 2024-09-16T10:22:25.088 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_4 2024-09-16T10:22:25.138 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_5 /etc/fstab 2024-09-16T10:22:25.138 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_5 /etc/fstab 2024-09-16T10:22:25.188 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_5 /etc/fstab 2024-09-16T10:22:25.189 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_5 /etc/fstab 2024-09-16T10:22:25.240 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_5 2024-09-16T10:22:25.240 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_5 2024-09-16T10:22:25.289 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_6 /etc/fstab 2024-09-16T10:22:25.289 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_6 /etc/fstab 2024-09-16T10:22:25.340 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_6 /etc/fstab 2024-09-16T10:22:25.340 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_6 /etc/fstab 2024-09-16T10:22:25.391 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_6 2024-09-16T10:22:25.391 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_6 2024-09-16T10:22:25.440 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_7 /etc/fstab 2024-09-16T10:22:25.440 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_7 /etc/fstab 2024-09-16T10:22:25.491 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_7 /etc/fstab 2024-09-16T10:22:25.491 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_7 /etc/fstab 2024-09-16T10:22:25.541 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_7 2024-09-16T10:22:25.541 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_7 2024-09-16T10:22:25.591 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_8 /etc/fstab 2024-09-16T10:22:25.591 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_8 /etc/fstab 2024-09-16T10:22:25.642 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_8 /etc/fstab 2024-09-16T10:22:25.642 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_8 /etc/fstab 2024-09-16T10:22:25.693 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_8 2024-09-16T10:22:25.693 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_8 2024-09-16T10:22:25.743 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_9 /etc/fstab 2024-09-16T10:22:25.743 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_9 /etc/fstab 2024-09-16T10:22:25.793 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_9 /etc/fstab 2024-09-16T10:22:25.794 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_9 /etc/fstab 2024-09-16T10:22:25.844 INFO:tasks.util.rados:rados -p unfounddel_pool rm existed_9 2024-09-16T10:22:25.844 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool rm existed_9 2024-09-16T10:22:25.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-09-16T10:22:26.016 INFO:teuthology.orchestra.run.smithi050.stdout:{} 2024-09-16T10:22:26.017 INFO:teuthology.orchestra.run.smithi050.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-09-16T10:22:26.027 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T10:22:26.027 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:22:26.065 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T10:22:26.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-09-16T10:22:26.395 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-16T10:22:26.394+0000 7f0322f76640 -1 mon.a@0(leader).osd e20 definitely_dead 0 2024-09-16T10:22:27.049 INFO:teuthology.orchestra.run.smithi050.stderr:marked down osd.0. 2024-09-16T10:22:27.067 INFO:tasks.util.rados:rados -p unfounddel_pool put new_1 /etc/fstab 2024-09-16T10:22:27.067 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_1 /etc/fstab 2024-09-16T10:22:28.061 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_1 /etc/fstab 2024-09-16T10:22:28.061 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_1 /etc/fstab 2024-09-16T10:22:28.119 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_1 /etc/fstab 2024-09-16T10:22:28.119 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_1 /etc/fstab 2024-09-16T10:22:28.172 INFO:tasks.util.rados:rados -p unfounddel_pool put new_2 /etc/fstab 2024-09-16T10:22:28.172 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_2 /etc/fstab 2024-09-16T10:22:28.224 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_2 /etc/fstab 2024-09-16T10:22:28.224 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_2 /etc/fstab 2024-09-16T10:22:28.274 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_2 /etc/fstab 2024-09-16T10:22:28.275 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_2 /etc/fstab 2024-09-16T10:22:28.325 INFO:tasks.util.rados:rados -p unfounddel_pool put new_3 /etc/fstab 2024-09-16T10:22:28.325 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_3 /etc/fstab 2024-09-16T10:22:28.376 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_3 /etc/fstab 2024-09-16T10:22:28.376 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_3 /etc/fstab 2024-09-16T10:22:28.426 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_3 /etc/fstab 2024-09-16T10:22:28.426 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_3 /etc/fstab 2024-09-16T10:22:28.477 INFO:tasks.util.rados:rados -p unfounddel_pool put new_4 /etc/fstab 2024-09-16T10:22:28.477 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_4 /etc/fstab 2024-09-16T10:22:28.527 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_4 /etc/fstab 2024-09-16T10:22:28.528 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_4 /etc/fstab 2024-09-16T10:22:28.578 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_4 /etc/fstab 2024-09-16T10:22:28.579 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_4 /etc/fstab 2024-09-16T10:22:28.630 INFO:tasks.util.rados:rados -p unfounddel_pool put new_5 /etc/fstab 2024-09-16T10:22:28.630 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_5 /etc/fstab 2024-09-16T10:22:28.681 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_5 /etc/fstab 2024-09-16T10:22:28.681 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_5 /etc/fstab 2024-09-16T10:22:28.732 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_5 /etc/fstab 2024-09-16T10:22:28.732 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_5 /etc/fstab 2024-09-16T10:22:28.783 INFO:tasks.util.rados:rados -p unfounddel_pool put new_6 /etc/fstab 2024-09-16T10:22:28.783 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_6 /etc/fstab 2024-09-16T10:22:28.833 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_6 /etc/fstab 2024-09-16T10:22:28.834 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_6 /etc/fstab 2024-09-16T10:22:28.884 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_6 /etc/fstab 2024-09-16T10:22:28.885 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_6 /etc/fstab 2024-09-16T10:22:28.935 INFO:tasks.util.rados:rados -p unfounddel_pool put new_7 /etc/fstab 2024-09-16T10:22:28.936 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_7 /etc/fstab 2024-09-16T10:22:28.986 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_7 /etc/fstab 2024-09-16T10:22:28.986 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_7 /etc/fstab 2024-09-16T10:22:29.037 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_7 /etc/fstab 2024-09-16T10:22:29.037 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_7 /etc/fstab 2024-09-16T10:22:29.088 INFO:tasks.util.rados:rados -p unfounddel_pool put new_8 /etc/fstab 2024-09-16T10:22:29.089 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_8 /etc/fstab 2024-09-16T10:22:29.139 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_8 /etc/fstab 2024-09-16T10:22:29.139 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_8 /etc/fstab 2024-09-16T10:22:29.190 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_8 /etc/fstab 2024-09-16T10:22:29.190 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_8 /etc/fstab 2024-09-16T10:22:29.240 INFO:tasks.util.rados:rados -p unfounddel_pool put new_9 /etc/fstab 2024-09-16T10:22:29.241 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put new_9 /etc/fstab 2024-09-16T10:22:29.291 INFO:tasks.util.rados:rados -p unfounddel_pool put existed_9 /etc/fstab 2024-09-16T10:22:29.291 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existed_9 /etc/fstab 2024-09-16T10:22:29.341 INFO:tasks.util.rados:rados -p unfounddel_pool put existing_9 /etc/fstab 2024-09-16T10:22:29.342 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool put existing_9 /etc/fstab 2024-09-16T10:22:29.393 INFO:tasks.rep_lost_unfound_delete:osd.0 command_args is foo 2024-09-16T10:22:29.393 INFO:tasks.rep_lost_unfound_delete:() 2024-09-16T10:22:29.393 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T10:22:29.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 --osd-recovery-delay-start 1000 2024-09-16T10:22:29.395 INFO:tasks.ceph.osd.0:Started 2024-09-16T10:22:29.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T10:22:29.496 INFO:teuthology.orchestra.run.smithi050.stderr:no valid command found; 10 closest matches: 2024-09-16T10:22:29.497 INFO:teuthology.orchestra.run.smithi050.stderr:0 2024-09-16T10:22:29.497 INFO:teuthology.orchestra.run.smithi050.stderr:1 2024-09-16T10:22:29.497 INFO:teuthology.orchestra.run.smithi050.stderr:2 2024-09-16T10:22:29.497 INFO:teuthology.orchestra.run.smithi050.stderr:abort 2024-09-16T10:22:29.497 INFO:teuthology.orchestra.run.smithi050.stderr:assert 2024-09-16T10:22:29.497 INFO:teuthology.orchestra.run.smithi050.stderr:config diff 2024-09-16T10:22:29.497 INFO:teuthology.orchestra.run.smithi050.stderr:config diff get 2024-09-16T10:22:29.497 INFO:teuthology.orchestra.run.smithi050.stderr:config get 2024-09-16T10:22:29.497 INFO:teuthology.orchestra.run.smithi050.stderr:config help [] 2024-09-16T10:22:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:config set ... 2024-09-16T10:22:29.498 INFO:teuthology.orchestra.run.smithi050.stderr:admin_socket: invalid command 2024-09-16T10:22:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:22:29.499 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-16T10:22:29.990 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:29.989+0000 7ff6eae1d640 -1 Falling back to public interface 2024-09-16T10:22:33.491 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:33.489+0000 7ff6eae1d640 -1 osd.0 20 log_to_monitors true 2024-09-16T10:22:34.117 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:34.116+0000 7ff6de62a640 -1 osd.0 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:22:34.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-16T10:22:34.621 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:22:34.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-09-16T10:22:34.923 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-16T10:22:34.921+0000 7f0322f76640 -1 mon.a@0(leader).osd e23 definitely_dead 0 2024-09-16T10:22:34.923 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0 is already in. 2024-09-16T10:22:34.935 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for osd.0 to be up 2024-09-16T10:22:34.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:22:35.226 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:35.226 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":24,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:22:35.111556+0000","last_up_change":"2024-09-16T10:22:35.111556+0000","last_in_change":"2024-09-16T10:22:17.015458+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T10:22:00.707491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8700000047683716,"score_stable":1.8700000047683716,"optimal_score":0.67000001668930054,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-16T10:22:12.124014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.8700000047683716,"score_stable":1.8700000047683716,"optimal_score":0.67000001668930054,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":20,"up_from":24,"up_thru":19,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2470025375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":2470025375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":2470025375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":2470025375}]},"public_addr":"172.21.15.50:6804/2470025375","cluster_addr":"172.21.15.50:6805/2470025375","heartbeat_back_addr":"172.21.15.50:6807/2470025375","heartbeat_front_addr":"172.21.15.50:6806/2470025375","state":["exists","up"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":202723672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":202723672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":202723672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":202723672}]},"public_addr":"172.21.15.50:6800/202723672","cluster_addr":"172.21.15.50:6801/202723672","heartbeat_back_addr":"172.21.15.50:6803/202723672","heartbeat_front_addr":"172.21.15.50:6802/202723672","state":["exists","up"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":15,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":2584589368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":2584589368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":2584589368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":2584589368}]},"public_addr":"172.21.15.50:6808/2584589368","cluster_addr":"172.21.15.50:6809/2584589368","heartbeat_back_addr":"172.21.15.50:6811/2584589368","heartbeat_front_addr":"172.21.15.50:6810/2584589368","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.244822+0000","dead_epoch":21},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.213317+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T10:21:58.228331+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:22:35.239 INFO:tasks.rep_lost_unfound_delete.ceph_manager:osd.0 is up 2024-09-16T10:22:35.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T10:22:35.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T10:22:35.376 INFO:teuthology.orchestra.run.smithi050.stdout:103079215106 2024-09-16T10:22:35.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:22:35.377 INFO:teuthology.orchestra.run.smithi050.stdout:34359738380 2024-09-16T10:22:35.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:22:35.704 INFO:teuthology.orchestra.run.smithi050.stdout:34359738379 2024-09-16T10:22:35.707 INFO:teuthology.orchestra.run.smithi050.stdout:1 2024-09-16T10:22:35.717 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738380 got 34359738379 for osd.1 2024-09-16T10:22:35.720 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 103079215106 got 1 for osd.0 2024-09-16T10:22:36.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:22:36.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:22:37.019 INFO:teuthology.orchestra.run.smithi050.stdout:34359738379 2024-09-16T10:22:37.031 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738380 got 34359738379 for osd.1 2024-09-16T10:22:37.052 INFO:teuthology.orchestra.run.smithi050.stdout:1 2024-09-16T10:22:37.065 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 103079215106 got 1 for osd.0 2024-09-16T10:22:38.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T10:22:38.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:22:38.332 INFO:teuthology.orchestra.run.smithi050.stdout:34359738381 2024-09-16T10:22:38.345 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 34359738380 got 34359738381 for osd.1 2024-09-16T10:22:38.345 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:38.370 INFO:teuthology.orchestra.run.smithi050.stdout:103079215106 2024-09-16T10:22:38.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 103079215106 got 103079215106 for osd.0 2024-09-16T10:22:38.384 DEBUG:teuthology.parallel:result is None 2024-09-16T10:22:38.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting till active 2024-09-16T10:22:38.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:38.703 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:38.703 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:38.715 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-09-16T10:22:38.551985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11611,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51,"ondisk_log_size":51,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":59112,"kb_used_data":528,"kb_used_omap":5,"kb_used_meta":58554,"kb_avail":188684568,"statfs":{"total":193273528320,"available":193212997632,"internally_reserved":0,"allocated":540672,"data_stored":124839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6034,"internal_metadata":59959406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1932,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-30,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7,"num_write_kb":-7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001026"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.126090+0000","last_change":"2024-09-16T10:22:34.113085+0000","last_active":"2024-09-16T10:22:34.113085+0000","last_peered":"2024-09-16T10:22:14.012743+0000","last_clean":"2024-09-16T10:22:14.012743+0000","last_became_active":"2024-09-16T10:22:14.012441+0000","last_became_peered":"2024-09-16T10:22:14.012441+0000","last_unstale":"2024-09-16T10:22:35.126090+0000","last_undegraded":"2024-09-16T10:22:35.126090+0000","last_fullsized":"2024-09-16T10:22:35.126090+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:15:18.226945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":18,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.123193+0000","last_change":"2024-09-16T10:22:35.121619+0000","last_active":"2024-09-16T10:22:34.117273+0000","last_peered":"2024-09-16T10:22:25.033354+0000","last_clean":"2024-09-16T10:22:25.033354+0000","last_became_active":"2024-09-16T10:22:18.024876+0000","last_became_peered":"2024-09-16T10:22:18.024876+0000","last_unstale":"2024-09-16T10:22:35.123193+0000","last_undegraded":"2024-09-16T10:22:35.123193+0000","last_fullsized":"2024-09-16T10:22:35.123193+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:35.348628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":36,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.121046+0000","last_change":"2024-09-16T10:22:35.121046+0000","last_active":"2024-09-16T10:22:35.120899+0000","last_peered":"2024-09-16T10:22:34.110394+0000","last_clean":"2024-09-16T10:22:25.588129+0000","last_became_active":"2024-09-16T10:22:28.053606+0000","last_became_peered":"2024-09-16T10:22:28.053606+0000","last_unstale":"2024-09-16T10:22:35.121046+0000","last_undegraded":"2024-09-16T10:22:35.121046+0000","last_fullsized":"2024-09-16T10:22:35.121046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:41:52.629551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.122283+0000","last_change":"2024-09-16T10:22:35.117828+0000","last_active":"2024-09-16T10:22:34.117232+0000","last_peered":"2024-09-16T10:22:18.025058+0000","last_clean":"2024-09-16T10:22:18.025058+0000","last_became_active":"2024-09-16T10:22:18.024675+0000","last_became_peered":"2024-09-16T10:22:18.024675+0000","last_unstale":"2024-09-16T10:22:35.122283+0000","last_undegraded":"2024-09-16T10:22:35.122283+0000","last_fullsized":"2024-09-16T10:22:35.122283+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:46:53.440821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"20'3","reported_seq":28,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.123233+0000","last_change":"2024-09-16T10:22:35.117102+0000","last_active":"2024-09-16T10:22:34.117198+0000","last_peered":"2024-09-16T10:22:24.933330+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:22:18.024458+0000","last_became_peered":"2024-09-16T10:22:18.024458+0000","last_unstale":"2024-09-16T10:22:35.123233+0000","last_undegraded":"2024-09-16T10:22:35.123233+0000","last_fullsized":"2024-09-16T10:22:35.123233+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:12:54.968146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":27,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.119459+0000","last_change":"2024-09-16T10:22:35.119459+0000","last_active":"2024-09-16T10:22:35.119317+0000","last_peered":"2024-09-16T10:22:34.110162+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:22:28.052921+0000","last_became_peered":"2024-09-16T10:22:28.052921+0000","last_unstale":"2024-09-16T10:22:35.119459+0000","last_undegraded":"2024-09-16T10:22:35.119459+0000","last_fullsized":"2024-09-16T10:22:35.119459+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:31.469343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"22'4","reported_seq":28,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.125035+0000","last_change":"2024-09-16T10:22:35.125035+0000","last_active":"2024-09-16T10:22:35.124912+0000","last_peered":"2024-09-16T10:22:34.110702+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:22:28.055141+0000","last_became_peered":"2024-09-16T10:22:28.055141+0000","last_unstale":"2024-09-16T10:22:35.125035+0000","last_undegraded":"2024-09-16T10:22:35.125035+0000","last_fullsized":"2024-09-16T10:22:35.125035+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:04.731476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.121519+0000","last_change":"2024-09-16T10:22:35.117228+0000","last_active":"2024-09-16T10:22:34.117101+0000","last_peered":"2024-09-16T10:22:18.024006+0000","last_clean":"2024-09-16T10:22:18.024006+0000","last_became_active":"2024-09-16T10:22:18.023698+0000","last_became_peered":"2024-09-16T10:22:18.023698+0000","last_unstale":"2024-09-16T10:22:35.121519+0000","last_undegraded":"2024-09-16T10:22:35.121519+0000","last_fullsized":"2024-09-16T10:22:35.121519+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:32:45.244369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.117613+0000","last_change":"2024-09-16T10:22:35.117763+0000","last_active":"2024-09-16T10:22:35.117613+0000","last_peered":"2024-09-16T10:22:34.110484+0000","last_clean":"2024-09-16T10:22:18.022590+0000","last_became_active":"2024-09-16T10:22:28.054632+0000","last_became_peered":"2024-09-16T10:22:28.054632+0000","last_unstale":"2024-09-16T10:22:35.117613+0000","last_undegraded":"2024-09-16T10:22:35.117613+0000","last_fullsized":"2024-09-16T10:22:35.117613+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:10:35.441411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'5","reported_seq":38,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.119374+0000","last_change":"2024-09-16T10:22:35.119374+0000","last_active":"2024-09-16T10:22:35.119239+0000","last_peered":"2024-09-16T10:22:34.110817+0000","last_clean":"2024-09-16T10:22:25.638957+0000","last_became_active":"2024-09-16T10:22:28.054256+0000","last_became_peered":"2024-09-16T10:22:28.054256+0000","last_unstale":"2024-09-16T10:22:35.119374+0000","last_undegraded":"2024-09-16T10:22:35.119374+0000","last_fullsized":"2024-09-16T10:22:35.119374+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:52.624679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.127874+0000","last_change":"2024-09-16T10:22:35.116017+0000","last_active":"2024-09-16T10:22:34.117132+0000","last_peered":"2024-09-16T10:22:18.024836+0000","last_clean":"2024-09-16T10:22:18.024836+0000","last_became_active":"2024-09-16T10:22:18.024529+0000","last_became_peered":"2024-09-16T10:22:18.024529+0000","last_unstale":"2024-09-16T10:22:35.127874+0000","last_undegraded":"2024-09-16T10:22:35.127874+0000","last_fullsized":"2024-09-16T10:22:35.127874+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:44:57.013657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"20'4","reported_seq":20,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.122919+0000","last_change":"2024-09-16T10:22:35.116393+0000","last_active":"2024-09-16T10:22:34.117176+0000","last_peered":"2024-09-16T10:22:25.841098+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:22:14.012491+0000","last_became_peered":"2024-09-16T10:22:14.012491+0000","last_unstale":"2024-09-16T10:22:35.122919+0000","last_undegraded":"2024-09-16T10:22:35.122919+0000","last_fullsized":"2024-09-16T10:22:35.122919+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:29:12.945637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":24,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.124152+0000","last_change":"2024-09-16T10:22:34.113130+0000","last_active":"2024-09-16T10:22:34.113130+0000","last_peered":"2024-09-16T10:22:10.999393+0000","last_clean":"2024-09-16T10:22:10.999393+0000","last_became_active":"2024-09-16T10:22:02.621803+0000","last_became_peered":"2024-09-16T10:22:02.621803+0000","last_unstale":"2024-09-16T10:22:35.124152+0000","last_undegraded":"2024-09-16T10:22:35.124152+0000","last_fullsized":"2024-09-16T10:22:35.124152+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:26:44.299691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'3","reported_seq":27,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.124106+0000","last_change":"2024-09-16T10:22:35.124106+0000","last_active":"2024-09-16T10:22:35.123958+0000","last_peered":"2024-09-16T10:22:34.110345+0000","last_clean":"2024-09-16T10:22:24.847222+0000","last_became_active":"2024-09-16T10:22:28.054310+0000","last_became_peered":"2024-09-16T10:22:28.054310+0000","last_unstale":"2024-09-16T10:22:35.124106+0000","last_undegraded":"2024-09-16T10:22:35.124106+0000","last_fullsized":"2024-09-16T10:22:35.124106+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:52.381096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":47,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.119872+0000","last_change":"2024-09-16T10:22:35.120131+0000","last_active":"2024-09-16T10:22:35.119872+0000","last_peered":"2024-09-16T10:22:34.110616+0000","last_clean":"2024-09-16T10:22:18.032042+0000","last_became_active":"2024-09-16T10:22:28.053555+0000","last_became_peered":"2024-09-16T10:22:28.053555+0000","last_unstale":"2024-09-16T10:22:35.119872+0000","last_undegraded":"2024-09-16T10:22:35.119872+0000","last_fullsized":"2024-09-16T10:22:35.119872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:59.769631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":28,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.117139+0000","last_change":"2024-09-16T10:22:35.117139+0000","last_active":"2024-09-16T10:22:35.117000+0000","last_peered":"2024-09-16T10:22:34.110442+0000","last_clean":"2024-09-16T10:22:25.286289+0000","last_became_active":"2024-09-16T10:22:28.054002+0000","last_became_peered":"2024-09-16T10:22:28.054002+0000","last_unstale":"2024-09-16T10:22:35.117139+0000","last_undegraded":"2024-09-16T10:22:35.117139+0000","last_fullsized":"2024-09-16T10:22:35.117139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:05.474700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"22'1","reported_seq":24,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.122300+0000","last_change":"2024-09-16T10:22:35.122436+0000","last_active":"2024-09-16T10:22:35.122300+0000","last_peered":"2024-09-16T10:22:34.110623+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:22:28.054187+0000","last_became_peered":"2024-09-16T10:22:28.054187+0000","last_unstale":"2024-09-16T10:22:35.122300+0000","last_undegraded":"2024-09-16T10:22:35.122300+0000","last_fullsized":"2024-09-16T10:22:35.122300+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:15:45.607374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.119379+0000","last_change":"2024-09-16T10:22:35.119499+0000","last_active":"2024-09-16T10:22:35.119379+0000","last_peered":"2024-09-16T10:22:34.110250+0000","last_clean":"2024-09-16T10:22:18.022743+0000","last_became_active":"2024-09-16T10:22:28.054731+0000","last_became_peered":"2024-09-16T10:22:28.054731+0000","last_unstale":"2024-09-16T10:22:35.119379+0000","last_undegraded":"2024-09-16T10:22:35.119379+0000","last_fullsized":"2024-09-16T10:22:35.119379+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:42.551889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'5","reported_seq":29,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.120674+0000","last_change":"2024-09-16T10:22:35.120674+0000","last_active":"2024-09-16T10:22:35.120541+0000","last_peered":"2024-09-16T10:22:34.110329+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:22:28.053154+0000","last_became_peered":"2024-09-16T10:22:28.053154+0000","last_unstale":"2024-09-16T10:22:35.120674+0000","last_undegraded":"2024-09-16T10:22:35.120674+0000","last_fullsized":"2024-09-16T10:22:35.120674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:14:43.529224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.122840+0000","last_change":"2024-09-16T10:22:35.119032+0000","last_active":"2024-09-16T10:22:34.117251+0000","last_peered":"2024-09-16T10:22:18.025021+0000","last_clean":"2024-09-16T10:22:18.025021+0000","last_became_active":"2024-09-16T10:22:18.024594+0000","last_became_peered":"2024-09-16T10:22:18.024594+0000","last_unstale":"2024-09-16T10:22:35.122840+0000","last_undegraded":"2024-09-16T10:22:35.122840+0000","last_fullsized":"2024-09-16T10:22:35.122840+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:06:30.433546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"22'2","reported_seq":25,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.121585+0000","last_change":"2024-09-16T10:22:35.121729+0000","last_active":"2024-09-16T10:22:35.121585+0000","last_peered":"2024-09-16T10:22:34.110550+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:22:28.054302+0000","last_became_peered":"2024-09-16T10:22:28.054302+0000","last_unstale":"2024-09-16T10:22:35.121585+0000","last_undegraded":"2024-09-16T10:22:35.121585+0000","last_fullsized":"2024-09-16T10:22:35.121585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:08.537911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.121174+0000","last_change":"2024-09-16T10:22:35.121294+0000","last_active":"2024-09-16T10:22:35.121174+0000","last_peered":"2024-09-16T10:22:34.110602+0000","last_clean":"2024-09-16T10:22:18.022823+0000","last_became_active":"2024-09-16T10:22:28.053691+0000","last_became_peered":"2024-09-16T10:22:28.053691+0000","last_unstale":"2024-09-16T10:22:35.121174+0000","last_undegraded":"2024-09-16T10:22:35.121174+0000","last_fullsized":"2024-09-16T10:22:35.121174+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:37.408921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":42,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.122893+0000","last_change":"2024-09-16T10:22:35.123011+0000","last_active":"2024-09-16T10:22:35.122893+0000","last_peered":"2024-09-16T10:22:34.110760+0000","last_clean":"2024-09-16T10:22:18.029366+0000","last_became_active":"2024-09-16T10:22:28.055395+0000","last_became_peered":"2024-09-16T10:22:28.055395+0000","last_unstale":"2024-09-16T10:22:35.122893+0000","last_undegraded":"2024-09-16T10:22:35.122893+0000","last_fullsized":"2024-09-16T10:22:35.122893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:36.080295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"22'9","reported_seq":33,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.118934+0000","last_change":"2024-09-16T10:22:35.118934+0000","last_active":"2024-09-16T10:22:35.118792+0000","last_peered":"2024-09-16T10:22:34.110133+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:22:28.054066+0000","last_became_peered":"2024-09-16T10:22:28.054066+0000","last_unstale":"2024-09-16T10:22:35.118934+0000","last_undegraded":"2024-09-16T10:22:35.118934+0000","last_fullsized":"2024-09-16T10:22:35.118934+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:27.220732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":11592,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":18354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48,"ondisk_log_size":48,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":73146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215106,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31772,"kb_used_data":212,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94340068,"statfs":{"total":96636764160,"available":96604229632,"internally_reserved":0,"allocated":217088,"data_stored":51693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":32304808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":4830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:38.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:22:39.062 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:39.063 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":47,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":25,"num_osds":3,"num_up_osds":2,"osd_up_since":1726482155,"num_in_osds":2,"osd_in_since":1726482137,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":24}],"num_pgs":24,"num_pools":2,"num_objects":26,"data_bytes":11611,"bytes_used":60530688,"bytes_avail":193212997632,"bytes_total":193273528320,"inactive_pgs_ratio":1,"write_bytes_sec":4095,"read_op_per_sec":0,"write_op_per_sec":4},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-16T10:22:18.548879+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:22:39.075 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '3f765c65-edd5-4e6d-930a-33ce25cc666f', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 47, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 25, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726482155, 'num_in_osds': 2, 'osd_in_since': 1726482137, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'peering', 'count': 24}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 26, 'data_bytes': 11611, 'bytes_used': 60530688, 'bytes_avail': 193212997632, 'bytes_total': 193273528320, 'inactive_pgs_ratio': 1, 'write_bytes_sec': 4095, 'read_op_per_sec': 0, 'write_op_per_sec': 4}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2024-09-16T10:22:18.548879+0000', 'services': {'mgr': {'daemons': {'summary': '', 'x': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'mon': {'daemons': {'summary': '', 'a': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'b': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'c': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '1': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-16T10:22:42.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:42.369 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:42.369 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:42.381 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-09-16T10:22:40.552202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13543,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":59164,"kb_used_data":568,"kb_used_omap":5,"kb_used_meta":58554,"kb_avail":188684516,"statfs":{"total":193273528320,"available":193212944384,"internally_reserved":0,"allocated":581632,"data_stored":140315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6034,"internal_metadata":59959406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":-23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.443925"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.127471+0000","last_change":"2024-09-16T10:22:36.127471+0000","last_active":"2024-09-16T10:22:36.127471+0000","last_peered":"2024-09-16T10:22:36.127471+0000","last_clean":"2024-09-16T10:22:36.127471+0000","last_became_active":"2024-09-16T10:22:36.126974+0000","last_became_peered":"2024-09-16T10:22:36.126974+0000","last_unstale":"2024-09-16T10:22:36.127471+0000","last_undegraded":"2024-09-16T10:22:36.127471+0000","last_fullsized":"2024-09-16T10:22:36.127471+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:21.922974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'4","reported_seq":24,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.127642+0000","last_change":"2024-09-16T10:22:36.127819+0000","last_active":"2024-09-16T10:22:36.127642+0000","last_peered":"2024-09-16T10:22:36.127642+0000","last_clean":"2024-09-16T10:22:25.033354+0000","last_became_active":"2024-09-16T10:22:36.127559+0000","last_became_peered":"2024-09-16T10:22:36.127559+0000","last_unstale":"2024-09-16T10:22:36.127642+0000","last_undegraded":"2024-09-16T10:22:36.119252+0000","last_fullsized":"2024-09-16T10:22:36.127642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:37:20.704128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":36,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.121046+0000","last_change":"2024-09-16T10:22:35.121046+0000","last_active":"2024-09-16T10:22:35.120899+0000","last_peered":"2024-09-16T10:22:34.110394+0000","last_clean":"2024-09-16T10:22:25.588129+0000","last_became_active":"2024-09-16T10:22:28.053606+0000","last_became_peered":"2024-09-16T10:22:28.053606+0000","last_unstale":"2024-09-16T10:22:35.121046+0000","last_undegraded":"2024-09-16T10:22:35.121046+0000","last_fullsized":"2024-09-16T10:22:35.121046+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:41:52.629551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.127664+0000","last_change":"2024-09-16T10:22:36.127664+0000","last_active":"2024-09-16T10:22:36.127664+0000","last_peered":"2024-09-16T10:22:36.127664+0000","last_clean":"2024-09-16T10:22:36.127664+0000","last_became_active":"2024-09-16T10:22:36.127167+0000","last_became_peered":"2024-09-16T10:22:36.127167+0000","last_unstale":"2024-09-16T10:22:36.127664+0000","last_undegraded":"2024-09-16T10:22:36.127664+0000","last_fullsized":"2024-09-16T10:22:36.127664+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:27.016711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"22'5","reported_seq":35,"reported_epoch":25,"state":"active+recovering+degraded","last_fresh":"2024-09-16T10:22:36.122205+0000","last_change":"2024-09-16T10:22:36.127068+0000","last_active":"2024-09-16T10:22:36.122205+0000","last_peered":"2024-09-16T10:22:36.122205+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:22:36.121736+0000","last_became_peered":"2024-09-16T10:22:36.121736+0000","last_unstale":"2024-09-16T10:22:36.122205+0000","last_undegraded":"2024-09-16T10:22:36.117716+0000","last_fullsized":"2024-09-16T10:22:36.122205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:57:32.787085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":27,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.119459+0000","last_change":"2024-09-16T10:22:35.119459+0000","last_active":"2024-09-16T10:22:35.119317+0000","last_peered":"2024-09-16T10:22:34.110162+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:22:28.052921+0000","last_became_peered":"2024-09-16T10:22:28.052921+0000","last_unstale":"2024-09-16T10:22:35.119459+0000","last_undegraded":"2024-09-16T10:22:35.119459+0000","last_fullsized":"2024-09-16T10:22:35.119459+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:31.469343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"22'4","reported_seq":28,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.125035+0000","last_change":"2024-09-16T10:22:35.125035+0000","last_active":"2024-09-16T10:22:35.124912+0000","last_peered":"2024-09-16T10:22:34.110702+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:22:28.055141+0000","last_became_peered":"2024-09-16T10:22:28.055141+0000","last_unstale":"2024-09-16T10:22:35.125035+0000","last_undegraded":"2024-09-16T10:22:35.125035+0000","last_fullsized":"2024-09-16T10:22:35.125035+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:04.731476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'1","reported_seq":31,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.127009+0000","last_change":"2024-09-16T10:22:36.127196+0000","last_active":"2024-09-16T10:22:36.127009+0000","last_peered":"2024-09-16T10:22:36.127009+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:22:36.126924+0000","last_became_peered":"2024-09-16T10:22:36.126924+0000","last_unstale":"2024-09-16T10:22:36.127009+0000","last_undegraded":"2024-09-16T10:22:36.115681+0000","last_fullsized":"2024-09-16T10:22:36.127009+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:52.323025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.117613+0000","last_change":"2024-09-16T10:22:35.117763+0000","last_active":"2024-09-16T10:22:35.117613+0000","last_peered":"2024-09-16T10:22:34.110484+0000","last_clean":"2024-09-16T10:22:18.022590+0000","last_became_active":"2024-09-16T10:22:28.054632+0000","last_became_peered":"2024-09-16T10:22:28.054632+0000","last_unstale":"2024-09-16T10:22:35.117613+0000","last_undegraded":"2024-09-16T10:22:35.117613+0000","last_fullsized":"2024-09-16T10:22:35.117613+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:10:35.441411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'5","reported_seq":38,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.119374+0000","last_change":"2024-09-16T10:22:35.119374+0000","last_active":"2024-09-16T10:22:35.119239+0000","last_peered":"2024-09-16T10:22:34.110817+0000","last_clean":"2024-09-16T10:22:25.638957+0000","last_became_active":"2024-09-16T10:22:28.054256+0000","last_became_peered":"2024-09-16T10:22:28.054256+0000","last_unstale":"2024-09-16T10:22:35.119374+0000","last_undegraded":"2024-09-16T10:22:35.119374+0000","last_fullsized":"2024-09-16T10:22:35.119374+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:52.624679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.125077+0000","last_change":"2024-09-16T10:22:36.125077+0000","last_active":"2024-09-16T10:22:36.125077+0000","last_peered":"2024-09-16T10:22:36.125077+0000","last_clean":"2024-09-16T10:22:36.125077+0000","last_became_active":"2024-09-16T10:22:36.124769+0000","last_became_peered":"2024-09-16T10:22:36.124769+0000","last_unstale":"2024-09-16T10:22:36.125077+0000","last_undegraded":"2024-09-16T10:22:36.125077+0000","last_fullsized":"2024-09-16T10:22:36.125077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:35:52.941164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"22'6","reported_seq":26,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.122385+0000","last_change":"2024-09-16T10:22:36.122554+0000","last_active":"2024-09-16T10:22:36.122385+0000","last_peered":"2024-09-16T10:22:36.122385+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:22:36.122276+0000","last_became_peered":"2024-09-16T10:22:36.122276+0000","last_unstale":"2024-09-16T10:22:36.122385+0000","last_undegraded":"2024-09-16T10:22:36.116861+0000","last_fullsized":"2024-09-16T10:22:36.122385+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:15:31.652616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.122288+0000","last_change":"2024-09-16T10:22:36.122288+0000","last_active":"2024-09-16T10:22:36.122288+0000","last_peered":"2024-09-16T10:22:36.122288+0000","last_clean":"2024-09-16T10:22:36.122288+0000","last_became_active":"2024-09-16T10:22:36.121757+0000","last_became_peered":"2024-09-16T10:22:36.121757+0000","last_unstale":"2024-09-16T10:22:36.122288+0000","last_undegraded":"2024-09-16T10:22:36.122288+0000","last_fullsized":"2024-09-16T10:22:36.122288+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:55:01.812618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'3","reported_seq":27,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.124106+0000","last_change":"2024-09-16T10:22:35.124106+0000","last_active":"2024-09-16T10:22:35.123958+0000","last_peered":"2024-09-16T10:22:34.110345+0000","last_clean":"2024-09-16T10:22:24.847222+0000","last_became_active":"2024-09-16T10:22:28.054310+0000","last_became_peered":"2024-09-16T10:22:28.054310+0000","last_unstale":"2024-09-16T10:22:35.124106+0000","last_undegraded":"2024-09-16T10:22:35.124106+0000","last_fullsized":"2024-09-16T10:22:35.124106+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:52.381096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":47,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.119872+0000","last_change":"2024-09-16T10:22:35.120131+0000","last_active":"2024-09-16T10:22:35.119872+0000","last_peered":"2024-09-16T10:22:34.110616+0000","last_clean":"2024-09-16T10:22:18.032042+0000","last_became_active":"2024-09-16T10:22:28.053555+0000","last_became_peered":"2024-09-16T10:22:28.053555+0000","last_unstale":"2024-09-16T10:22:35.119872+0000","last_undegraded":"2024-09-16T10:22:35.119872+0000","last_fullsized":"2024-09-16T10:22:35.119872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:47:59.769631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":28,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.117139+0000","last_change":"2024-09-16T10:22:35.117139+0000","last_active":"2024-09-16T10:22:35.117000+0000","last_peered":"2024-09-16T10:22:34.110442+0000","last_clean":"2024-09-16T10:22:25.286289+0000","last_became_active":"2024-09-16T10:22:28.054002+0000","last_became_peered":"2024-09-16T10:22:28.054002+0000","last_unstale":"2024-09-16T10:22:35.117139+0000","last_undegraded":"2024-09-16T10:22:35.117139+0000","last_fullsized":"2024-09-16T10:22:35.117139+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:05.474700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"22'1","reported_seq":24,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.122300+0000","last_change":"2024-09-16T10:22:35.122436+0000","last_active":"2024-09-16T10:22:35.122300+0000","last_peered":"2024-09-16T10:22:34.110623+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:22:28.054187+0000","last_became_peered":"2024-09-16T10:22:28.054187+0000","last_unstale":"2024-09-16T10:22:35.122300+0000","last_undegraded":"2024-09-16T10:22:35.122300+0000","last_fullsized":"2024-09-16T10:22:35.122300+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:15:45.607374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.119379+0000","last_change":"2024-09-16T10:22:35.119499+0000","last_active":"2024-09-16T10:22:35.119379+0000","last_peered":"2024-09-16T10:22:34.110250+0000","last_clean":"2024-09-16T10:22:18.022743+0000","last_became_active":"2024-09-16T10:22:28.054731+0000","last_became_peered":"2024-09-16T10:22:28.054731+0000","last_unstale":"2024-09-16T10:22:35.119379+0000","last_undegraded":"2024-09-16T10:22:35.119379+0000","last_fullsized":"2024-09-16T10:22:35.119379+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:43:42.551889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'5","reported_seq":29,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.120674+0000","last_change":"2024-09-16T10:22:35.120674+0000","last_active":"2024-09-16T10:22:35.120541+0000","last_peered":"2024-09-16T10:22:34.110329+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:22:28.053154+0000","last_became_peered":"2024-09-16T10:22:28.053154+0000","last_unstale":"2024-09-16T10:22:35.120674+0000","last_undegraded":"2024-09-16T10:22:35.120674+0000","last_fullsized":"2024-09-16T10:22:35.120674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:14:43.529224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.121997+0000","last_change":"2024-09-16T10:22:36.121997+0000","last_active":"2024-09-16T10:22:36.121997+0000","last_peered":"2024-09-16T10:22:36.121997+0000","last_clean":"2024-09-16T10:22:36.121997+0000","last_became_active":"2024-09-16T10:22:36.121405+0000","last_became_peered":"2024-09-16T10:22:36.121405+0000","last_unstale":"2024-09-16T10:22:36.121997+0000","last_undegraded":"2024-09-16T10:22:36.121997+0000","last_fullsized":"2024-09-16T10:22:36.121997+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:36:46.964390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"22'2","reported_seq":25,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.121585+0000","last_change":"2024-09-16T10:22:35.121729+0000","last_active":"2024-09-16T10:22:35.121585+0000","last_peered":"2024-09-16T10:22:34.110550+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:22:28.054302+0000","last_became_peered":"2024-09-16T10:22:28.054302+0000","last_unstale":"2024-09-16T10:22:35.121585+0000","last_undegraded":"2024-09-16T10:22:35.121585+0000","last_fullsized":"2024-09-16T10:22:35.121585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:32:08.537911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.121174+0000","last_change":"2024-09-16T10:22:35.121294+0000","last_active":"2024-09-16T10:22:35.121174+0000","last_peered":"2024-09-16T10:22:34.110602+0000","last_clean":"2024-09-16T10:22:18.022823+0000","last_became_active":"2024-09-16T10:22:28.053691+0000","last_became_peered":"2024-09-16T10:22:28.053691+0000","last_unstale":"2024-09-16T10:22:35.121174+0000","last_undegraded":"2024-09-16T10:22:35.121174+0000","last_fullsized":"2024-09-16T10:22:35.121174+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:34:37.408921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":42,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.122893+0000","last_change":"2024-09-16T10:22:35.123011+0000","last_active":"2024-09-16T10:22:35.122893+0000","last_peered":"2024-09-16T10:22:34.110760+0000","last_clean":"2024-09-16T10:22:18.029366+0000","last_became_active":"2024-09-16T10:22:28.055395+0000","last_became_peered":"2024-09-16T10:22:28.055395+0000","last_unstale":"2024-09-16T10:22:35.122893+0000","last_undegraded":"2024-09-16T10:22:35.122893+0000","last_fullsized":"2024-09-16T10:22:35.122893+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:10:36.080295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"22'9","reported_seq":33,"reported_epoch":24,"state":"peering","last_fresh":"2024-09-16T10:22:35.118934+0000","last_change":"2024-09-16T10:22:35.118934+0000","last_active":"2024-09-16T10:22:35.118792+0000","last_peered":"2024-09-16T10:22:34.110133+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:22:28.054066+0000","last_became_peered":"2024-09-16T10:22:28.054066+0000","last_unstale":"2024-09-16T10:22:35.118934+0000","last_undegraded":"2024-09-16T10:22:35.118934+0000","last_fullsized":"2024-09-16T10:22:35.118934+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:27.220732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":13524,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":18354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":73146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215107,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":252,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":258048,"data_stored":67169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":32304808},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":4830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:42.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:22:42.730 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:42.730 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7/60 objects degraded (11.667%), 4 pgs degraded","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":51,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":25,"num_osds":3,"num_up_osds":2,"osd_up_since":1726482155,"num_in_osds":2,"osd_in_since":1726482137,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":15},{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+degraded","count":3},{"state_name":"active+recovering+degraded","count":1}],"num_pgs":24,"num_pools":2,"num_objects":30,"data_bytes":13543,"bytes_used":60583936,"bytes_avail":193212944384,"bytes_total":193273528320,"inactive_pgs_ratio":0.625,"degraded_objects":7,"degraded_total":60,"degraded_ratio":0.11666666666666665},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-16T10:22:18.548879+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:22:42.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '3f765c65-edd5-4e6d-930a-33ce25cc666f', 'health': {'status': 'HEALTH_WARN', 'checks': {'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 7/60 objects degraded (11.667%), 4 pgs degraded', 'count': 4}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 51, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 25, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726482155, 'num_in_osds': 2, 'osd_in_since': 1726482137, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'peering', 'count': 15}, {'state_name': 'active+clean', 'count': 5}, {'state_name': 'active+recovery_wait+degraded', 'count': 3}, {'state_name': 'active+recovering+degraded', 'count': 1}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 30, 'data_bytes': 13543, 'bytes_used': 60583936, 'bytes_avail': 193212944384, 'bytes_total': 193273528320, 'inactive_pgs_ratio': 0.625, 'degraded_objects': 7, 'degraded_total': 60, 'degraded_ratio': 0.11666666666666665}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2024-09-16T10:22:18.548879+0000', 'services': {'mgr': {'daemons': {'summary': '', 'x': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'mon': {'daemons': {'summary': '', 'a': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'b': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'c': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '1': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-16T10:22:45.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:22:46.035 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:46.035 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:22:46.047 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-16T10:22:44.552845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13543,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":59176,"kb_used_data":576,"kb_used_omap":5,"kb_used_meta":58554,"kb_avail":188684504,"statfs":{"total":193273528320,"available":193212932096,"internally_reserved":0,"allocated":589824,"data_stored":143032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6034,"internal_metadata":59959406},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":-3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.439413"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.127471+0000","last_change":"2024-09-16T10:22:36.127471+0000","last_active":"2024-09-16T10:22:36.127471+0000","last_peered":"2024-09-16T10:22:36.127471+0000","last_clean":"2024-09-16T10:22:36.127471+0000","last_became_active":"2024-09-16T10:22:36.126974+0000","last_became_peered":"2024-09-16T10:22:36.126974+0000","last_unstale":"2024-09-16T10:22:36.127471+0000","last_undegraded":"2024-09-16T10:22:36.127471+0000","last_fullsized":"2024-09-16T10:22:36.127471+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:21:21.922974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'4","reported_seq":24,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.127642+0000","last_change":"2024-09-16T10:22:36.127819+0000","last_active":"2024-09-16T10:22:36.127642+0000","last_peered":"2024-09-16T10:22:36.127642+0000","last_clean":"2024-09-16T10:22:25.033354+0000","last_became_active":"2024-09-16T10:22:36.127559+0000","last_became_peered":"2024-09-16T10:22:36.127559+0000","last_unstale":"2024-09-16T10:22:36.127642+0000","last_undegraded":"2024-09-16T10:22:36.119252+0000","last_fullsized":"2024-09-16T10:22:36.127642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:37:20.704128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":42,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.123065+0000","last_change":"2024-09-16T10:22:36.123229+0000","last_active":"2024-09-16T10:22:36.123065+0000","last_peered":"2024-09-16T10:22:36.123065+0000","last_clean":"2024-09-16T10:22:25.588129+0000","last_became_active":"2024-09-16T10:22:36.122986+0000","last_became_peered":"2024-09-16T10:22:36.122986+0000","last_unstale":"2024-09-16T10:22:36.123065+0000","last_undegraded":"2024-09-16T10:22:36.116811+0000","last_fullsized":"2024-09-16T10:22:36.123065+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:41:52.629551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.127664+0000","last_change":"2024-09-16T10:22:36.127664+0000","last_active":"2024-09-16T10:22:36.127664+0000","last_peered":"2024-09-16T10:22:36.127664+0000","last_clean":"2024-09-16T10:22:36.127664+0000","last_became_active":"2024-09-16T10:22:36.127167+0000","last_became_peered":"2024-09-16T10:22:36.127167+0000","last_unstale":"2024-09-16T10:22:36.127664+0000","last_undegraded":"2024-09-16T10:22:36.127664+0000","last_fullsized":"2024-09-16T10:22:36.127664+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:27.016711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"22'5","reported_seq":35,"reported_epoch":25,"state":"active+recovering+degraded","last_fresh":"2024-09-16T10:22:36.122205+0000","last_change":"2024-09-16T10:22:36.127068+0000","last_active":"2024-09-16T10:22:36.122205+0000","last_peered":"2024-09-16T10:22:36.122205+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:22:36.121736+0000","last_became_peered":"2024-09-16T10:22:36.121736+0000","last_unstale":"2024-09-16T10:22:36.122205+0000","last_undegraded":"2024-09-16T10:22:36.117716+0000","last_fullsized":"2024-09-16T10:22:36.122205+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:57:32.787085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":33,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.124066+0000","last_change":"2024-09-16T10:22:36.124645+0000","last_active":"2024-09-16T10:22:36.124066+0000","last_peered":"2024-09-16T10:22:36.124066+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:22:36.123981+0000","last_became_peered":"2024-09-16T10:22:36.123981+0000","last_unstale":"2024-09-16T10:22:36.124066+0000","last_undegraded":"2024-09-16T10:22:36.117348+0000","last_fullsized":"2024-09-16T10:22:36.124066+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:55:31.469343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"22'4","reported_seq":34,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.121514+0000","last_change":"2024-09-16T10:22:36.122482+0000","last_active":"2024-09-16T10:22:36.121514+0000","last_peered":"2024-09-16T10:22:36.121514+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:22:36.121430+0000","last_became_peered":"2024-09-16T10:22:36.121430+0000","last_unstale":"2024-09-16T10:22:36.121514+0000","last_undegraded":"2024-09-16T10:22:36.116722+0000","last_fullsized":"2024-09-16T10:22:36.121514+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:03:04.731476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"22'1","reported_seq":31,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.127009+0000","last_change":"2024-09-16T10:22:36.127196+0000","last_active":"2024-09-16T10:22:36.127009+0000","last_peered":"2024-09-16T10:22:36.127009+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:22:36.126924+0000","last_became_peered":"2024-09-16T10:22:36.126924+0000","last_unstale":"2024-09-16T10:22:36.127009+0000","last_undegraded":"2024-09-16T10:22:36.115681+0000","last_fullsized":"2024-09-16T10:22:36.127009+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:52.323025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":41,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.123801+0000","last_change":"2024-09-16T10:22:36.123801+0000","last_active":"2024-09-16T10:22:36.123801+0000","last_peered":"2024-09-16T10:22:36.123801+0000","last_clean":"2024-09-16T10:22:36.123801+0000","last_became_active":"2024-09-16T10:22:36.123463+0000","last_became_peered":"2024-09-16T10:22:36.123463+0000","last_unstale":"2024-09-16T10:22:36.123801+0000","last_undegraded":"2024-09-16T10:22:36.123801+0000","last_fullsized":"2024-09-16T10:22:36.123801+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:27:48.640506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'5","reported_seq":44,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.123710+0000","last_change":"2024-09-16T10:22:36.124680+0000","last_active":"2024-09-16T10:22:36.123710+0000","last_peered":"2024-09-16T10:22:36.123710+0000","last_clean":"2024-09-16T10:22:25.638957+0000","last_became_active":"2024-09-16T10:22:36.123624+0000","last_became_peered":"2024-09-16T10:22:36.123624+0000","last_unstale":"2024-09-16T10:22:36.123710+0000","last_undegraded":"2024-09-16T10:22:36.117276+0000","last_fullsized":"2024-09-16T10:22:36.123710+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:02:52.624679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.125077+0000","last_change":"2024-09-16T10:22:36.125077+0000","last_active":"2024-09-16T10:22:36.125077+0000","last_peered":"2024-09-16T10:22:36.125077+0000","last_clean":"2024-09-16T10:22:36.125077+0000","last_became_active":"2024-09-16T10:22:36.124769+0000","last_became_peered":"2024-09-16T10:22:36.124769+0000","last_unstale":"2024-09-16T10:22:36.125077+0000","last_undegraded":"2024-09-16T10:22:36.125077+0000","last_fullsized":"2024-09-16T10:22:36.125077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:35:52.941164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"22'6","reported_seq":26,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.122385+0000","last_change":"2024-09-16T10:22:36.122554+0000","last_active":"2024-09-16T10:22:36.122385+0000","last_peered":"2024-09-16T10:22:36.122385+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:22:36.122276+0000","last_became_peered":"2024-09-16T10:22:36.122276+0000","last_unstale":"2024-09-16T10:22:36.122385+0000","last_undegraded":"2024-09-16T10:22:36.116861+0000","last_fullsized":"2024-09-16T10:22:36.122385+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:15:31.652616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.122288+0000","last_change":"2024-09-16T10:22:36.122288+0000","last_active":"2024-09-16T10:22:36.122288+0000","last_peered":"2024-09-16T10:22:36.122288+0000","last_clean":"2024-09-16T10:22:36.122288+0000","last_became_active":"2024-09-16T10:22:36.121757+0000","last_became_peered":"2024-09-16T10:22:36.121757+0000","last_unstale":"2024-09-16T10:22:36.122288+0000","last_undegraded":"2024-09-16T10:22:36.122288+0000","last_fullsized":"2024-09-16T10:22:36.122288+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:55:01.812618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'3","reported_seq":34,"reported_epoch":25,"state":"active+recovering+degraded","last_fresh":"2024-09-16T10:22:36.121332+0000","last_change":"2024-09-16T10:22:36.126190+0000","last_active":"2024-09-16T10:22:36.121332+0000","last_peered":"2024-09-16T10:22:36.121332+0000","last_clean":"2024-09-16T10:22:24.847222+0000","last_became_active":"2024-09-16T10:22:36.121067+0000","last_became_peered":"2024-09-16T10:22:36.121067+0000","last_unstale":"2024-09-16T10:22:36.121332+0000","last_undegraded":"2024-09-16T10:22:36.116320+0000","last_fullsized":"2024-09-16T10:22:36.121332+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:44:52.381096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":55,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.124859+0000","last_change":"2024-09-16T10:22:36.124859+0000","last_active":"2024-09-16T10:22:36.124859+0000","last_peered":"2024-09-16T10:22:36.124859+0000","last_clean":"2024-09-16T10:22:36.124859+0000","last_became_active":"2024-09-16T10:22:36.124050+0000","last_became_peered":"2024-09-16T10:22:36.124050+0000","last_unstale":"2024-09-16T10:22:36.124859+0000","last_undegraded":"2024-09-16T10:22:36.124859+0000","last_fullsized":"2024-09-16T10:22:36.124859+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:48.214129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":34,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.123851+0000","last_change":"2024-09-16T10:22:36.124045+0000","last_active":"2024-09-16T10:22:36.123851+0000","last_peered":"2024-09-16T10:22:36.123851+0000","last_clean":"2024-09-16T10:22:25.286289+0000","last_became_active":"2024-09-16T10:22:36.123758+0000","last_became_peered":"2024-09-16T10:22:36.123758+0000","last_unstale":"2024-09-16T10:22:36.123851+0000","last_undegraded":"2024-09-16T10:22:36.116218+0000","last_fullsized":"2024-09-16T10:22:36.123851+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:05.474700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"22'1","reported_seq":30,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.123052+0000","last_change":"2024-09-16T10:22:36.123231+0000","last_active":"2024-09-16T10:22:36.123052+0000","last_peered":"2024-09-16T10:22:36.123052+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:22:36.122966+0000","last_became_peered":"2024-09-16T10:22:36.122966+0000","last_unstale":"2024-09-16T10:22:36.123052+0000","last_undegraded":"2024-09-16T10:22:36.116263+0000","last_fullsized":"2024-09-16T10:22:36.123052+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:53:52.436582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":41,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.124787+0000","last_change":"2024-09-16T10:22:36.124787+0000","last_active":"2024-09-16T10:22:36.124787+0000","last_peered":"2024-09-16T10:22:36.124787+0000","last_clean":"2024-09-16T10:22:36.124787+0000","last_became_active":"2024-09-16T10:22:36.124418+0000","last_became_peered":"2024-09-16T10:22:36.124418+0000","last_unstale":"2024-09-16T10:22:36.124787+0000","last_undegraded":"2024-09-16T10:22:36.124787+0000","last_fullsized":"2024-09-16T10:22:36.124787+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:19:01.730908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'5","reported_seq":35,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.122989+0000","last_change":"2024-09-16T10:22:36.123145+0000","last_active":"2024-09-16T10:22:36.122989+0000","last_peered":"2024-09-16T10:22:36.122989+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:22:36.122911+0000","last_became_peered":"2024-09-16T10:22:36.122911+0000","last_unstale":"2024-09-16T10:22:36.122989+0000","last_undegraded":"2024-09-16T10:22:36.117157+0000","last_fullsized":"2024-09-16T10:22:36.122989+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:14:43.529224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.121997+0000","last_change":"2024-09-16T10:22:36.121997+0000","last_active":"2024-09-16T10:22:36.121997+0000","last_peered":"2024-09-16T10:22:36.121997+0000","last_clean":"2024-09-16T10:22:36.121997+0000","last_became_active":"2024-09-16T10:22:36.121405+0000","last_became_peered":"2024-09-16T10:22:36.121405+0000","last_unstale":"2024-09-16T10:22:36.121997+0000","last_undegraded":"2024-09-16T10:22:36.121997+0000","last_fullsized":"2024-09-16T10:22:36.121997+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:36:46.964390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"22'2","reported_seq":31,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.124122+0000","last_change":"2024-09-16T10:22:36.124344+0000","last_active":"2024-09-16T10:22:36.124122+0000","last_peered":"2024-09-16T10:22:36.124122+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:22:36.124039+0000","last_became_peered":"2024-09-16T10:22:36.124039+0000","last_unstale":"2024-09-16T10:22:36.124122+0000","last_undegraded":"2024-09-16T10:22:36.117136+0000","last_fullsized":"2024-09-16T10:22:36.124122+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:47:25.025147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":41,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.123159+0000","last_change":"2024-09-16T10:22:36.123159+0000","last_active":"2024-09-16T10:22:36.123159+0000","last_peered":"2024-09-16T10:22:36.123159+0000","last_clean":"2024-09-16T10:22:36.123159+0000","last_became_active":"2024-09-16T10:22:36.122886+0000","last_became_peered":"2024-09-16T10:22:36.122886+0000","last_unstale":"2024-09-16T10:22:36.123159+0000","last_undegraded":"2024-09-16T10:22:36.123159+0000","last_fullsized":"2024-09-16T10:22:36.123159+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:42:06.663867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":50,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-16T10:22:36.121394+0000","last_change":"2024-09-16T10:22:36.121394+0000","last_active":"2024-09-16T10:22:36.121394+0000","last_peered":"2024-09-16T10:22:36.121394+0000","last_clean":"2024-09-16T10:22:36.121394+0000","last_became_active":"2024-09-16T10:22:36.121115+0000","last_became_peered":"2024-09-16T10:22:36.121115+0000","last_unstale":"2024-09-16T10:22:36.121394+0000","last_undegraded":"2024-09-16T10:22:36.121394+0000","last_fullsized":"2024-09-16T10:22:36.121394+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:29:11.857074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"22'9","reported_seq":39,"reported_epoch":25,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:22:36.124135+0000","last_change":"2024-09-16T10:22:36.124622+0000","last_active":"2024-09-16T10:22:36.124135+0000","last_peered":"2024-09-16T10:22:36.124135+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:22:36.124063+0000","last_became_peered":"2024-09-16T10:22:36.124063+0000","last_unstale":"2024-09-16T10:22:36.124135+0000","last_undegraded":"2024-09-16T10:22:36.116625+0000","last_fullsized":"2024-09-16T10:22:36.124135+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:57:27.220732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"1","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":13524,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":7,"num_objects_missing":7,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":18354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738382,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27352,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344488,"statfs":{"total":96636764160,"available":96608755712,"internally_reserved":0,"allocated":331776,"data_stored":75863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23999999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215108,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31824,"kb_used_data":252,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94340016,"statfs":{"total":96636764160,"available":96604176384,"internally_reserved":0,"allocated":258048,"data_stored":67169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":32304808},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.19900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":4830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:22:46.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:22:46.396 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:22:46.396 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27/60 objects degraded (45.000%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":55,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":25,"num_osds":3,"num_up_osds":2,"osd_up_since":1726482155,"num_in_osds":2,"osd_in_since":1726482137,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+degraded","count":12},{"state_name":"active+clean","count":10},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":24,"num_pools":2,"num_objects":30,"data_bytes":13543,"bytes_used":60596224,"bytes_avail":193212932096,"bytes_total":193273528320,"degraded_objects":27,"degraded_total":60,"degraded_ratio":0.45000000000000001},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-16T10:22:18.548879+0000","services":{"mgr":{"daemons":{"summary":"","x":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"mon":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}},"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:22:46.408 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '3f765c65-edd5-4e6d-930a-33ce25cc666f', 'health': {'status': 'HEALTH_WARN', 'checks': {'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 27/60 objects degraded (45.000%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 55, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 25, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726482155, 'num_in_osds': 2, 'osd_in_since': 1726482137, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_wait+degraded', 'count': 12}, {'state_name': 'active+clean', 'count': 10}, {'state_name': 'active+recovering+degraded', 'count': 2}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 30, 'data_bytes': 13543, 'bytes_used': 60596224, 'bytes_avail': 193212932096, 'bytes_total': 193273528320, 'degraded_objects': 27, 'degraded_total': 60, 'degraded_ratio': 0.45}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2024-09-16T10:22:18.548879+0000', 'services': {'mgr': {'daemons': {'summary': '', 'x': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'mon': {'daemons': {'summary': '', 'a': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'b': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, 'c': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}, 'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '1': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-16T10:22:46.409 INFO:tasks.rep_lost_unfound_delete.ceph_manager:active! 2024-09-16T10:22:46.409 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T10:22:46.409 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:22:46.441 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T10:22:46.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-16T10:22:46.773 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-16T10:22:46.770+0000 7f0322f76640 -1 mon.a@0(leader).osd e25 definitely_dead 0 2024-09-16T10:22:47.198 INFO:teuthology.orchestra.run.smithi050.stderr:marked down osd.1. 2024-09-16T10:22:47.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-09-16T10:22:47.507 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-16T10:22:47.505+0000 7f0322f76640 -1 mon.a@0(leader).osd e26 definitely_dead 0 2024-09-16T10:22:48.198 INFO:teuthology.orchestra.run.smithi050.stderr:marked out osd.1. 2024-09-16T10:22:48.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-09-16T10:22:49.206 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.202+0000 7ff6cbde7640 -1 log_channel(cluster) log [ERR] : 2.a has 2 objects unfound and apparently lost 2024-09-16T10:22:49.206 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.202+0000 7ff6c85e0640 -1 log_channel(cluster) log [ERR] : 2.9 has 2 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.202+0000 7ff6c7ddf640 -1 log_channel(cluster) log [ERR] : 2.2 has 2 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.202+0000 7ff6ccde9640 -1 log_channel(cluster) log [ERR] : 2.0 has 2 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.202+0000 7ff6cc5e8640 -1 log_channel(cluster) log [ERR] : 2.1 has 2 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.202+0000 7ff6cade5640 -1 log_channel(cluster) log [ERR] : 2.4 has 3 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.202+0000 7ff6c75de640 -1 log_channel(cluster) log [ERR] : 2.3 has 2 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.203+0000 7ff6c65dc640 -1 log_channel(cluster) log [ERR] : 2.d has 1 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.203+0000 7ff6c8de1640 -1 log_channel(cluster) log [ERR] : 2.8 has 1 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.203+0000 7ff6c75de640 -1 log_channel(cluster) log [ERR] : 2.b has 2 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.203+0000 7ff6c65dc640 -1 log_channel(cluster) log [ERR] : 2.5 has 2 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.203+0000 7ff6c9de3640 -1 log_channel(cluster) log [ERR] : 2.e has 2 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.203+0000 7ff6c5ddb640 -1 log_channel(cluster) log [ERR] : 2.6 has 3 objects unfound and apparently lost 2024-09-16T10:22:49.207 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:49.203+0000 7ff6c95e2640 -1 log_channel(cluster) log [ERR] : 2.7 has 1 objects unfound and apparently lost 2024-09-16T10:22:49.208 INFO:teuthology.orchestra.run.smithi050.stderr:marked osd lost in epoch 26 2024-09-16T10:22:49.222 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T10:22:49.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T10:22:49.225 INFO:tasks.ceph.osd.2:Started 2024-09-16T10:22:49.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T10:22:49.333 INFO:teuthology.orchestra.run.smithi050.stderr:no valid command found; 10 closest matches: 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:0 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:1 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:2 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:abort 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:assert 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:config diff 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:config diff get 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:config get 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:config help [] 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:config set ... 2024-09-16T10:22:49.334 INFO:teuthology.orchestra.run.smithi050.stderr:admin_socket: invalid command 2024-09-16T10:22:49.336 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:22:49.336 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-16T10:22:49.821 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-09-16T10:22:49.819+0000 7f580d5c5640 -1 Falling back to public interface 2024-09-16T10:22:53.290 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-09-16T10:22:53.289+0000 7f580d5c5640 -1 osd.2 17 log_to_monitors true 2024-09-16T10:22:54.222 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.218+0000 7ff6ccde9640 -1 log_channel(cluster) log [ERR] : 2.0 has 2 objects unfound and apparently lost 2024-09-16T10:22:54.222 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.218+0000 7ff6c85e0640 -1 log_channel(cluster) log [ERR] : 2.9 has 2 objects unfound and apparently lost 2024-09-16T10:22:54.222 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.218+0000 7ff6cc5e8640 -1 log_channel(cluster) log [ERR] : 2.1 has 2 objects unfound and apparently lost 2024-09-16T10:22:54.223 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.218+0000 7ff6ccde9640 -1 log_channel(cluster) log [ERR] : 2.8 has 1 objects unfound and apparently lost 2024-09-16T10:22:54.223 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.218+0000 7ff6cbde7640 -1 log_channel(cluster) log [ERR] : 2.a has 2 objects unfound and apparently lost 2024-09-16T10:22:54.223 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.219+0000 7ff6c7ddf640 -1 log_channel(cluster) log [ERR] : 2.2 has 2 objects unfound and apparently lost 2024-09-16T10:22:54.223 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.219+0000 7ff6c6ddd640 -1 log_channel(cluster) log [ERR] : 2.4 has 3 objects unfound and apparently lost 2024-09-16T10:22:54.223 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.219+0000 7ff6cb5e6640 -1 log_channel(cluster) log [ERR] : 2.3 has 2 objects unfound and apparently lost 2024-09-16T10:22:54.223 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.219+0000 7ff6c75de640 -1 log_channel(cluster) log [ERR] : 2.b has 2 objects unfound and apparently lost 2024-09-16T10:22:54.224 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.219+0000 7ff6ca5e4640 -1 log_channel(cluster) log [ERR] : 2.d has 1 objects unfound and apparently lost 2024-09-16T10:22:54.224 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.219+0000 7ff6c65dc640 -1 log_channel(cluster) log [ERR] : 2.5 has 2 objects unfound and apparently lost 2024-09-16T10:22:54.224 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.219+0000 7ff6c9de3640 -1 log_channel(cluster) log [ERR] : 2.e has 2 objects unfound and apparently lost 2024-09-16T10:22:54.224 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.219+0000 7ff6c5ddb640 -1 log_channel(cluster) log [ERR] : 2.6 has 3 objects unfound and apparently lost 2024-09-16T10:22:54.224 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:54.220+0000 7ff6c55da640 -1 log_channel(cluster) log [ERR] : 2.7 has 1 objects unfound and apparently lost 2024-09-16T10:22:54.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:no valid command found; 10 closest matches: 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:0 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:1 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:2 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:abort 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:assert 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:bluefs debug_inject_read_zeros 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:bluefs files list 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:bluefs stats 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:bluestore allocator dump block 2024-09-16T10:22:54.464 INFO:teuthology.orchestra.run.smithi050.stderr:bluestore allocator fragmentation block 2024-09-16T10:22:54.465 INFO:teuthology.orchestra.run.smithi050.stderr:admin_socket: invalid command 2024-09-16T10:22:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:22:54.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-16T10:22:54.836 INFO:tasks.ceph.osd.2.smithi050.stderr:2024-09-16T10:22:54.835+0000 7f5801373640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:22:55.846 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.843+0000 7ff6c8de1640 -1 log_channel(cluster) log [ERR] : 2.8 has 1 objects unfound and apparently lost 2024-09-16T10:22:55.846 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.843+0000 7ff6c75de640 -1 log_channel(cluster) log [ERR] : 2.b has 2 objects unfound and apparently lost 2024-09-16T10:22:55.846 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.843+0000 7ff6ccde9640 -1 log_channel(cluster) log [ERR] : 2.0 has 2 objects unfound and apparently lost 2024-09-16T10:22:55.846 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.843+0000 7ff6c85e0640 -1 log_channel(cluster) log [ERR] : 2.1 has 2 objects unfound and apparently lost 2024-09-16T10:22:55.846 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.843+0000 7ff6c9de3640 -1 log_channel(cluster) log [ERR] : 2.e has 2 objects unfound and apparently lost 2024-09-16T10:22:55.847 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.843+0000 7ff6c65dc640 -1 log_channel(cluster) log [ERR] : 2.d has 1 objects unfound and apparently lost 2024-09-16T10:22:55.847 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.843+0000 7ff6cbde7640 -1 log_channel(cluster) log [ERR] : 2.2 has 2 objects unfound and apparently lost 2024-09-16T10:22:55.847 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.843+0000 7ff6c6ddd640 -1 log_channel(cluster) log [ERR] : 2.4 has 3 objects unfound and apparently lost 2024-09-16T10:22:55.847 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.843+0000 7ff6c7ddf640 -1 log_channel(cluster) log [ERR] : 2.a has 2 objects unfound and apparently lost 2024-09-16T10:22:55.847 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.844+0000 7ff6cb5e6640 -1 log_channel(cluster) log [ERR] : 2.3 has 2 objects unfound and apparently lost 2024-09-16T10:22:55.847 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.844+0000 7ff6cc5e8640 -1 log_channel(cluster) log [ERR] : 2.9 has 2 objects unfound and apparently lost 2024-09-16T10:22:55.847 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.844+0000 7ff6c95e2640 -1 log_channel(cluster) log [ERR] : 2.7 has 1 objects unfound and apparently lost 2024-09-16T10:22:55.847 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.844+0000 7ff6c5ddb640 -1 log_channel(cluster) log [ERR] : 2.6 has 3 objects unfound and apparently lost 2024-09-16T10:22:55.847 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:55.844+0000 7ff6c65dc640 -1 log_channel(cluster) log [ERR] : 2.5 has 2 objects unfound and apparently lost 2024-09-16T10:22:56.009 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:56.005+0000 7ff6c6ddd640 -1 log_channel(cluster) log [ERR] : 2.4 has 3 objects unfound and apparently lost 2024-09-16T10:22:56.009 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:56.005+0000 7ff6c7ddf640 -1 log_channel(cluster) log [ERR] : 2.a has 2 objects unfound and apparently lost 2024-09-16T10:22:56.009 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:56.005+0000 7ff6c7ddf640 -1 log_channel(cluster) log [ERR] : 2.2 has 2 objects unfound and apparently lost 2024-09-16T10:22:56.009 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:56.007+0000 7ff6c55da640 -1 log_channel(cluster) log [ERR] : 2.7 has 1 objects unfound and apparently lost 2024-09-16T10:22:56.009 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:56.007+0000 7ff6c85e0640 -1 log_channel(cluster) log [ERR] : 2.9 has 2 objects unfound and apparently lost 2024-09-16T10:22:56.010 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:56.007+0000 7ff6c65dc640 -1 log_channel(cluster) log [ERR] : 2.5 has 2 objects unfound and apparently lost 2024-09-16T10:22:56.010 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:56.008+0000 7ff6c5ddb640 -1 log_channel(cluster) log [ERR] : 2.6 has 3 objects unfound and apparently lost 2024-09-16T10:22:57.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:57.010+0000 7ff6c7ddf640 -1 log_channel(cluster) log [ERR] : 2.a has 2 objects unfound and apparently lost 2024-09-16T10:22:57.015 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:57.011+0000 7ff6c7ddf640 -1 log_channel(cluster) log [ERR] : 2.2 has 2 objects unfound and apparently lost 2024-09-16T10:22:57.015 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:57.011+0000 7ff6c95e2640 -1 log_channel(cluster) log [ERR] : 2.7 has 1 objects unfound and apparently lost 2024-09-16T10:22:57.015 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:57.011+0000 7ff6c5ddb640 -1 log_channel(cluster) log [ERR] : 2.6 has 3 objects unfound and apparently lost 2024-09-16T10:22:57.016 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:57.012+0000 7ff6ca5e4640 -1 log_channel(cluster) log [ERR] : 2.5 has 2 objects unfound and apparently lost 2024-09-16T10:22:57.016 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:57.012+0000 7ff6cade5640 -1 log_channel(cluster) log [ERR] : 2.4 has 3 objects unfound and apparently lost 2024-09-16T10:22:57.016 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:22:57.013+0000 7ff6cc5e8640 -1 log_channel(cluster) log [ERR] : 2.9 has 2 objects unfound and apparently lost 2024-09-16T10:22:59.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T10:22:59.587 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:22:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-16T10:22:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-16T10:22:59.588 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:22:59.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-09-16T10:22:59.889 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-16T10:22:59.887+0000 7f0322f76640 -1 mon.a@0(leader).osd e32 definitely_dead 0 2024-09-16T10:23:00.018 INFO:teuthology.orchestra.run.smithi050.stderr:marked in osd.2. 2024-09-16T10:23:00.034 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for osd.2 to be up 2024-09-16T10:23:00.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:23:00.316 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:00.317 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":33,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:23:00.010308+0000","last_up_change":"2024-09-16T10:22:55.836931+0000","last_in_change":"2024-09-16T10:23:00.010308+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T10:22:00.707491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":1.4900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-16T10:22:12.124014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":1.8700000047683716,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":20,"up_from":24,"up_thru":31,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2470025375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":2470025375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":2470025375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":2470025375}]},"public_addr":"172.21.15.50:6804/2470025375","cluster_addr":"172.21.15.50:6805/2470025375","heartbeat_back_addr":"172.21.15.50:6807/2470025375","heartbeat_front_addr":"172.21.15.50:6806/2470025375","state":["exists","up"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":26,"lost_at":26,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":202723672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":202723672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":202723672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":202723672}]},"public_addr":"172.21.15.50:6800/202723672","cluster_addr":"172.21.15.50:6801/202723672","heartbeat_back_addr":"172.21.15.50:6803/202723672","heartbeat_front_addr":"172.21.15.50:6802/202723672","state":["exists"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":17,"up_from":30,"up_thru":15,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1896614959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1896614959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":1896614959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1896614959}]},"public_addr":"172.21.15.50:6800/1896614959","cluster_addr":"172.21.15.50:6801/1896614959","heartbeat_back_addr":"172.21.15.50:6803/1896614959","heartbeat_front_addr":"172.21.15.50:6802/1896614959","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.244822+0000","dead_epoch":21},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T10:21:58.213317+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.228331+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.1","osds":[0,2]},{"pgid":"1.3","osds":[0,2]},{"pgid":"1.5","osds":[0,2]},{"pgid":"1.7","osds":[0,2]},{"pgid":"2.0","osds":[0,2]},{"pgid":"2.1","osds":[0,2]},{"pgid":"2.3","osds":[0,2]},{"pgid":"2.c","osds":[0,2]},{"pgid":"2.d","osds":[0,2]},{"pgid":"2.e","osds":[0,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:23:00.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:osd.2 is up 2024-09-16T10:23:00.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T10:23:00.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T10:23:00.466 INFO:teuthology.orchestra.run.smithi050.stdout:128849018884 2024-09-16T10:23:00.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:23:00.467 INFO:teuthology.orchestra.run.smithi050.stdout:103079215112 2024-09-16T10:23:00.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:23:00.797 INFO:teuthology.orchestra.run.smithi050.stdout:2 2024-09-16T10:23:00.802 INFO:teuthology.orchestra.run.smithi050.stdout:103079215110 2024-09-16T10:23:00.811 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 128849018884 got 2 for osd.2 2024-09-16T10:23:00.815 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 103079215112 got 103079215110 for osd.0 2024-09-16T10:23:01.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:23:01.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:23:02.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:23:02.010+0000 7ff6cbde7640 -1 log_channel(cluster) log [ERR] : 2.a has 2 objects unfound and apparently lost 2024-09-16T10:23:02.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:23:02.010+0000 7ff6c75de640 -1 log_channel(cluster) log [ERR] : 2.b has 2 objects unfound and apparently lost 2024-09-16T10:23:02.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:23:02.010+0000 7ff6ccde9640 -1 log_channel(cluster) log [ERR] : 2.8 has 1 objects unfound and apparently lost 2024-09-16T10:23:02.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:23:02.010+0000 7ff6cbde7640 -1 log_channel(cluster) log [ERR] : 2.2 has 2 objects unfound and apparently lost 2024-09-16T10:23:02.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:23:02.010+0000 7ff6ca5e4640 -1 log_channel(cluster) log [ERR] : 2.5 has 2 objects unfound and apparently lost 2024-09-16T10:23:02.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:23:02.010+0000 7ff6c85e0640 -1 log_channel(cluster) log [ERR] : 2.9 has 2 objects unfound and apparently lost 2024-09-16T10:23:02.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:23:02.010+0000 7ff6cade5640 -1 log_channel(cluster) log [ERR] : 2.4 has 3 objects unfound and apparently lost 2024-09-16T10:23:02.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:23:02.011+0000 7ff6c95e2640 -1 log_channel(cluster) log [ERR] : 2.7 has 1 objects unfound and apparently lost 2024-09-16T10:23:02.013 INFO:tasks.ceph.osd.0.smithi050.stderr:2024-09-16T10:23:02.011+0000 7ff6c5ddb640 -1 log_channel(cluster) log [ERR] : 2.6 has 3 objects unfound and apparently lost 2024-09-16T10:23:02.136 INFO:teuthology.orchestra.run.smithi050.stdout:128849018884 2024-09-16T10:23:02.149 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 128849018884 got 128849018884 for osd.2 2024-09-16T10:23:02.149 DEBUG:teuthology.parallel:result is None 2024-09-16T10:23:02.172 INFO:teuthology.orchestra.run.smithi050.stdout:103079215112 2024-09-16T10:23:02.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 103079215112 got 103079215112 for osd.0 2024-09-16T10:23:02.184 DEBUG:teuthology.parallel:result is None 2024-09-16T10:23:02.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting till active 2024-09-16T10:23:02.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:02.504 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:02.505 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:02.516 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-16T10:23:02.007229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13543,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":63776,"kb_used_data":584,"kb_used_omap":8,"kb_used_meta":63159,"kb_avail":188679904,"statfs":{"total":193273528320,"available":193208221696,"internally_reserved":0,"allocated":598016,"data_stored":178661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8872,"internal_metadata":64675160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-57,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.452023"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":34,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.016293+0000","last_change":"2024-09-16T10:23:00.017863+0000","last_active":"2024-09-16T10:23:00.016293+0000","last_peered":"2024-09-16T10:22:57.012706+0000","last_clean":"2024-09-16T10:22:36.128344+0000","last_became_active":"2024-09-16T10:22:48.200661+0000","last_became_peered":"2024-09-16T10:22:48.200661+0000","last_unstale":"2024-09-16T10:23:00.016293+0000","last_undegraded":"2024-09-16T10:23:00.016293+0000","last_fullsized":"2024-09-16T10:23:00.016293+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:26:49.061549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'4","reported_seq":44,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.014818+0000","last_change":"2024-09-16T10:23:00.014976+0000","last_active":"2024-09-16T10:23:00.014818+0000","last_peered":"2024-09-16T10:22:57.015526+0000","last_clean":"2024-09-16T10:22:25.033354+0000","last_became_active":"2024-09-16T10:22:57.015078+0000","last_became_peered":"2024-09-16T10:22:57.015078+0000","last_unstale":"2024-09-16T10:23:00.014818+0000","last_undegraded":"2024-09-16T10:23:00.014818+0000","last_fullsized":"2024-09-16T10:23:00.014818+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:05.684252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":57,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.015491+0000","last_change":"2024-09-16T10:23:00.015653+0000","last_active":"2024-09-16T10:23:00.015491+0000","last_peered":"2024-09-16T10:22:57.019006+0000","last_clean":"2024-09-16T10:22:25.588129+0000","last_became_active":"2024-09-16T10:22:57.018710+0000","last_became_peered":"2024-09-16T10:22:57.018710+0000","last_unstale":"2024-09-16T10:23:00.015491+0000","last_undegraded":"2024-09-16T10:23:00.015491+0000","last_fullsized":"2024-09-16T10:23:00.015491+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:04.949838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.015493+0000","last_change":"2024-09-16T10:23:00.015627+0000","last_active":"2024-09-16T10:23:00.015493+0000","last_peered":"2024-09-16T10:22:57.016848+0000","last_clean":"2024-09-16T10:22:57.016848+0000","last_became_active":"2024-09-16T10:22:57.014864+0000","last_became_peered":"2024-09-16T10:22:57.014864+0000","last_unstale":"2024-09-16T10:23:00.015493+0000","last_undegraded":"2024-09-16T10:23:00.015493+0000","last_fullsized":"2024-09-16T10:23:00.015493+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:46.368618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"22'5","reported_seq":56,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.015736+0000","last_change":"2024-09-16T10:23:00.015736+0000","last_active":"2024-09-16T10:23:00.015578+0000","last_peered":"2024-09-16T10:22:57.017732+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:22:57.017445+0000","last_became_peered":"2024-09-16T10:22:57.017445+0000","last_unstale":"2024-09-16T10:23:00.015736+0000","last_undegraded":"2024-09-16T10:23:00.015736+0000","last_fullsized":"2024-09-16T10:23:00.015736+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:12:00.306083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":42,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.013295+0000","last_change":"2024-09-16T10:23:00.013438+0000","last_active":"2024-09-16T10:23:00.013295+0000","last_peered":"2024-09-16T10:22:57.011728+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:22:48.198352+0000","last_became_peered":"2024-09-16T10:22:48.198352+0000","last_unstale":"2024-09-16T10:23:00.013295+0000","last_undegraded":"2024-09-16T10:23:00.013295+0000","last_fullsized":"2024-09-16T10:23:00.013295+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:32:41.645526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"22'4","reported_seq":43,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.014197+0000","last_change":"2024-09-16T10:23:00.014372+0000","last_active":"2024-09-16T10:23:00.014197+0000","last_peered":"2024-09-16T10:22:57.014078+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:22:48.202715+0000","last_became_peered":"2024-09-16T10:22:48.202715+0000","last_unstale":"2024-09-16T10:23:00.014197+0000","last_undegraded":"2024-09-16T10:23:00.014197+0000","last_fullsized":"2024-09-16T10:23:00.014197+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:53:44.552125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'1","reported_seq":52,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.014742+0000","last_change":"2024-09-16T10:23:00.014742+0000","last_active":"2024-09-16T10:23:00.014187+0000","last_peered":"2024-09-16T10:22:57.017478+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:22:57.017161+0000","last_became_peered":"2024-09-16T10:22:57.017161+0000","last_unstale":"2024-09-16T10:23:00.014742+0000","last_undegraded":"2024-09-16T10:23:00.014742+0000","last_fullsized":"2024-09-16T10:23:00.014742+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:58.055496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":48,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.014787+0000","last_change":"2024-09-16T10:23:00.014917+0000","last_active":"2024-09-16T10:23:00.014787+0000","last_peered":"2024-09-16T10:22:57.012438+0000","last_clean":"2024-09-16T10:22:18.022590+0000","last_became_active":"2024-09-16T10:22:48.197536+0000","last_became_peered":"2024-09-16T10:22:48.197536+0000","last_unstale":"2024-09-16T10:23:00.014787+0000","last_undegraded":"2024-09-16T10:23:00.014787+0000","last_fullsized":"2024-09-16T10:23:00.014787+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:58:03.345037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"22'5","reported_seq":59,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.014865+0000","last_change":"2024-09-16T10:23:00.015016+0000","last_active":"2024-09-16T10:23:00.014865+0000","last_peered":"2024-09-16T10:22:57.017568+0000","last_clean":"2024-09-16T10:22:25.638957+0000","last_became_active":"2024-09-16T10:22:57.017150+0000","last_became_peered":"2024-09-16T10:22:57.017150+0000","last_unstale":"2024-09-16T10:23:00.014865+0000","last_undegraded":"2024-09-16T10:23:00.014865+0000","last_fullsized":"2024-09-16T10:23:00.014865+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:31:56.901192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.013228+0000","last_change":"2024-09-16T10:23:00.013366+0000","last_active":"2024-09-16T10:23:00.013228+0000","last_peered":"2024-09-16T10:22:57.018488+0000","last_clean":"2024-09-16T10:22:57.018488+0000","last_became_active":"2024-09-16T10:22:57.016819+0000","last_became_peered":"2024-09-16T10:22:57.016819+0000","last_unstale":"2024-09-16T10:23:00.013228+0000","last_undegraded":"2024-09-16T10:23:00.013228+0000","last_fullsized":"2024-09-16T10:23:00.013228+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:29.249666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"22'6","reported_seq":40,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.014499+0000","last_change":"2024-09-16T10:23:00.014665+0000","last_active":"2024-09-16T10:23:00.014499+0000","last_peered":"2024-09-16T10:22:57.012279+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:22:48.198692+0000","last_became_peered":"2024-09-16T10:22:48.198692+0000","last_unstale":"2024-09-16T10:23:00.014499+0000","last_undegraded":"2024-09-16T10:23:00.014499+0000","last_fullsized":"2024-09-16T10:23:00.014499+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:49:09.127884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":46,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.013265+0000","last_change":"2024-09-16T10:23:00.013418+0000","last_active":"2024-09-16T10:23:00.013265+0000","last_peered":"2024-09-16T10:22:57.011666+0000","last_clean":"2024-09-16T10:22:36.125521+0000","last_became_active":"2024-09-16T10:22:48.197708+0000","last_became_peered":"2024-09-16T10:22:48.197708+0000","last_unstale":"2024-09-16T10:23:00.013265+0000","last_undegraded":"2024-09-16T10:23:00.013265+0000","last_fullsized":"2024-09-16T10:23:00.013265+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:32:06.321673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'3","reported_seq":48,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.014445+0000","last_change":"2024-09-16T10:23:00.014887+0000","last_active":"2024-09-16T10:23:00.014445+0000","last_peered":"2024-09-16T10:22:57.017434+0000","last_clean":"2024-09-16T10:22:24.847222+0000","last_became_active":"2024-09-16T10:22:57.017138+0000","last_became_peered":"2024-09-16T10:22:57.017138+0000","last_unstale":"2024-09-16T10:23:00.014445+0000","last_undegraded":"2024-09-16T10:23:00.014445+0000","last_fullsized":"2024-09-16T10:23:00.014445+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:35:14.811553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":48,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.014495+0000","last_change":"2024-09-16T10:23:00.014643+0000","last_active":"2024-09-16T10:23:00.014495+0000","last_peered":"2024-09-16T10:22:57.017452+0000","last_clean":"2024-09-16T10:22:57.017452+0000","last_became_active":"2024-09-16T10:22:57.015587+0000","last_became_peered":"2024-09-16T10:22:57.015587+0000","last_unstale":"2024-09-16T10:23:00.014495+0000","last_undegraded":"2024-09-16T10:23:00.014495+0000","last_fullsized":"2024-09-16T10:23:00.014495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:31:34.592553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":49,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.013209+0000","last_change":"2024-09-16T10:23:00.013369+0000","last_active":"2024-09-16T10:23:00.013209+0000","last_peered":"2024-09-16T10:22:57.015281+0000","last_clean":"2024-09-16T10:22:25.286289+0000","last_became_active":"2024-09-16T10:22:57.014982+0000","last_became_peered":"2024-09-16T10:22:57.014982+0000","last_unstale":"2024-09-16T10:23:00.013209+0000","last_undegraded":"2024-09-16T10:23:00.013209+0000","last_fullsized":"2024-09-16T10:23:00.013209+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:01.336630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"22'1","reported_seq":39,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.016205+0000","last_change":"2024-09-16T10:23:00.016357+0000","last_active":"2024-09-16T10:23:00.016205+0000","last_peered":"2024-09-16T10:22:57.012532+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:22:48.198877+0000","last_became_peered":"2024-09-16T10:22:48.198877+0000","last_unstale":"2024-09-16T10:23:00.016205+0000","last_undegraded":"2024-09-16T10:23:00.016205+0000","last_fullsized":"2024-09-16T10:23:00.016205+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:36.177867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":48,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.014499+0000","last_change":"2024-09-16T10:23:00.014644+0000","last_active":"2024-09-16T10:23:00.014499+0000","last_peered":"2024-09-16T10:22:57.012449+0000","last_clean":"2024-09-16T10:22:18.022743+0000","last_became_active":"2024-09-16T10:22:48.197755+0000","last_became_peered":"2024-09-16T10:22:48.197755+0000","last_unstale":"2024-09-16T10:23:00.014499+0000","last_undegraded":"2024-09-16T10:23:00.014499+0000","last_fullsized":"2024-09-16T10:23:00.014499+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:37.750360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"22'5","reported_seq":44,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.019798+0000","last_change":"2024-09-16T10:23:00.019978+0000","last_active":"2024-09-16T10:23:00.019798+0000","last_peered":"2024-09-16T10:22:57.012657+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:22:48.200386+0000","last_became_peered":"2024-09-16T10:22:48.200386+0000","last_unstale":"2024-09-16T10:23:00.019798+0000","last_undegraded":"2024-09-16T10:23:00.019798+0000","last_fullsized":"2024-09-16T10:23:00.019798+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:41:13.740112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.015354+0000","last_change":"2024-09-16T10:23:00.015467+0000","last_active":"2024-09-16T10:23:00.015354+0000","last_peered":"2024-09-16T10:22:57.018973+0000","last_clean":"2024-09-16T10:22:57.018973+0000","last_became_active":"2024-09-16T10:22:57.016872+0000","last_became_peered":"2024-09-16T10:22:57.016872+0000","last_unstale":"2024-09-16T10:23:00.015354+0000","last_undegraded":"2024-09-16T10:23:00.015354+0000","last_fullsized":"2024-09-16T10:23:00.015354+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:22.008355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"22'2","reported_seq":40,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.016084+0000","last_change":"2024-09-16T10:23:00.016237+0000","last_active":"2024-09-16T10:23:00.016084+0000","last_peered":"2024-09-16T10:22:57.013470+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:22:48.200268+0000","last_became_peered":"2024-09-16T10:22:48.200268+0000","last_unstale":"2024-09-16T10:23:00.016084+0000","last_undegraded":"2024-09-16T10:23:00.016084+0000","last_fullsized":"2024-09-16T10:23:00.016084+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:40.052967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":48,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.014743+0000","last_change":"2024-09-16T10:23:00.014879+0000","last_active":"2024-09-16T10:23:00.014743+0000","last_peered":"2024-09-16T10:22:57.012319+0000","last_clean":"2024-09-16T10:22:18.022823+0000","last_became_active":"2024-09-16T10:22:48.198525+0000","last_became_peered":"2024-09-16T10:22:48.198525+0000","last_unstale":"2024-09-16T10:23:00.014743+0000","last_undegraded":"2024-09-16T10:23:00.014743+0000","last_fullsized":"2024-09-16T10:23:00.014743+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:30:49.335124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":65,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.015670+0000","last_change":"2024-09-16T10:23:00.015812+0000","last_active":"2024-09-16T10:23:00.015670+0000","last_peered":"2024-09-16T10:22:57.016621+0000","last_clean":"2024-09-16T10:22:57.016621+0000","last_became_active":"2024-09-16T10:22:57.014921+0000","last_became_peered":"2024-09-16T10:22:57.014921+0000","last_unstale":"2024-09-16T10:23:00.015670+0000","last_undegraded":"2024-09-16T10:23:00.015670+0000","last_fullsized":"2024-09-16T10:23:00.015670+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:41.258884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'9","reported_seq":49,"reported_epoch":33,"state":"peering","last_fresh":"2024-09-16T10:23:00.014110+0000","last_change":"2024-09-16T10:23:00.014269+0000","last_active":"2024-09-16T10:23:00.014110+0000","last_peered":"2024-09-16T10:22:57.013536+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:22:48.198034+0000","last_became_peered":"2024-09-16T10:22:48.198034+0000","last_unstale":"2024-09-16T10:23:00.014110+0000","last_undegraded":"2024-09-16T10:23:00.014110+0000","last_fullsized":"2024-09-16T10:23:00.014110+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:42.626750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":13524,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":18354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31900,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":278528,"data_stored":87147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32370352},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215112,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":312,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":319488,"data_stored":91514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":32304808},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":4830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:02.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:23:02.865 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:02.865 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":71,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":2,"osd_up_since":1726482175,"num_in_osds":2,"osd_in_since":1726482180,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"peering","count":14},{"state_name":"remapped+peering","count":10}],"num_pgs":24,"num_pools":2,"num_objects":30,"data_bytes":13543,"bytes_used":65306624,"bytes_avail":193208221696,"bytes_total":193273528320,"inactive_pgs_ratio":1},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T10:22:58.555770+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:23:02.877 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '3f765c65-edd5-4e6d-930a-33ce25cc666f', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 71, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 35, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726482175, 'num_in_osds': 2, 'osd_in_since': 1726482180, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'peering', 'count': 14}, {'state_name': 'remapped+peering', 'count': 10}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 30, 'data_bytes': 13543, 'bytes_used': 65306624, 'bytes_avail': 193208221696, 'bytes_total': 193273528320, 'inactive_pgs_ratio': 1}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 4, 'modified': '2024-09-16T10:22:58.555770+0000', 'services': {'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '2': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-16T10:23:05.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:06.175 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:06.175 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:06.187 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-16T10:23:04.557049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13543,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":18,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":36,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":63796,"kb_used_data":600,"kb_used_omap":8,"kb_used_meta":63159,"kb_avail":188679884,"statfs":{"total":193273528320,"available":193208201216,"internally_reserved":0,"allocated":614400,"data_stored":184289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8872,"internal_metadata":64675160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-23,"num_objects_misplaced":0,"num_objects_unfound":-9,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001364"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":43,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.018447+0000","last_change":"2024-09-16T10:23:01.017258+0000","last_active":"2024-09-16T10:23:01.018447+0000","last_peered":"2024-09-16T10:23:01.018447+0000","last_clean":"2024-09-16T10:23:01.018447+0000","last_became_active":"2024-09-16T10:23:01.016994+0000","last_became_peered":"2024-09-16T10:23:01.016994+0000","last_unstale":"2024-09-16T10:23:01.018447+0000","last_undegraded":"2024-09-16T10:23:01.018447+0000","last_fullsized":"2024-09-16T10:23:01.018447+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:53.799911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'4","reported_seq":44,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.014818+0000","last_change":"2024-09-16T10:23:00.014976+0000","last_active":"2024-09-16T10:23:00.014818+0000","last_peered":"2024-09-16T10:22:57.015526+0000","last_clean":"2024-09-16T10:22:25.033354+0000","last_became_active":"2024-09-16T10:22:57.015078+0000","last_became_peered":"2024-09-16T10:22:57.015078+0000","last_unstale":"2024-09-16T10:23:00.014818+0000","last_undegraded":"2024-09-16T10:23:00.014818+0000","last_fullsized":"2024-09-16T10:23:00.014818+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:31:05.684252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":57,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.015491+0000","last_change":"2024-09-16T10:23:00.015653+0000","last_active":"2024-09-16T10:23:00.015491+0000","last_peered":"2024-09-16T10:22:57.019006+0000","last_clean":"2024-09-16T10:22:25.588129+0000","last_became_active":"2024-09-16T10:22:57.018710+0000","last_became_peered":"2024-09-16T10:22:57.018710+0000","last_unstale":"2024-09-16T10:23:00.015491+0000","last_undegraded":"2024-09-16T10:23:00.015491+0000","last_fullsized":"2024-09-16T10:23:00.015491+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:04.949838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.015493+0000","last_change":"2024-09-16T10:23:00.015627+0000","last_active":"2024-09-16T10:23:00.015493+0000","last_peered":"2024-09-16T10:22:57.016848+0000","last_clean":"2024-09-16T10:22:57.016848+0000","last_became_active":"2024-09-16T10:22:57.014864+0000","last_became_peered":"2024-09-16T10:22:57.014864+0000","last_unstale":"2024-09-16T10:23:00.015493+0000","last_undegraded":"2024-09-16T10:23:00.015493+0000","last_fullsized":"2024-09-16T10:23:00.015493+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:32:46.368618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"22'5","reported_seq":63,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.016337+0000","last_change":"2024-09-16T10:23:01.016337+0000","last_active":"2024-09-16T10:23:01.016337+0000","last_peered":"2024-09-16T10:23:01.016337+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:01.016337+0000","last_undegraded":"2024-09-16T10:23:01.010093+0000","last_fullsized":"2024-09-16T10:23:01.016337+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:12:00.306083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":50,"reported_epoch":34,"state":"active+recovering","last_fresh":"2024-09-16T10:23:01.017010+0000","last_change":"2024-09-16T10:23:02.010889+0000","last_active":"2024-09-16T10:23:01.017010+0000","last_peered":"2024-09-16T10:23:01.017010+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:23:01.014691+0000","last_became_peered":"2024-09-16T10:23:01.014691+0000","last_unstale":"2024-09-16T10:23:01.017010+0000","last_undegraded":"2024-09-16T10:23:01.009924+0000","last_fullsized":"2024-09-16T10:23:01.017010+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:33:37.234042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"22'4","reported_seq":50,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.018034+0000","last_change":"2024-09-16T10:23:01.018034+0000","last_active":"2024-09-16T10:23:01.018034+0000","last_peered":"2024-09-16T10:23:01.018034+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:23:01.017728+0000","last_became_peered":"2024-09-16T10:23:01.017728+0000","last_unstale":"2024-09-16T10:23:01.018034+0000","last_undegraded":"2024-09-16T10:23:01.011997+0000","last_fullsized":"2024-09-16T10:23:01.018034+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:40:26.023040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'1","reported_seq":59,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.016059+0000","last_change":"2024-09-16T10:23:01.016059+0000","last_active":"2024-09-16T10:23:01.016059+0000","last_peered":"2024-09-16T10:23:01.016059+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:01.016059+0000","last_undegraded":"2024-09-16T10:23:01.009538+0000","last_fullsized":"2024-09-16T10:23:01.016059+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:58.055496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017433+0000","last_change":"2024-09-16T10:23:01.016598+0000","last_active":"2024-09-16T10:23:01.017433+0000","last_peered":"2024-09-16T10:23:01.017433+0000","last_clean":"2024-09-16T10:23:01.017433+0000","last_became_active":"2024-09-16T10:23:01.016110+0000","last_became_peered":"2024-09-16T10:23:01.016110+0000","last_unstale":"2024-09-16T10:23:01.017433+0000","last_undegraded":"2024-09-16T10:23:01.017433+0000","last_fullsized":"2024-09-16T10:23:01.017433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:12.204767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"22'5","reported_seq":59,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.014865+0000","last_change":"2024-09-16T10:23:00.015016+0000","last_active":"2024-09-16T10:23:00.014865+0000","last_peered":"2024-09-16T10:22:57.017568+0000","last_clean":"2024-09-16T10:22:25.638957+0000","last_became_active":"2024-09-16T10:22:57.017150+0000","last_became_peered":"2024-09-16T10:22:57.017150+0000","last_unstale":"2024-09-16T10:23:00.014865+0000","last_undegraded":"2024-09-16T10:23:00.014865+0000","last_fullsized":"2024-09-16T10:23:00.014865+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:31:56.901192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.013228+0000","last_change":"2024-09-16T10:23:00.013366+0000","last_active":"2024-09-16T10:23:00.013228+0000","last_peered":"2024-09-16T10:22:57.018488+0000","last_clean":"2024-09-16T10:22:57.018488+0000","last_became_active":"2024-09-16T10:22:57.016819+0000","last_became_peered":"2024-09-16T10:22:57.016819+0000","last_unstale":"2024-09-16T10:23:00.013228+0000","last_undegraded":"2024-09-16T10:23:00.013228+0000","last_fullsized":"2024-09-16T10:23:00.013228+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:17:29.249666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"22'6","reported_seq":47,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015673+0000","last_change":"2024-09-16T10:23:01.015673+0000","last_active":"2024-09-16T10:23:01.015673+0000","last_peered":"2024-09-16T10:23:01.015673+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:23:01.015452+0000","last_became_peered":"2024-09-16T10:23:01.015452+0000","last_unstale":"2024-09-16T10:23:01.015673+0000","last_undegraded":"2024-09-16T10:23:01.010375+0000","last_fullsized":"2024-09-16T10:23:01.015673+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:41:44.348135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":53,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015931+0000","last_change":"2024-09-16T10:23:01.015931+0000","last_active":"2024-09-16T10:23:01.015931+0000","last_peered":"2024-09-16T10:23:01.015931+0000","last_clean":"2024-09-16T10:22:36.125521+0000","last_became_active":"2024-09-16T10:23:01.015674+0000","last_became_peered":"2024-09-16T10:23:01.015674+0000","last_unstale":"2024-09-16T10:23:01.015931+0000","last_undegraded":"2024-09-16T10:23:01.009899+0000","last_fullsized":"2024-09-16T10:23:01.015931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:04.682617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'3","reported_seq":48,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.014445+0000","last_change":"2024-09-16T10:23:00.014887+0000","last_active":"2024-09-16T10:23:00.014445+0000","last_peered":"2024-09-16T10:22:57.017434+0000","last_clean":"2024-09-16T10:22:24.847222+0000","last_became_active":"2024-09-16T10:22:57.017138+0000","last_became_peered":"2024-09-16T10:22:57.017138+0000","last_unstale":"2024-09-16T10:23:00.014445+0000","last_undegraded":"2024-09-16T10:23:00.014445+0000","last_fullsized":"2024-09-16T10:23:00.014445+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:35:14.811553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":48,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.014495+0000","last_change":"2024-09-16T10:23:00.014643+0000","last_active":"2024-09-16T10:23:00.014495+0000","last_peered":"2024-09-16T10:22:57.017452+0000","last_clean":"2024-09-16T10:22:57.017452+0000","last_became_active":"2024-09-16T10:22:57.015587+0000","last_became_peered":"2024-09-16T10:22:57.015587+0000","last_unstale":"2024-09-16T10:23:00.014495+0000","last_undegraded":"2024-09-16T10:23:00.014495+0000","last_fullsized":"2024-09-16T10:23:00.014495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:31:34.592553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":49,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.013209+0000","last_change":"2024-09-16T10:23:00.013369+0000","last_active":"2024-09-16T10:23:00.013209+0000","last_peered":"2024-09-16T10:22:57.015281+0000","last_clean":"2024-09-16T10:22:25.286289+0000","last_became_active":"2024-09-16T10:22:57.014982+0000","last_became_peered":"2024-09-16T10:22:57.014982+0000","last_unstale":"2024-09-16T10:23:00.013209+0000","last_undegraded":"2024-09-16T10:23:00.013209+0000","last_fullsized":"2024-09-16T10:23:00.013209+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:10:01.336630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"22'1","reported_seq":46,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.018002+0000","last_change":"2024-09-16T10:23:01.018002+0000","last_active":"2024-09-16T10:23:01.018002+0000","last_peered":"2024-09-16T10:23:01.018002+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:23:01.017742+0000","last_became_peered":"2024-09-16T10:23:01.017742+0000","last_unstale":"2024-09-16T10:23:01.018002+0000","last_undegraded":"2024-09-16T10:23:01.012206+0000","last_fullsized":"2024-09-16T10:23:01.018002+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:53.161791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.016349+0000","last_change":"2024-09-16T10:23:01.014951+0000","last_active":"2024-09-16T10:23:01.016349+0000","last_peered":"2024-09-16T10:23:01.016349+0000","last_clean":"2024-09-16T10:23:01.016349+0000","last_became_active":"2024-09-16T10:23:01.014674+0000","last_became_peered":"2024-09-16T10:23:01.014674+0000","last_unstale":"2024-09-16T10:23:01.016349+0000","last_undegraded":"2024-09-16T10:23:01.016349+0000","last_fullsized":"2024-09-16T10:23:01.016349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:05.742015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"22'5","reported_seq":51,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.017334+0000","last_change":"2024-09-16T10:23:01.017334+0000","last_active":"2024-09-16T10:23:01.017334+0000","last_peered":"2024-09-16T10:23:01.017334+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:23:01.016901+0000","last_became_peered":"2024-09-16T10:23:01.016901+0000","last_unstale":"2024-09-16T10:23:01.017334+0000","last_undegraded":"2024-09-16T10:23:01.010804+0000","last_fullsized":"2024-09-16T10:23:01.017334+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:21:36.430119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.015354+0000","last_change":"2024-09-16T10:23:00.015467+0000","last_active":"2024-09-16T10:23:00.015354+0000","last_peered":"2024-09-16T10:22:57.018973+0000","last_clean":"2024-09-16T10:22:57.018973+0000","last_became_active":"2024-09-16T10:22:57.016872+0000","last_became_peered":"2024-09-16T10:22:57.016872+0000","last_unstale":"2024-09-16T10:23:00.015354+0000","last_undegraded":"2024-09-16T10:23:00.015354+0000","last_fullsized":"2024-09-16T10:23:00.015354+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:51:22.008355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"22'2","reported_seq":47,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.017225+0000","last_change":"2024-09-16T10:23:01.017225+0000","last_active":"2024-09-16T10:23:01.017225+0000","last_peered":"2024-09-16T10:23:01.017225+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:23:01.016945+0000","last_became_peered":"2024-09-16T10:23:01.016945+0000","last_unstale":"2024-09-16T10:23:01.017225+0000","last_undegraded":"2024-09-16T10:23:01.011253+0000","last_fullsized":"2024-09-16T10:23:01.017225+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:36:48.394527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017286+0000","last_change":"2024-09-16T10:23:01.015639+0000","last_active":"2024-09-16T10:23:01.017286+0000","last_peered":"2024-09-16T10:23:01.017286+0000","last_clean":"2024-09-16T10:23:01.017286+0000","last_became_active":"2024-09-16T10:23:01.015380+0000","last_became_peered":"2024-09-16T10:23:01.015380+0000","last_unstale":"2024-09-16T10:23:01.017286+0000","last_undegraded":"2024-09-16T10:23:01.017286+0000","last_fullsized":"2024-09-16T10:23:01.017286+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:03.395240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":65,"reported_epoch":33,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:00.015670+0000","last_change":"2024-09-16T10:23:00.015812+0000","last_active":"2024-09-16T10:23:00.015670+0000","last_peered":"2024-09-16T10:22:57.016621+0000","last_clean":"2024-09-16T10:22:57.016621+0000","last_became_active":"2024-09-16T10:22:57.014921+0000","last_became_peered":"2024-09-16T10:22:57.014921+0000","last_unstale":"2024-09-16T10:23:00.015670+0000","last_undegraded":"2024-09-16T10:23:00.015670+0000","last_fullsized":"2024-09-16T10:23:00.015670+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:41.258884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"22'9","reported_seq":56,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015555+0000","last_change":"2024-09-16T10:23:01.015555+0000","last_active":"2024-09-16T10:23:01.015555+0000","last_peered":"2024-09-16T10:23:01.015555+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:23:01.015304+0000","last_became_peered":"2024-09-16T10:23:01.015304+0000","last_unstale":"2024-09-16T10:23:01.015555+0000","last_undegraded":"2024-09-16T10:23:01.009840+0000","last_fullsized":"2024-09-16T10:23:01.015555+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:52.764019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":13524,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":18,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":18354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31900,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339940,"statfs":{"total":96636764160,"available":96604098560,"internally_reserved":0,"allocated":278528,"data_stored":87147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32370352},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.246}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215113,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31896,"kb_used_data":328,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94339944,"statfs":{"total":96636764160,"available":96604102656,"internally_reserved":0,"allocated":335872,"data_stored":97142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":32304808},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":4830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:06.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:23:06.534 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:06.534 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"18/30 objects unfound (60.000%)","count":18},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34/60 objects degraded (56.667%), 9 pgs degraded","count":9},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":75,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":2,"osd_up_since":1726482175,"num_in_osds":2,"osd_in_since":1726482180,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"remapped+peering","count":10},{"state_name":"active+recovery_wait+degraded","count":9},{"state_name":"active+clean","count":4},{"state_name":"active+recovering","count":1}],"num_pgs":24,"num_pools":2,"num_objects":30,"data_bytes":13543,"bytes_used":65327104,"bytes_avail":193208201216,"bytes_total":193273528320,"inactive_pgs_ratio":0.4166666567325592,"degraded_objects":34,"degraded_total":60,"degraded_ratio":0.56666666666666665,"unfound_objects":18,"unfound_total":30,"unfound_ratio":0.59999999999999998},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T10:22:58.555770+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:23:06.546 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '3f765c65-edd5-4e6d-930a-33ce25cc666f', 'health': {'status': 'HEALTH_WARN', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '18/30 objects unfound (60.000%)', 'count': 18}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 34/60 objects degraded (56.667%), 9 pgs degraded', 'count': 9}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 75, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 35, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726482175, 'num_in_osds': 2, 'osd_in_since': 1726482180, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'remapped+peering', 'count': 10}, {'state_name': 'active+recovery_wait+degraded', 'count': 9}, {'state_name': 'active+clean', 'count': 4}, {'state_name': 'active+recovering', 'count': 1}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 30, 'data_bytes': 13543, 'bytes_used': 65327104, 'bytes_avail': 193208201216, 'bytes_total': 193273528320, 'inactive_pgs_ratio': 0.4166666567325592, 'degraded_objects': 34, 'degraded_total': 60, 'degraded_ratio': 0.5666666666666667, 'unfound_objects': 18, 'unfound_total': 30, 'unfound_ratio': 0.6}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 4, 'modified': '2024-09-16T10:22:58.555770+0000', 'services': {'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '2': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-16T10:23:07.079 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-16T10:23:07.077+0000 7f032577b640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 5 pgs recovery_unfound (PG_DAMAGED) 2024-09-16T10:23:09.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:09.840 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:09.841 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:09.853 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-09-16T10:23:08.558180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13543,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":57,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58,"ondisk_log_size":58,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":63820,"kb_used_data":616,"kb_used_omap":8,"kb_used_meta":63159,"kb_avail":188679860,"statfs":{"total":193273528320,"available":193208176640,"internally_reserved":0,"allocated":630784,"data_stored":189917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8872,"internal_metadata":64675160},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001797"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":43,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.018447+0000","last_change":"2024-09-16T10:23:01.017258+0000","last_active":"2024-09-16T10:23:01.018447+0000","last_peered":"2024-09-16T10:23:01.018447+0000","last_clean":"2024-09-16T10:23:01.018447+0000","last_became_active":"2024-09-16T10:23:01.016994+0000","last_became_peered":"2024-09-16T10:23:01.016994+0000","last_unstale":"2024-09-16T10:23:01.018447+0000","last_undegraded":"2024-09-16T10:23:01.018447+0000","last_fullsized":"2024-09-16T10:23:01.018447+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:53.799911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'4","reported_seq":47,"reported_epoch":35,"state":"active+recovery_unfound","last_fresh":"2024-09-16T10:23:02.017774+0000","last_change":"2024-09-16T10:23:02.017932+0000","last_active":"2024-09-16T10:23:02.017774+0000","last_peered":"2024-09-16T10:23:02.017774+0000","last_clean":"2024-09-16T10:22:25.033354+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:02.017774+0000","last_undegraded":"2024-09-16T10:23:02.008669+0000","last_fullsized":"2024-09-16T10:23:02.017774+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.440596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"22'3","reported_seq":61,"reported_epoch":35,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-16T10:23:02.017458+0000","last_change":"2024-09-16T10:23:05.225578+0000","last_active":"2024-09-16T10:23:02.017458+0000","last_peered":"2024-09-16T10:23:02.017458+0000","last_clean":"2024-09-16T10:22:25.588129+0000","last_became_active":"2024-09-16T10:23:02.013253+0000","last_became_peered":"2024-09-16T10:23:02.013253+0000","last_unstale":"2024-09-16T10:23:02.017458+0000","last_undegraded":"2024-09-16T10:23:02.017458+0000","last_fullsized":"2024-09-16T10:23:02.017458+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:21:14.524853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.014955+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:02.014955+0000","last_peered":"2024-09-16T10:23:02.014955+0000","last_clean":"2024-09-16T10:23:02.014955+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:02.014955+0000","last_undegraded":"2024-09-16T10:23:02.014955+0000","last_fullsized":"2024-09-16T10:23:02.014955+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:59.836622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"22'5","reported_seq":63,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.016337+0000","last_change":"2024-09-16T10:23:01.016337+0000","last_active":"2024-09-16T10:23:01.016337+0000","last_peered":"2024-09-16T10:23:01.016337+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:01.016337+0000","last_undegraded":"2024-09-16T10:23:01.010093+0000","last_fullsized":"2024-09-16T10:23:01.016337+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:12:00.306083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"22'3","reported_seq":50,"reported_epoch":34,"state":"active+recovering","last_fresh":"2024-09-16T10:23:01.017010+0000","last_change":"2024-09-16T10:23:02.010889+0000","last_active":"2024-09-16T10:23:01.017010+0000","last_peered":"2024-09-16T10:23:01.017010+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:23:01.014691+0000","last_became_peered":"2024-09-16T10:23:01.014691+0000","last_unstale":"2024-09-16T10:23:01.017010+0000","last_undegraded":"2024-09-16T10:23:01.009924+0000","last_fullsized":"2024-09-16T10:23:01.017010+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:33:37.234042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"22'4","reported_seq":50,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.018034+0000","last_change":"2024-09-16T10:23:01.018034+0000","last_active":"2024-09-16T10:23:01.018034+0000","last_peered":"2024-09-16T10:23:01.018034+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:23:01.017728+0000","last_became_peered":"2024-09-16T10:23:01.017728+0000","last_unstale":"2024-09-16T10:23:01.018034+0000","last_undegraded":"2024-09-16T10:23:01.011997+0000","last_fullsized":"2024-09-16T10:23:01.018034+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:40:26.023040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"22'1","reported_seq":59,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.016059+0000","last_change":"2024-09-16T10:23:01.016059+0000","last_active":"2024-09-16T10:23:01.016059+0000","last_peered":"2024-09-16T10:23:01.016059+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:01.016059+0000","last_undegraded":"2024-09-16T10:23:01.009538+0000","last_fullsized":"2024-09-16T10:23:01.016059+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:58.055496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017433+0000","last_change":"2024-09-16T10:23:01.016598+0000","last_active":"2024-09-16T10:23:01.017433+0000","last_peered":"2024-09-16T10:23:01.017433+0000","last_clean":"2024-09-16T10:23:01.017433+0000","last_became_active":"2024-09-16T10:23:01.016110+0000","last_became_peered":"2024-09-16T10:23:01.016110+0000","last_unstale":"2024-09-16T10:23:01.017433+0000","last_undegraded":"2024-09-16T10:23:01.017433+0000","last_fullsized":"2024-09-16T10:23:01.017433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:12.204767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"22'5","reported_seq":63,"reported_epoch":35,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-16T10:23:02.018379+0000","last_change":"2024-09-16T10:23:05.225522+0000","last_active":"2024-09-16T10:23:02.018379+0000","last_peered":"2024-09-16T10:23:02.018379+0000","last_clean":"2024-09-16T10:22:25.638957+0000","last_became_active":"2024-09-16T10:23:02.014504+0000","last_became_peered":"2024-09-16T10:23:02.014504+0000","last_unstale":"2024-09-16T10:23:02.018379+0000","last_undegraded":"2024-09-16T10:23:02.018379+0000","last_fullsized":"2024-09-16T10:23:02.018379+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:29.772851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013960+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:02.013960+0000","last_peered":"2024-09-16T10:23:02.013960+0000","last_clean":"2024-09-16T10:23:02.013960+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:02.013960+0000","last_undegraded":"2024-09-16T10:23:02.013960+0000","last_fullsized":"2024-09-16T10:23:02.013960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"22'6","reported_seq":47,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015673+0000","last_change":"2024-09-16T10:23:01.015673+0000","last_active":"2024-09-16T10:23:01.015673+0000","last_peered":"2024-09-16T10:23:01.015673+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:23:01.015452+0000","last_became_peered":"2024-09-16T10:23:01.015452+0000","last_unstale":"2024-09-16T10:23:01.015673+0000","last_undegraded":"2024-09-16T10:23:01.010375+0000","last_fullsized":"2024-09-16T10:23:01.015673+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:41:44.348135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":53,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015931+0000","last_change":"2024-09-16T10:23:01.015931+0000","last_active":"2024-09-16T10:23:01.015931+0000","last_peered":"2024-09-16T10:23:01.015931+0000","last_clean":"2024-09-16T10:22:36.125521+0000","last_became_active":"2024-09-16T10:23:01.015674+0000","last_became_peered":"2024-09-16T10:23:01.015674+0000","last_unstale":"2024-09-16T10:23:01.015931+0000","last_undegraded":"2024-09-16T10:23:01.009899+0000","last_fullsized":"2024-09-16T10:23:01.015931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:04.682617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"22'3","reported_seq":51,"reported_epoch":35,"state":"active+recovery_unfound","last_fresh":"2024-09-16T10:23:02.016705+0000","last_change":"2024-09-16T10:23:02.016983+0000","last_active":"2024-09-16T10:23:02.016705+0000","last_peered":"2024-09-16T10:23:02.016705+0000","last_clean":"2024-09-16T10:22:24.847222+0000","last_became_active":"2024-09-16T10:23:02.012723+0000","last_became_peered":"2024-09-16T10:23:02.012723+0000","last_unstale":"2024-09-16T10:23:02.016705+0000","last_undegraded":"2024-09-16T10:23:02.009156+0000","last_fullsized":"2024-09-16T10:23:02.016705+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:24:06.439541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.015929+0000","last_change":"2024-09-16T10:23:02.014534+0000","last_active":"2024-09-16T10:23:02.015929+0000","last_peered":"2024-09-16T10:23:02.015929+0000","last_clean":"2024-09-16T10:23:02.015929+0000","last_became_active":"2024-09-16T10:23:02.014138+0000","last_became_peered":"2024-09-16T10:23:02.014138+0000","last_unstale":"2024-09-16T10:23:02.015929+0000","last_undegraded":"2024-09-16T10:23:02.015929+0000","last_fullsized":"2024-09-16T10:23:02.015929+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:19:32.676442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"22'4","reported_seq":52,"reported_epoch":35,"state":"active+recovery_unfound","last_fresh":"2024-09-16T10:23:02.014480+0000","last_change":"2024-09-16T10:23:02.014847+0000","last_active":"2024-09-16T10:23:02.014480+0000","last_peered":"2024-09-16T10:23:02.014480+0000","last_clean":"2024-09-16T10:22:25.286289+0000","last_became_active":"2024-09-16T10:23:02.012604+0000","last_became_peered":"2024-09-16T10:23:02.012604+0000","last_unstale":"2024-09-16T10:23:02.014480+0000","last_undegraded":"2024-09-16T10:23:02.008188+0000","last_fullsized":"2024-09-16T10:23:02.014480+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:09:46.793580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"22'1","reported_seq":46,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.018002+0000","last_change":"2024-09-16T10:23:01.018002+0000","last_active":"2024-09-16T10:23:01.018002+0000","last_peered":"2024-09-16T10:23:01.018002+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:23:01.017742+0000","last_became_peered":"2024-09-16T10:23:01.017742+0000","last_unstale":"2024-09-16T10:23:01.018002+0000","last_undegraded":"2024-09-16T10:23:01.012206+0000","last_fullsized":"2024-09-16T10:23:01.018002+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:53.161791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.016349+0000","last_change":"2024-09-16T10:23:01.014951+0000","last_active":"2024-09-16T10:23:01.016349+0000","last_peered":"2024-09-16T10:23:01.016349+0000","last_clean":"2024-09-16T10:23:01.016349+0000","last_became_active":"2024-09-16T10:23:01.014674+0000","last_became_peered":"2024-09-16T10:23:01.014674+0000","last_unstale":"2024-09-16T10:23:01.016349+0000","last_undegraded":"2024-09-16T10:23:01.016349+0000","last_fullsized":"2024-09-16T10:23:01.016349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:05.742015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"22'5","reported_seq":51,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.017334+0000","last_change":"2024-09-16T10:23:01.017334+0000","last_active":"2024-09-16T10:23:01.017334+0000","last_peered":"2024-09-16T10:23:01.017334+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:23:01.016901+0000","last_became_peered":"2024-09-16T10:23:01.016901+0000","last_unstale":"2024-09-16T10:23:01.017334+0000","last_undegraded":"2024-09-16T10:23:01.010804+0000","last_fullsized":"2024-09-16T10:23:01.017334+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:21:36.430119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013918+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:02.013918+0000","last_peered":"2024-09-16T10:23:02.013918+0000","last_clean":"2024-09-16T10:23:02.013918+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:02.013918+0000","last_undegraded":"2024-09-16T10:23:02.013918+0000","last_fullsized":"2024-09-16T10:23:02.013918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'2","reported_seq":47,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.017225+0000","last_change":"2024-09-16T10:23:01.017225+0000","last_active":"2024-09-16T10:23:01.017225+0000","last_peered":"2024-09-16T10:23:01.017225+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:23:01.016945+0000","last_became_peered":"2024-09-16T10:23:01.016945+0000","last_unstale":"2024-09-16T10:23:01.017225+0000","last_undegraded":"2024-09-16T10:23:01.011253+0000","last_fullsized":"2024-09-16T10:23:01.017225+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:36:48.394527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017286+0000","last_change":"2024-09-16T10:23:01.015639+0000","last_active":"2024-09-16T10:23:01.017286+0000","last_peered":"2024-09-16T10:23:01.017286+0000","last_clean":"2024-09-16T10:23:01.017286+0000","last_became_active":"2024-09-16T10:23:01.015380+0000","last_became_peered":"2024-09-16T10:23:01.015380+0000","last_unstale":"2024-09-16T10:23:01.017286+0000","last_undegraded":"2024-09-16T10:23:01.017286+0000","last_fullsized":"2024-09-16T10:23:01.017286+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:03.395240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.016743+0000","last_change":"2024-09-16T10:23:02.015813+0000","last_active":"2024-09-16T10:23:02.016743+0000","last_peered":"2024-09-16T10:23:02.016743+0000","last_clean":"2024-09-16T10:23:02.016743+0000","last_became_active":"2024-09-16T10:23:02.015242+0000","last_became_peered":"2024-09-16T10:23:02.015242+0000","last_unstale":"2024-09-16T10:23:02.016743+0000","last_undegraded":"2024-09-16T10:23:02.016743+0000","last_fullsized":"2024-09-16T10:23:02.016743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:34:28.723524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"22'9","reported_seq":56,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015555+0000","last_change":"2024-09-16T10:23:01.015555+0000","last_active":"2024-09-16T10:23:01.015555+0000","last_peered":"2024-09-16T10:23:01.015555+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:23:01.015304+0000","last_became_peered":"2024-09-16T10:23:01.015304+0000","last_unstale":"2024-09-16T10:23:01.015555+0000","last_undegraded":"2024-09-16T10:23:01.009840+0000","last_fullsized":"2024-09-16T10:23:01.015555+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:52.764019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":13524,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":51,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":18354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018885,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31924,"kb_used_data":288,"kb_used_omap":4,"kb_used_meta":31611,"kb_avail":94339916,"statfs":{"total":96636764160,"available":96604073984,"internally_reserved":0,"allocated":294912,"data_stored":92775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32370352},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25600000000000001}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215113,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31896,"kb_used_data":328,"kb_used_omap":4,"kb_used_meta":31547,"kb_avail":94339944,"statfs":{"total":96636764160,"available":96604102656,"internally_reserved":0,"allocated":335872,"data_stored":97142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":32304808},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":4830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:09.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:23:10.198 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:10.198 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/30 objects unfound (90.000%)","count":27},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 5 pgs recovery_unfound","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 52/60 objects degraded (86.667%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":78,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":2,"osd_up_since":1726482175,"num_in_osds":2,"osd_in_since":1726482180,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+degraded","count":9},{"state_name":"active+recovery_unfound","count":3},{"state_name":"active+recovery_unfound+degraded","count":2},{"state_name":"active+recovering","count":1}],"num_pgs":24,"num_pools":2,"num_objects":30,"data_bytes":13543,"bytes_used":65351680,"bytes_avail":193208176640,"bytes_total":193273528320,"degraded_objects":52,"degraded_total":60,"degraded_ratio":0.8666666666666667,"unfound_objects":27,"unfound_total":30,"unfound_ratio":0.90000000000000002},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T10:22:58.555770+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:23:10.211 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '3f765c65-edd5-4e6d-930a-33ce25cc666f', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/30 objects unfound (90.000%)', 'count': 27}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 5 pgs recovery_unfound', 'count': 5}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 52/60 objects degraded (86.667%), 11 pgs degraded', 'count': 11}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 78, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 35, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726482175, 'num_in_osds': 2, 'osd_in_since': 1726482180, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 9}, {'state_name': 'active+recovery_wait+degraded', 'count': 9}, {'state_name': 'active+recovery_unfound', 'count': 3}, {'state_name': 'active+recovery_unfound+degraded', 'count': 2}, {'state_name': 'active+recovering', 'count': 1}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 30, 'data_bytes': 13543, 'bytes_used': 65351680, 'bytes_avail': 193208176640, 'bytes_total': 193273528320, 'degraded_objects': 52, 'degraded_total': 60, 'degraded_ratio': 0.8666666666666667, 'unfound_objects': 27, 'unfound_total': 30, 'unfound_ratio': 0.9}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 4, 'modified': '2024-09-16T10:22:58.555770+0000', 'services': {'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '2': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-16T10:23:10.211 INFO:tasks.rep_lost_unfound_delete.ceph_manager:active! 2024-09-16T10:23:10.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T10:23:10.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T10:23:10.348 INFO:teuthology.orchestra.run.smithi050.stdout:103079215115 2024-09-16T10:23:10.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:23:10.349 INFO:teuthology.orchestra.run.smithi050.stdout:128849018887 2024-09-16T10:23:10.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:23:10.729 INFO:teuthology.orchestra.run.smithi050.stdout:103079215113 2024-09-16T10:23:10.732 INFO:teuthology.orchestra.run.smithi050.stdout:128849018885 2024-09-16T10:23:10.743 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 103079215115 got 103079215113 for osd.0 2024-09-16T10:23:10.745 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 128849018887 got 128849018885 for osd.2 2024-09-16T10:23:11.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:23:11.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:23:12.044 INFO:teuthology.orchestra.run.smithi050.stdout:128849018887 2024-09-16T10:23:12.047 INFO:teuthology.orchestra.run.smithi050.stdout:103079215115 2024-09-16T10:23:12.057 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 128849018887 got 128849018887 for osd.2 2024-09-16T10:23:12.058 DEBUG:teuthology.parallel:result is None 2024-09-16T10:23:12.059 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 103079215115 got 103079215115 for osd.0 2024-09-16T10:23:12.060 DEBUG:teuthology.parallel:result is None 2024-09-16T10:23:12.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:23:12.449 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:12.449 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/30 objects unfound (90.000%)","count":27},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 5 pgs recovery_unfound","count":5},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 52/60 objects degraded (86.667%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":81,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":3,"num_up_osds":2,"osd_up_since":1726482175,"num_in_osds":2,"osd_in_since":1726482180,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+degraded","count":9},{"state_name":"active+recovery_unfound","count":3},{"state_name":"active+recovery_unfound+degraded","count":2},{"state_name":"active+recovering","count":1}],"num_pgs":24,"num_pools":2,"num_objects":30,"data_bytes":13543,"bytes_used":65351680,"bytes_avail":193208176640,"bytes_total":193273528320,"degraded_objects":52,"degraded_total":60,"degraded_ratio":0.8666666666666667,"unfound_objects":27,"unfound_total":30,"unfound_ratio":0.90000000000000002},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-16T10:22:58.555770+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:23:12.462 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'fsid': '3f765c65-edd5-4e6d-930a-33ce25cc666f', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/30 objects unfound (90.000%)', 'count': 27}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 5 pgs recovery_unfound', 'count': 5}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 52/60 objects degraded (86.667%), 11 pgs degraded', 'count': 11}, 'muted': False}}, 'mutes': []}, 'election_epoch': 8, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 81, 'monmap': {'epoch': 1, 'min_mon_release_name': 'reef', 'num_mons': 3}, 'osdmap': {'epoch': 35, 'num_osds': 3, 'num_up_osds': 2, 'osd_up_since': 1726482175, 'num_in_osds': 2, 'osd_in_since': 1726482180, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 9}, {'state_name': 'active+recovery_wait+degraded', 'count': 9}, {'state_name': 'active+recovery_unfound', 'count': 3}, {'state_name': 'active+recovery_unfound+degraded', 'count': 2}, {'state_name': 'active+recovering', 'count': 1}], 'num_pgs': 24, 'num_pools': 2, 'num_objects': 30, 'data_bytes': 13543, 'bytes_used': 65351680, 'bytes_avail': 193208176640, 'bytes_total': 193273528320, 'degraded_objects': 52, 'degraded_total': 60, 'degraded_ratio': 0.8666666666666667, 'unfound_objects': 27, 'unfound_total': 30, 'unfound_ratio': 0.9}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 4, 'modified': '2024-09-16T10:22:58.555770+0000', 'services': {'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '2': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-16T10:23:12.462 INFO:tasks.rep_lost_unfound_delete:there are 27 unfound objects 2024-09-16T10:23:12.462 DEBUG:teuthology.orchestra.run.smithi050:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unfounddel_pool -t 20 bench 240 write' 2024-09-16T10:23:12.514 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:hints = 1 2024-09-16T10:23:12.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Maintaining 20 concurrent writes of 4096 bytes to objects of size 4096 for up to 240 seconds or 0 objects 2024-09-16T10:23:12.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Object prefix: benchmark_data_smithi050_42850 2024-09-16T10:23:12.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:23:12.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 0 0 0 0 0 0 - 0 2024-09-16T10:23:13.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 1 20 3425 3405 13.3 13.3008 0.00784421 0.00585239 2024-09-16T10:23:14.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 2 20 6633 6613 12.9151 12.5312 0.00620135 0.00603268 2024-09-16T10:23:15.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 3 20 9667 9647 12.5603 11.8516 0.00754531 0.00621038 2024-09-16T10:23:16.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 4 20 12557 12537 12.2422 11.2891 0.00637847 0.00637185 2024-09-16T10:23:17.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 5 20 15341 15321 11.9685 10.875 0.00828557 0.0065206 2024-09-16T10:23:18.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 6 20 17967 17947 11.6833 10.2578 0.00719438 0.00667918 2024-09-16T10:23:19.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 7 20 20497 20477 11.426 9.88281 0.00693545 0.00682903 2024-09-16T10:23:20.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 8 20 22512 22492 10.9814 7.87109 0.00196251 0.00695473 2024-09-16T10:23:21.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 9 20 23872 23852 10.3515 5.3125 0.00549126 0.00754082 2024-09-16T10:23:22.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:22.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 10 20 26077 26057 10.1776 8.61328 0.0147478 0.00766775 2024-09-16T10:23:22.983 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:22.983 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:23.003 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-16T10:23:22.562032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":84141287,"num_objects":20569,"num_object_clones":0,"num_object_copies":41138,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":20569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20596,"num_write_kb":82204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20597,"ondisk_log_size":20597,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":299628,"kb_used_data":160156,"kb_used_omap":8,"kb_used_meta":139447,"kb_avail":188444052,"statfs":{"total":193273528320,"available":192966709248,"internally_reserved":0,"allocated":163999744,"data_stored":163558877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8872,"internal_metadata":142794072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,14,10,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":84127744,"num_objects":20539,"num_object_clones":0,"num_object_copies":41078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20539,"num_write_kb":82156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003300"},"pg_stats":[{"pgid":"2.f","version":"35'1221","reported_seq":1263,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:19.121702+0000","last_change":"2024-09-16T10:23:01.017258+0000","last_active":"2024-09-16T10:23:19.121702+0000","last_peered":"2024-09-16T10:23:19.121702+0000","last_clean":"2024-09-16T10:23:19.121702+0000","last_became_active":"2024-09-16T10:23:01.016994+0000","last_became_peered":"2024-09-16T10:23:01.016994+0000","last_unstale":"2024-09-16T10:23:19.121702+0000","last_undegraded":"2024-09-16T10:23:19.121702+0000","last_fullsized":"2024-09-16T10:23:19.121702+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1221,"log_dups_size":0,"ondisk_log_size":1221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:53.799911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5001216,"num_objects":1221,"num_object_clones":0,"num_object_copies":2442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1221,"num_write_kb":4884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'1372","reported_seq":1414,"reported_epoch":35,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-16T10:23:20.231150+0000","last_change":"2024-09-16T10:23:12.522985+0000","last_active":"2024-09-16T10:23:20.231150+0000","last_peered":"2024-09-16T10:23:20.231150+0000","last_clean":"2024-09-16T10:22:25.033354+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:20.231150+0000","last_undegraded":"2024-09-16T10:23:02.017932+0000","last_fullsized":"2024-09-16T10:23:20.231150+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1372,"log_dups_size":0,"ondisk_log_size":1372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:13:02.440596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5604294,"num_objects":1370,"num_object_clones":0,"num_object_copies":2740,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1372,"num_write_kb":5476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1370}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'1409","reported_seq":1467,"reported_epoch":35,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-16T10:23:20.224104+0000","last_change":"2024-09-16T10:23:05.225578+0000","last_active":"2024-09-16T10:23:20.224104+0000","last_peered":"2024-09-16T10:23:20.224104+0000","last_clean":"2024-09-16T10:22:25.588129+0000","last_became_active":"2024-09-16T10:23:02.013253+0000","last_became_peered":"2024-09-16T10:23:02.013253+0000","last_unstale":"2024-09-16T10:23:20.224104+0000","last_undegraded":"2024-09-16T10:23:02.017458+0000","last_fullsized":"2024-09-16T10:23:20.224104+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1409,"log_dups_size":0,"ondisk_log_size":1409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:21:14.524853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5759459,"num_objects":1407,"num_object_clones":0,"num_object_copies":2814,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1409,"num_write_kb":5626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1407}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'1389","reported_seq":1435,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:20.224124+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:20.224124+0000","last_peered":"2024-09-16T10:23:20.224124+0000","last_clean":"2024-09-16T10:23:20.224124+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:20.224124+0000","last_undegraded":"2024-09-16T10:23:20.224124+0000","last_fullsized":"2024-09-16T10:23:20.224124+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1389,"log_dups_size":0,"ondisk_log_size":1389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:59.836622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5689344,"num_objects":1389,"num_object_clones":0,"num_object_copies":2778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1389,"num_write_kb":5556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'1215","reported_seq":1273,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:19.111273+0000","last_change":"2024-09-16T10:23:01.016337+0000","last_active":"2024-09-16T10:23:19.111273+0000","last_peered":"2024-09-16T10:23:19.111273+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:19.111273+0000","last_undegraded":"2024-09-16T10:23:01.010093+0000","last_fullsized":"2024-09-16T10:23:19.111273+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:12:00.306083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4957126,"num_objects":1212,"num_object_clones":0,"num_object_copies":2424,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1215,"num_write_kb":4844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1212}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'1171","reported_seq":1218,"reported_epoch":35,"state":"active+recovering","last_fresh":"2024-09-16T10:23:19.113723+0000","last_change":"2024-09-16T10:23:02.010889+0000","last_active":"2024-09-16T10:23:19.113723+0000","last_peered":"2024-09-16T10:23:19.113723+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:23:01.014691+0000","last_became_peered":"2024-09-16T10:23:01.014691+0000","last_unstale":"2024-09-16T10:23:19.113723+0000","last_undegraded":"2024-09-16T10:23:19.113723+0000","last_fullsized":"2024-09-16T10:23:19.113723+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:33:37.234042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4785094,"num_objects":1170,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":4675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1170}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'1279","reported_seq":1324,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:19.120358+0000","last_change":"2024-09-16T10:23:01.018034+0000","last_active":"2024-09-16T10:23:19.120358+0000","last_peered":"2024-09-16T10:23:19.120358+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:23:01.017728+0000","last_became_peered":"2024-09-16T10:23:01.017728+0000","last_unstale":"2024-09-16T10:23:19.120358+0000","last_undegraded":"2024-09-16T10:23:01.011997+0000","last_fullsized":"2024-09-16T10:23:19.120358+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1279,"log_dups_size":0,"ondisk_log_size":1279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:40:26.023040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5223366,"num_objects":1277,"num_object_clones":0,"num_object_copies":2554,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1279,"num_write_kb":5104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1277}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'1148","reported_seq":1206,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:19.099765+0000","last_change":"2024-09-16T10:23:01.016059+0000","last_active":"2024-09-16T10:23:19.099765+0000","last_peered":"2024-09-16T10:23:19.099765+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:19.099765+0000","last_undegraded":"2024-09-16T10:23:01.009538+0000","last_fullsized":"2024-09-16T10:23:19.099765+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1148,"log_dups_size":0,"ondisk_log_size":1148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:49:58.055496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4698595,"num_objects":1148,"num_object_clones":0,"num_object_copies":2296,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1148,"num_write_kb":4589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1148}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017433+0000","last_change":"2024-09-16T10:23:01.016598+0000","last_active":"2024-09-16T10:23:01.017433+0000","last_peered":"2024-09-16T10:23:01.017433+0000","last_clean":"2024-09-16T10:23:01.017433+0000","last_became_active":"2024-09-16T10:23:01.016110+0000","last_became_peered":"2024-09-16T10:23:01.016110+0000","last_unstale":"2024-09-16T10:23:01.017433+0000","last_undegraded":"2024-09-16T10:23:01.017433+0000","last_fullsized":"2024-09-16T10:23:01.017433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:12.204767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"35'1385","reported_seq":1443,"reported_epoch":35,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-16T10:23:20.204557+0000","last_change":"2024-09-16T10:23:05.225522+0000","last_active":"2024-09-16T10:23:20.204557+0000","last_peered":"2024-09-16T10:23:20.204557+0000","last_clean":"2024-09-16T10:22:25.638957+0000","last_became_active":"2024-09-16T10:23:02.014504+0000","last_became_peered":"2024-09-16T10:23:02.014504+0000","last_unstale":"2024-09-16T10:23:20.204557+0000","last_undegraded":"2024-09-16T10:23:02.018379+0000","last_fullsized":"2024-09-16T10:23:20.204557+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1385,"log_dups_size":0,"ondisk_log_size":1385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:45:29.772851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5653446,"num_objects":1382,"num_object_clones":0,"num_object_copies":2764,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1385,"num_write_kb":5524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1382}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013960+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:02.013960+0000","last_peered":"2024-09-16T10:23:02.013960+0000","last_clean":"2024-09-16T10:23:02.013960+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:02.013960+0000","last_undegraded":"2024-09-16T10:23:02.013960+0000","last_fullsized":"2024-09-16T10:23:02.013960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'1301","reported_seq":1342,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:19.120801+0000","last_change":"2024-09-16T10:23:01.015673+0000","last_active":"2024-09-16T10:23:19.120801+0000","last_peered":"2024-09-16T10:23:19.120801+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:23:01.015452+0000","last_became_peered":"2024-09-16T10:23:01.015452+0000","last_unstale":"2024-09-16T10:23:19.120801+0000","last_undegraded":"2024-09-16T10:23:01.010375+0000","last_fullsized":"2024-09-16T10:23:19.120801+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1301,"log_dups_size":0,"ondisk_log_size":1301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:41:44.348135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5305286,"num_objects":1297,"num_object_clones":0,"num_object_copies":2594,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1301,"num_write_kb":5184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1297}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":53,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015931+0000","last_change":"2024-09-16T10:23:01.015931+0000","last_active":"2024-09-16T10:23:01.015931+0000","last_peered":"2024-09-16T10:23:01.015931+0000","last_clean":"2024-09-16T10:22:36.125521+0000","last_became_active":"2024-09-16T10:23:01.015674+0000","last_became_peered":"2024-09-16T10:23:01.015674+0000","last_unstale":"2024-09-16T10:23:01.015931+0000","last_undegraded":"2024-09-16T10:23:01.009899+0000","last_fullsized":"2024-09-16T10:23:01.015931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:04.682617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'1358","reported_seq":1406,"reported_epoch":35,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-16T10:23:20.218207+0000","last_change":"2024-09-16T10:23:12.530451+0000","last_active":"2024-09-16T10:23:20.218207+0000","last_peered":"2024-09-16T10:23:20.218207+0000","last_clean":"2024-09-16T10:22:24.847222+0000","last_became_active":"2024-09-16T10:23:02.012723+0000","last_became_peered":"2024-09-16T10:23:02.012723+0000","last_unstale":"2024-09-16T10:23:20.218207+0000","last_undegraded":"2024-09-16T10:23:02.016983+0000","last_fullsized":"2024-09-16T10:23:20.218207+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1358,"log_dups_size":0,"ondisk_log_size":1358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:24:06.439541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5551046,"num_objects":1357,"num_object_clones":0,"num_object_copies":2714,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1358,"num_write_kb":5423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.015929+0000","last_change":"2024-09-16T10:23:02.014534+0000","last_active":"2024-09-16T10:23:02.015929+0000","last_peered":"2024-09-16T10:23:02.015929+0000","last_clean":"2024-09-16T10:23:02.015929+0000","last_became_active":"2024-09-16T10:23:02.014138+0000","last_became_peered":"2024-09-16T10:23:02.014138+0000","last_unstale":"2024-09-16T10:23:02.015929+0000","last_undegraded":"2024-09-16T10:23:02.015929+0000","last_fullsized":"2024-09-16T10:23:02.015929+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:19:32.676442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"35'1394","reported_seq":1442,"reported_epoch":35,"state":"active+recovery_unfound+degraded","last_fresh":"2024-09-16T10:23:20.211672+0000","last_change":"2024-09-16T10:23:12.518815+0000","last_active":"2024-09-16T10:23:20.211672+0000","last_peered":"2024-09-16T10:23:20.211672+0000","last_clean":"2024-09-16T10:22:25.286289+0000","last_became_active":"2024-09-16T10:23:02.012604+0000","last_became_peered":"2024-09-16T10:23:02.012604+0000","last_unstale":"2024-09-16T10:23:20.211672+0000","last_undegraded":"2024-09-16T10:23:02.014847+0000","last_fullsized":"2024-09-16T10:23:20.211672+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1394,"log_dups_size":0,"ondisk_log_size":1394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:09:46.793580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5694406,"num_objects":1392,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1394,"num_write_kb":5563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1392}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'1162","reported_seq":1207,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:19.101677+0000","last_change":"2024-09-16T10:23:01.018002+0000","last_active":"2024-09-16T10:23:19.101677+0000","last_peered":"2024-09-16T10:23:19.101677+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:23:01.017742+0000","last_became_peered":"2024-09-16T10:23:01.017742+0000","last_unstale":"2024-09-16T10:23:19.101677+0000","last_undegraded":"2024-09-16T10:23:01.012206+0000","last_fullsized":"2024-09-16T10:23:19.101677+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1162,"log_dups_size":0,"ondisk_log_size":1162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:01:53.161791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4755939,"num_objects":1162,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1162,"num_write_kb":4645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1162}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.016349+0000","last_change":"2024-09-16T10:23:01.014951+0000","last_active":"2024-09-16T10:23:01.016349+0000","last_peered":"2024-09-16T10:23:01.016349+0000","last_clean":"2024-09-16T10:23:01.016349+0000","last_became_active":"2024-09-16T10:23:01.014674+0000","last_became_peered":"2024-09-16T10:23:01.014674+0000","last_unstale":"2024-09-16T10:23:01.016349+0000","last_undegraded":"2024-09-16T10:23:01.016349+0000","last_fullsized":"2024-09-16T10:23:01.016349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:05.742015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"35'1264","reported_seq":1310,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:19.112625+0000","last_change":"2024-09-16T10:23:01.017334+0000","last_active":"2024-09-16T10:23:19.112625+0000","last_peered":"2024-09-16T10:23:19.112625+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:23:01.016901+0000","last_became_peered":"2024-09-16T10:23:01.016901+0000","last_unstale":"2024-09-16T10:23:19.112625+0000","last_undegraded":"2024-09-16T10:23:01.010804+0000","last_fullsized":"2024-09-16T10:23:19.112625+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1264,"log_dups_size":0,"ondisk_log_size":1264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:21:36.430119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5158313,"num_objects":1262,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":1262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":5040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013918+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:02.013918+0000","last_peered":"2024-09-16T10:23:02.013918+0000","last_clean":"2024-09-16T10:23:02.013918+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:02.013918+0000","last_undegraded":"2024-09-16T10:23:02.013918+0000","last_fullsized":"2024-09-16T10:23:02.013918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'1242","reported_seq":1287,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:19.119867+0000","last_change":"2024-09-16T10:23:01.017225+0000","last_active":"2024-09-16T10:23:19.119867+0000","last_peered":"2024-09-16T10:23:19.119867+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:23:01.016945+0000","last_became_peered":"2024-09-16T10:23:01.016945+0000","last_unstale":"2024-09-16T10:23:19.119867+0000","last_undegraded":"2024-09-16T10:23:01.011253+0000","last_fullsized":"2024-09-16T10:23:19.119867+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1242,"log_dups_size":0,"ondisk_log_size":1242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:36:48.394527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5080006,"num_objects":1242,"num_object_clones":0,"num_object_copies":2484,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1242,"num_write_kb":4962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":1242}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017286+0000","last_change":"2024-09-16T10:23:01.015639+0000","last_active":"2024-09-16T10:23:01.017286+0000","last_peered":"2024-09-16T10:23:01.017286+0000","last_clean":"2024-09-16T10:23:01.017286+0000","last_became_active":"2024-09-16T10:23:01.015380+0000","last_became_peered":"2024-09-16T10:23:01.015380+0000","last_unstale":"2024-09-16T10:23:01.017286+0000","last_undegraded":"2024-09-16T10:23:01.017286+0000","last_fullsized":"2024-09-16T10:23:01.017286+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:03.395240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.016743+0000","last_change":"2024-09-16T10:23:02.015813+0000","last_active":"2024-09-16T10:23:02.016743+0000","last_peered":"2024-09-16T10:23:02.016743+0000","last_clean":"2024-09-16T10:23:02.016743+0000","last_became_active":"2024-09-16T10:23:02.015242+0000","last_became_peered":"2024-09-16T10:23:02.015242+0000","last_unstale":"2024-09-16T10:23:02.016743+0000","last_undegraded":"2024-09-16T10:23:02.016743+0000","last_fullsized":"2024-09-16T10:23:02.016743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:34:28.723524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"35'1284","reported_seq":1331,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:19.113341+0000","last_change":"2024-09-16T10:23:01.015555+0000","last_active":"2024-09-16T10:23:19.113341+0000","last_peered":"2024-09-16T10:23:19.113341+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:23:01.015304+0000","last_became_peered":"2024-09-16T10:23:01.015304+0000","last_unstale":"2024-09-16T10:23:19.113341+0000","last_undegraded":"2024-09-16T10:23:01.009840+0000","last_fullsized":"2024-09-16T10:23:19.113341+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":1284,"log_dups_size":0,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:58:52.764019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5224332,"num_objects":1279,"num_object_clones":0,"num_object_copies":2558,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":1279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1284,"num_write_kb":5107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":1278}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":84141268,"num_objects":20567,"num_object_clones":0,"num_object_copies":41134,"num_objects_missing_on_primary":27,"num_objects_missing":27,"num_objects_degraded":51,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":20567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20594,"num_write_kb":82202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":171225088,"data_stored":171087794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20594,"ondisk_log_size":20594,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018889,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164956,"kb_used_data":89044,"kb_used_omap":4,"kb_used_meta":75899,"kb_avail":94206884,"statfs":{"total":96636764160,"available":96467849216,"internally_reserved":0,"allocated":91181056,"data_stored":90978919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":77721264},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,7,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215117,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":134672,"kb_used_data":71112,"kb_used_omap":4,"kb_used_meta":63547,"kb_avail":94237168,"statfs":{"total":96636764160,"available":96498860032,"internally_reserved":0,"allocated":72818688,"data_stored":72579958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":65072808},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,8,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.182}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":80023552,"data_stored":79987422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":91086848,"data_stored":91086848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:23.004 INFO:tasks.rep_lost_unfound_delete:no unfound in 2.f 2024-09-16T10:23:23.004 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.e state active+recovery_unfound+degraded 2024-09-16T10:23:23.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.e list_unfound '{}' 2024-09-16T10:23:23.226 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:23.226 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:23.226 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:23.226 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:23.226 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:23.226 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 2, 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 2, 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existing_4", 2024-09-16T10:23:23.227 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 1224592654, 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'3", 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "20'1", 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0", 2024-09-16T10:23:23.228 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existing_6", 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 3101443614, 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'4", 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "20'2", 2024-09-16T10:23:23.229 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0", 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "NotRecovering", 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:23.230 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:23.231 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:23.246 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.e on osd.2 2024-09-16T10:23:23.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.e mark_unfound_lost delete 2024-09-16T10:23:23.483 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T10:23:23.510 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.d state active+recovery_unfound+degraded 2024-09-16T10:23:23.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.d list_unfound '{}' 2024-09-16T10:23:23.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 11 20 27964 27944 9.92242 7.37109 0.0119488 0.00786604 2024-09-16T10:23:23.722 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:23.722 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:23.722 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:23.722 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 1, 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 1, 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existed_7", 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:23.723 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 2358579165, 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'3", 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "NotRecovering", 2024-09-16T10:23:23.724 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:23.725 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:23.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:23.725 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:23.725 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:23.725 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:23.725 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:23.725 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:23.725 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:23.744 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.d on osd.2 2024-09-16T10:23:23.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.d mark_unfound_lost delete 2024-09-16T10:23:24.013 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-16T10:23:24.033 INFO:tasks.rep_lost_unfound_delete:no unfound in 2.c 2024-09-16T10:23:24.034 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.b state active+recovery_wait+degraded 2024-09-16T10:23:24.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.b list_unfound '{}' 2024-09-16T10:23:24.228 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:24.228 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:24.228 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:24.228 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:24.228 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:24.228 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 2, 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 2, 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existed_3", 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:24.229 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 943960171, 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'5", 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existing_1", 2024-09-16T10:23:24.230 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 2885884443, 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'4", 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "20'1", 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0", 2024-09-16T10:23:24.231 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "Active", 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:24.232 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:24.248 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.b on osd.0 2024-09-16T10:23:24.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.b mark_unfound_lost delete 2024-09-16T10:23:24.492 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T10:23:24.514 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.a state active+recovering 2024-09-16T10:23:24.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.a list_unfound '{}' 2024-09-16T10:23:24.517 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 12 19 29739 29720 9.67364 6.9375 0.00851201 0.00806822 2024-09-16T10:23:24.730 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:24.730 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 2, 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 2, 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existing_9", 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:24.731 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 1217469130, 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'3", 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "20'1", 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0", 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:24.732 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "new_9", 2024-09-16T10:23:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 170713514, 2024-09-16T10:23:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'2", 2024-09-16T10:23:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:24.733 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "Active", 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:24.734 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:24.749 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.a on osd.0 2024-09-16T10:23:24.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.a mark_unfound_lost delete 2024-09-16T10:23:24.991 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T10:23:25.012 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.9 state active+recovery_wait+degraded 2024-09-16T10:23:25.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.9 list_unfound '{}' 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:25.227 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 2, 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 2, 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existing_5", 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 2998549577, 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:25.228 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'4", 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "20'2", 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0", 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existing_2", 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 4032093161, 2024-09-16T10:23:25.229 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'3", 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "20'1", 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0", 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "Active", 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:25.230 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:25.231 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:25.231 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:25.231 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:25.231 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:25.231 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:25.246 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.9 on osd.0 2024-09-16T10:23:25.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.9 mark_unfound_lost delete 2024-09-16T10:23:25.484 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T10:23:25.503 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.8 state active+recovery_wait+degraded 2024-09-16T10:23:25.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.8 list_unfound '{}' 2024-09-16T10:23:25.515 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 13 20 31539 31519 9.47004 7.02734 0.00647632 0.00824373 2024-09-16T10:23:25.721 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:25.721 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:25.721 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 1, 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 1, 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:25.722 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "new_8", 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 2730327960, 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'1", 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:25.723 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:25.724 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "Active", 2024-09-16T10:23:25.724 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:25.724 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:25.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:25.724 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:25.724 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:25.724 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:25.724 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:25.724 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:25.724 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:25.744 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.8 on osd.0 2024-09-16T10:23:25.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.8 mark_unfound_lost delete 2024-09-16T10:23:26.018 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-16T10:23:26.039 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.0 2024-09-16T10:23:26.039 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.3 state active+recovery_unfound+degraded 2024-09-16T10:23:26.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.3 list_unfound '{}' 2024-09-16T10:23:26.255 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:26.255 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:26.255 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 2, 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 2, 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:26.256 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existing_8", 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 1901787523, 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'5", 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "20'3", 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0", 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:26.257 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existed_1", 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 3122379363, 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'4", 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:26.258 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "NotRecovering", 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:26.259 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:26.278 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.3 on osd.2 2024-09-16T10:23:26.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.3 mark_unfound_lost delete 2024-09-16T10:23:26.516 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 14 20 33306 33286 9.28659 6.90234 0.012451 0.00840746 2024-09-16T10:23:26.551 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T10:23:26.576 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.1 2024-09-16T10:23:26.576 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.2 state active+recovery_wait+degraded 2024-09-16T10:23:26.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.2 list_unfound '{}' 2024-09-16T10:23:26.795 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:26.796 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 2, 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 2, 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existed_6", 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 1708673474, 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:26.797 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'5", 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existed_9", 2024-09-16T10:23:26.798 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 831402034, 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'6", 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:26.799 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "Active", 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:26.800 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:26.823 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.2 on osd.0 2024-09-16T10:23:26.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.2 mark_unfound_lost delete 2024-09-16T10:23:27.061 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T10:23:27.083 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.2 2024-09-16T10:23:27.084 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.1 state active+recovery_unfound+degraded 2024-09-16T10:23:27.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.1 list_unfound '{}' 2024-09-16T10:23:27.136 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-16T10:23:27.133+0000 7f032577b640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 3 pgs recovery_unfound (PG_DAMAGED) 2024-09-16T10:23:27.312 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:27.312 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:27.312 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:27.312 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:27.312 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 2, 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 2, 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existing_3", 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:27.313 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 459768081, 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'3", 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "20'1", 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0", 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "new_3", 2024-09-16T10:23:27.314 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 480221105, 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'2", 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:27.315 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:27.316 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "NotRecovering", 2024-09-16T10:23:27.316 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:27.316 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:27.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:27.316 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:27.316 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:27.316 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:27.316 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:27.316 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:27.316 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:27.331 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.1 on osd.2 2024-09-16T10:23:27.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.1 mark_unfound_lost delete 2024-09-16T10:23:27.516 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 15 20 34997 34977 9.10764 6.60547 0.0128531 0.00857146 2024-09-16T10:23:27.565 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T10:23:27.588 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.3 2024-09-16T10:23:27.588 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.0 state active+recovery_unfound+degraded 2024-09-16T10:23:27.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.0 list_unfound '{}' 2024-09-16T10:23:27.808 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:27.808 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:27.808 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:27.808 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:27.808 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:27.808 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:27.808 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:27.808 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 2, 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 2, 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "new_5", 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 4248945696, 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:27.809 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'3", 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existed_5", 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:27.810 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 4212789968, 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'4", 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "NotRecovering", 2024-09-16T10:23:27.811 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:27.812 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:27.812 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:27.833 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.0 on osd.2 2024-09-16T10:23:27.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.0 mark_unfound_lost delete 2024-09-16T10:23:28.077 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T10:23:28.097 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.7 state active+recovery_wait+degraded 2024-09-16T10:23:28.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.7 list_unfound '{}' 2024-09-16T10:23:28.317 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:28.317 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:28.317 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:28.317 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:28.317 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 1, 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 1, 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "new_7", 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:28.318 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 3761017527, 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'1", 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "Active", 2024-09-16T10:23:28.319 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:28.320 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:28.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:28.320 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:28.320 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:28.320 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:28.320 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:28.320 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:28.320 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:28.339 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.7 on osd.0 2024-09-16T10:23:28.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.7 mark_unfound_lost delete 2024-09-16T10:23:28.516 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 16 20 36646 36626 8.94097 6.44141 0.0210019 0.00873084 2024-09-16T10:23:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-16T10:23:28.614 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.4 2024-09-16T10:23:28.614 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.6 state active+recovery_wait+degraded 2024-09-16T10:23:28.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.6 list_unfound '{}' 2024-09-16T10:23:28.840 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:28.840 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 3, 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 3, 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existed_8", 2024-09-16T10:23:28.841 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 1861424966, 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'5", 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:28.842 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "new_6", 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 3910131350, 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'4", 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:28.843 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "new_4", 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 2787839990, 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'3", 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:28.844 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "Active", 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:28.845 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:28.857 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.6 on osd.0 2024-09-16T10:23:28.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.6 mark_unfound_lost delete 2024-09-16T10:23:29.071 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-16T10:23:29.092 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.5 2024-09-16T10:23:29.092 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.5 state active+recovery_wait+degraded 2024-09-16T10:23:29.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.5 list_unfound '{}' 2024-09-16T10:23:29.311 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:29.311 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:29.311 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:29.311 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:29.311 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:29.311 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:29.311 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 2, 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 2, 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "new_1", 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 26281365, 2024-09-16T10:23:29.312 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'1", 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "new_2", 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:29.313 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 3196340981, 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'2", 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "Active", 2024-09-16T10:23:29.314 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:29.315 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:29.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:29.315 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:29.315 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:29.315 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:29.315 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:29.315 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:29.315 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:29.332 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.5 on osd.0 2024-09-16T10:23:29.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.5 mark_unfound_lost delete 2024-09-16T10:23:29.516 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 17 20 38144 38124 8.75923 5.85156 0.0126542 0.00891283 2024-09-16T10:23:29.572 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-16T10:23:29.596 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.6 2024-09-16T10:23:29.596 INFO:tasks.rep_lost_unfound_delete:no unfound in 1.7 2024-09-16T10:23:29.596 INFO:tasks.rep_lost_unfound_delete:listing missing/lost in 2.4 state active+recovery_wait+degraded 2024-09-16T10:23:29.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.4 list_unfound '{}' 2024-09-16T10:23:29.811 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "offset": { 2024-09-16T10:23:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "", 2024-09-16T10:23:29.811 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": 0, 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 0, 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": -9223372036854775808, 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "num_missing": 4, 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "num_unfound": 3, 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "objects": [ 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existed_4", 2024-09-16T10:23:29.812 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 873891716, 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'8", 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:29.813 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existing_7", 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 3996155460, 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'9", 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "20'6", 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0", 2024-09-16T10:23:29.814 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": { 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "oid": "existed_2", 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "snapid": -2, 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "hash": 1767667508, 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "max": 0, 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "namespace": "" 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "need": "22'7", 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "have": "0'0", 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "none", 2024-09-16T10:23:29.815 INFO:teuthology.orchestra.run.smithi050.stdout: "clean_regions": "clean_offsets: [], clean_omap: 0, new_object: 1", 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "locations": [] 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "state": "Active", 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "available_might_have_unfound": true, 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "might_have_unfound": [ 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": "1", 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "status": "osd is down" 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout: "more": false 2024-09-16T10:23:29.816 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:29.832 INFO:tasks.rep_lost_unfound_delete:reverting unfound in 2.4 on osd.0 2024-09-16T10:23:29.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.4 mark_unfound_lost delete 2024-09-16T10:23:30.071 INFO:teuthology.orchestra.run.smithi050.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-16T10:23:30.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-09-16T10:23:30.314 INFO:teuthology.orchestra.run.smithi050.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-16T10:23:30.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-09-16T10:23:30.517 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 18 20 39739 39719 8.61833 6.23047 0.0129048 0.00905824 2024-09-16T10:23:30.559 INFO:teuthology.orchestra.run.smithi050.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-16T10:23:30.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T10:23:30.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T10:23:30.826 INFO:teuthology.orchestra.run.smithi050.stdout:103079215120 2024-09-16T10:23:30.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:23:30.831 INFO:teuthology.orchestra.run.smithi050.stdout:128849018892 2024-09-16T10:23:30.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:23:31.369 INFO:teuthology.orchestra.run.smithi050.stdout:103079215119 2024-09-16T10:23:31.393 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 103079215120 got 103079215119 for osd.0 2024-09-16T10:23:31.421 INFO:teuthology.orchestra.run.smithi050.stdout:128849018891 2024-09-16T10:23:31.437 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 128849018892 got 128849018891 for osd.2 2024-09-16T10:23:31.517 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 19 20 41136 41116 8.45193 5.45703 0.00886975 0.00923684 2024-09-16T10:23:32.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:23:32.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:23:32.517 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:23:32.516857+0000 min lat: 0.00139595 max lat: 0.21445 avg lat: 0.00933068 2024-09-16T10:23:32.518 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:23:32.518 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 20 20 42863 42843 8.36661 6.74609 0.020265 0.00933068 2024-09-16T10:23:32.939 INFO:teuthology.orchestra.run.smithi050.stdout:103079215119 2024-09-16T10:23:32.960 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 103079215120 got 103079215119 for osd.0 2024-09-16T10:23:32.988 INFO:teuthology.orchestra.run.smithi050.stdout:128849018891 2024-09-16T10:23:33.012 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 128849018892 got 128849018891 for osd.2 2024-09-16T10:23:33.517 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 21 20 44380 44360 8.25036 5.92578 0.00743118 0.00946158 2024-09-16T10:23:33.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T10:23:34.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T10:23:34.508 INFO:teuthology.orchestra.run.smithi050.stdout:103079215120 2024-09-16T10:23:34.517 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 22 20 45853 45833 8.13687 5.75391 0.0121826 0.00959363 2024-09-16T10:23:34.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 103079215120 got 103079215120 for osd.0 2024-09-16T10:23:34.528 DEBUG:teuthology.parallel:result is None 2024-09-16T10:23:34.544 INFO:teuthology.orchestra.run.smithi050.stdout:128849018892 2024-09-16T10:23:34.566 INFO:tasks.rep_lost_unfound_delete.ceph_manager:need seq 128849018892 got 128849018892 for osd.2 2024-09-16T10:23:34.566 DEBUG:teuthology.parallel:result is None 2024-09-16T10:23:34.566 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for recovery to complete 2024-09-16T10:23:34.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:35.142 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:35.144 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:35.166 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-16T10:23:34.565030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":177927399,"num_objects":43466,"num_object_clones":0,"num_object_copies":86932,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43493,"num_write_kb":173792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43521,"ondisk_log_size":43521,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":545620,"kb_used_data":333956,"kb_used_omap":8,"kb_used_meta":211639,"kb_avail":188198060,"statfs":{"total":193273528320,"available":192714813440,"internally_reserved":0,"allocated":341970944,"data_stored":341544529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8872,"internal_metadata":216718680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,9,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":93786112,"num_objects":22897,"num_object_clones":0,"num_object_copies":45794,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":22897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22897,"num_write_kb":91588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002997"},"pg_stats":[{"pgid":"2.f","version":"35'2753","reported_seq":2796,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:34.152446+0000","last_change":"2024-09-16T10:23:01.017258+0000","last_active":"2024-09-16T10:23:34.152446+0000","last_peered":"2024-09-16T10:23:34.152446+0000","last_clean":"2024-09-16T10:23:34.152446+0000","last_became_active":"2024-09-16T10:23:01.016994+0000","last_became_peered":"2024-09-16T10:23:01.016994+0000","last_unstale":"2024-09-16T10:23:34.152446+0000","last_undegraded":"2024-09-16T10:23:34.152446+0000","last_fullsized":"2024-09-16T10:23:34.152446+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2753,"log_dups_size":0,"ondisk_log_size":2753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:53.799911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11276288,"num_objects":2753,"num_object_clones":0,"num_object_copies":5506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2753,"num_write_kb":11012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'2491","reported_seq":2545,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.779968+0000","last_change":"2024-09-16T10:23:23.523752+0000","last_active":"2024-09-16T10:23:30.779968+0000","last_peered":"2024-09-16T10:23:30.779968+0000","last_clean":"2024-09-16T10:23:30.779968+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:30.779968+0000","last_undegraded":"2024-09-16T10:23:30.779968+0000","last_fullsized":"2024-09-16T10:23:30.779968+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2491,"log_dups_size":0,"ondisk_log_size":2491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10179526,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2489,"num_write_kb":9944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'2539","reported_seq":2606,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.782271+0000","last_change":"2024-09-16T10:23:24.034506+0000","last_active":"2024-09-16T10:23:30.782271+0000","last_peered":"2024-09-16T10:23:30.782271+0000","last_clean":"2024-09-16T10:23:30.782271+0000","last_became_active":"2024-09-16T10:23:02.013253+0000","last_became_peered":"2024-09-16T10:23:02.013253+0000","last_unstale":"2024-09-16T10:23:30.782271+0000","last_undegraded":"2024-09-16T10:23:30.782271+0000","last_fullsized":"2024-09-16T10:23:30.782271+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2539,"log_dups_size":0,"ondisk_log_size":2539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10383843,"num_objects":2536,"num_object_clones":0,"num_object_copies":5072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":10142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'2515","reported_seq":2559,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.766203+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:30.766203+0000","last_peered":"2024-09-16T10:23:30.766203+0000","last_clean":"2024-09-16T10:23:30.766203+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:30.766203+0000","last_undegraded":"2024-09-16T10:23:30.766203+0000","last_fullsized":"2024-09-16T10:23:30.766203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2515,"log_dups_size":0,"ondisk_log_size":2515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:59.836622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10301440,"num_objects":2515,"num_object_clones":0,"num_object_copies":5030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2515,"num_write_kb":10060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'2813","reported_seq":2871,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.137696+0000","last_change":"2024-09-16T10:23:01.016337+0000","last_active":"2024-09-16T10:23:34.137696+0000","last_peered":"2024-09-16T10:23:34.137696+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:34.137696+0000","last_undegraded":"2024-09-16T10:23:01.010093+0000","last_fullsized":"2024-09-16T10:23:34.137696+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2813,"log_dups_size":0,"ondisk_log_size":2813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11494342,"num_objects":2808,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2808}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'2803","reported_seq":2850,"reported_epoch":35,"state":"active+recovering","last_fresh":"2024-09-16T10:23:34.112185+0000","last_change":"2024-09-16T10:23:02.010889+0000","last_active":"2024-09-16T10:23:34.112185+0000","last_peered":"2024-09-16T10:23:34.112185+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:23:01.014691+0000","last_became_peered":"2024-09-16T10:23:01.014691+0000","last_unstale":"2024-09-16T10:23:34.112185+0000","last_undegraded":"2024-09-16T10:23:34.112185+0000","last_fullsized":"2024-09-16T10:23:34.112185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11461574,"num_objects":2800,"num_object_clones":0,"num_object_copies":5600,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":11195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2800}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'2949","reported_seq":2994,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.142426+0000","last_change":"2024-09-16T10:23:01.018034+0000","last_active":"2024-09-16T10:23:34.142426+0000","last_peered":"2024-09-16T10:23:34.142426+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:23:01.017728+0000","last_became_peered":"2024-09-16T10:23:01.017728+0000","last_unstale":"2024-09-16T10:23:34.142426+0000","last_undegraded":"2024-09-16T10:23:01.011997+0000","last_fullsized":"2024-09-16T10:23:34.142426+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2949,"log_dups_size":0,"ondisk_log_size":2949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055494,"num_objects":2945,"num_object_clones":0,"num_object_copies":5890,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":11776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'2748","reported_seq":2807,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.133871+0000","last_change":"2024-09-16T10:23:01.016059+0000","last_active":"2024-09-16T10:23:34.133871+0000","last_peered":"2024-09-16T10:23:34.133871+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:34.133871+0000","last_undegraded":"2024-09-16T10:23:01.009538+0000","last_fullsized":"2024-09-16T10:23:34.133871+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11248099,"num_objects":2747,"num_object_clones":0,"num_object_copies":5494,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":10985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2747}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017433+0000","last_change":"2024-09-16T10:23:01.016598+0000","last_active":"2024-09-16T10:23:01.017433+0000","last_peered":"2024-09-16T10:23:01.017433+0000","last_clean":"2024-09-16T10:23:01.017433+0000","last_became_active":"2024-09-16T10:23:01.016110+0000","last_became_peered":"2024-09-16T10:23:01.016110+0000","last_unstale":"2024-09-16T10:23:01.017433+0000","last_undegraded":"2024-09-16T10:23:01.017433+0000","last_fullsized":"2024-09-16T10:23:01.017433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:12.204767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"35'2480","reported_seq":2549,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.760321+0000","last_change":"2024-09-16T10:23:26.615524+0000","last_active":"2024-09-16T10:23:30.760321+0000","last_peered":"2024-09-16T10:23:30.760321+0000","last_clean":"2024-09-16T10:23:30.760321+0000","last_became_active":"2024-09-16T10:23:02.014504+0000","last_became_peered":"2024-09-16T10:23:02.014504+0000","last_unstale":"2024-09-16T10:23:30.760321+0000","last_undegraded":"2024-09-16T10:23:30.760321+0000","last_fullsized":"2024-09-16T10:23:30.760321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2480,"log_dups_size":0,"ondisk_log_size":2480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10130374,"num_objects":2475,"num_object_clones":0,"num_object_copies":4950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2478,"num_write_kb":9896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013960+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:02.013960+0000","last_peered":"2024-09-16T10:23:02.013960+0000","last_clean":"2024-09-16T10:23:02.013960+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:02.013960+0000","last_undegraded":"2024-09-16T10:23:02.013960+0000","last_fullsized":"2024-09-16T10:23:02.013960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'2973","reported_seq":3014,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.116724+0000","last_change":"2024-09-16T10:23:01.015673+0000","last_active":"2024-09-16T10:23:34.116724+0000","last_peered":"2024-09-16T10:23:34.116724+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:23:01.015452+0000","last_became_peered":"2024-09-16T10:23:01.015452+0000","last_unstale":"2024-09-16T10:23:34.116724+0000","last_undegraded":"2024-09-16T10:23:01.010375+0000","last_fullsized":"2024-09-16T10:23:34.116724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12145606,"num_objects":2967,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":11864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2967}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":53,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015931+0000","last_change":"2024-09-16T10:23:01.015931+0000","last_active":"2024-09-16T10:23:01.015931+0000","last_peered":"2024-09-16T10:23:01.015931+0000","last_clean":"2024-09-16T10:22:36.125521+0000","last_became_active":"2024-09-16T10:23:01.015674+0000","last_became_peered":"2024-09-16T10:23:01.015674+0000","last_unstale":"2024-09-16T10:23:01.015931+0000","last_undegraded":"2024-09-16T10:23:01.009899+0000","last_fullsized":"2024-09-16T10:23:01.015931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:04.682617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'2508","reported_seq":2567,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.782115+0000","last_change":"2024-09-16T10:23:27.615947+0000","last_active":"2024-09-16T10:23:30.782115+0000","last_peered":"2024-09-16T10:23:30.782115+0000","last_clean":"2024-09-16T10:23:30.782115+0000","last_became_active":"2024-09-16T10:23:02.012723+0000","last_became_peered":"2024-09-16T10:23:02.012723+0000","last_unstale":"2024-09-16T10:23:30.782115+0000","last_undegraded":"2024-09-16T10:23:30.782115+0000","last_fullsized":"2024-09-16T10:23:30.782115+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10253254,"num_objects":2505,"num_object_clones":0,"num_object_copies":5010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2506,"num_write_kb":10015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.015929+0000","last_change":"2024-09-16T10:23:02.014534+0000","last_active":"2024-09-16T10:23:02.015929+0000","last_peered":"2024-09-16T10:23:02.015929+0000","last_clean":"2024-09-16T10:23:02.015929+0000","last_became_active":"2024-09-16T10:23:02.014138+0000","last_became_peered":"2024-09-16T10:23:02.014138+0000","last_unstale":"2024-09-16T10:23:02.015929+0000","last_undegraded":"2024-09-16T10:23:02.015929+0000","last_fullsized":"2024-09-16T10:23:02.015929+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:19:32.676442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"35'2514","reported_seq":2571,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.782911+0000","last_change":"2024-09-16T10:23:28.126723+0000","last_active":"2024-09-16T10:23:30.782911+0000","last_peered":"2024-09-16T10:23:30.782911+0000","last_clean":"2024-09-16T10:23:30.782911+0000","last_became_active":"2024-09-16T10:23:02.012604+0000","last_became_peered":"2024-09-16T10:23:02.012604+0000","last_unstale":"2024-09-16T10:23:30.782911+0000","last_undegraded":"2024-09-16T10:23:30.782911+0000","last_fullsized":"2024-09-16T10:23:30.782911+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2514,"log_dups_size":0,"ondisk_log_size":2514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10273734,"num_objects":2510,"num_object_clones":0,"num_object_copies":5020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2512,"num_write_kb":10035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'2758","reported_seq":2803,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.152638+0000","last_change":"2024-09-16T10:23:01.018002+0000","last_active":"2024-09-16T10:23:34.152638+0000","last_peered":"2024-09-16T10:23:34.152638+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:23:01.017742+0000","last_became_peered":"2024-09-16T10:23:01.017742+0000","last_unstale":"2024-09-16T10:23:34.152638+0000","last_undegraded":"2024-09-16T10:23:01.012206+0000","last_fullsized":"2024-09-16T10:23:34.152638+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2758,"log_dups_size":0,"ondisk_log_size":2758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11289059,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2757,"num_write_kb":11025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2757}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.016349+0000","last_change":"2024-09-16T10:23:01.014951+0000","last_active":"2024-09-16T10:23:01.016349+0000","last_peered":"2024-09-16T10:23:01.016349+0000","last_clean":"2024-09-16T10:23:01.016349+0000","last_became_active":"2024-09-16T10:23:01.014674+0000","last_became_peered":"2024-09-16T10:23:01.014674+0000","last_unstale":"2024-09-16T10:23:01.016349+0000","last_undegraded":"2024-09-16T10:23:01.016349+0000","last_fullsized":"2024-09-16T10:23:01.016349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:05.742015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"35'2899","reported_seq":2944,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.155997+0000","last_change":"2024-09-16T10:23:01.017334+0000","last_active":"2024-09-16T10:23:34.155997+0000","last_peered":"2024-09-16T10:23:34.155997+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:23:01.016901+0000","last_became_peered":"2024-09-16T10:23:01.016901+0000","last_unstale":"2024-09-16T10:23:34.155997+0000","last_undegraded":"2024-09-16T10:23:01.010804+0000","last_fullsized":"2024-09-16T10:23:34.155997+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2899,"log_dups_size":0,"ondisk_log_size":2899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11842985,"num_objects":2894,"num_object_clones":0,"num_object_copies":5788,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013918+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:02.013918+0000","last_peered":"2024-09-16T10:23:02.013918+0000","last_clean":"2024-09-16T10:23:02.013918+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:02.013918+0000","last_undegraded":"2024-09-16T10:23:02.013918+0000","last_fullsized":"2024-09-16T10:23:02.013918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'2815","reported_seq":2860,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.131261+0000","last_change":"2024-09-16T10:23:01.017225+0000","last_active":"2024-09-16T10:23:34.131261+0000","last_peered":"2024-09-16T10:23:34.131261+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:23:01.016945+0000","last_became_peered":"2024-09-16T10:23:01.016945+0000","last_unstale":"2024-09-16T10:23:34.131261+0000","last_undegraded":"2024-09-16T10:23:01.011253+0000","last_fullsized":"2024-09-16T10:23:34.131261+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11514822,"num_objects":2813,"num_object_clones":0,"num_object_copies":5626,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2813,"num_write_kb":11246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017286+0000","last_change":"2024-09-16T10:23:01.015639+0000","last_active":"2024-09-16T10:23:01.017286+0000","last_peered":"2024-09-16T10:23:01.017286+0000","last_clean":"2024-09-16T10:23:01.017286+0000","last_became_active":"2024-09-16T10:23:01.015380+0000","last_became_peered":"2024-09-16T10:23:01.015380+0000","last_unstale":"2024-09-16T10:23:01.017286+0000","last_undegraded":"2024-09-16T10:23:01.017286+0000","last_fullsized":"2024-09-16T10:23:01.017286+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:03.395240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.016743+0000","last_change":"2024-09-16T10:23:02.015813+0000","last_active":"2024-09-16T10:23:02.016743+0000","last_peered":"2024-09-16T10:23:02.016743+0000","last_clean":"2024-09-16T10:23:02.016743+0000","last_became_active":"2024-09-16T10:23:02.015242+0000","last_became_peered":"2024-09-16T10:23:02.015242+0000","last_unstale":"2024-09-16T10:23:02.016743+0000","last_undegraded":"2024-09-16T10:23:02.016743+0000","last_fullsized":"2024-09-16T10:23:02.016743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:34:28.723524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"35'2960","reported_seq":3006,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.153263+0000","last_change":"2024-09-16T10:23:01.015555+0000","last_active":"2024-09-16T10:23:34.153263+0000","last_peered":"2024-09-16T10:23:34.153263+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:23:01.015304+0000","last_became_peered":"2024-09-16T10:23:01.015304+0000","last_unstale":"2024-09-16T10:23:34.153263+0000","last_undegraded":"2024-09-16T10:23:01.009840+0000","last_fullsized":"2024-09-16T10:23:34.153263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2960,"log_dups_size":0,"ondisk_log_size":2960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12076940,"num_objects":2952,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":11799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":177927380,"num_objects":43464,"num_object_clones":0,"num_object_copies":86928,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43491,"num_write_kb":173790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":350490624,"data_stored":350367782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43518,"ondisk_log_size":43518,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018892,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":268328,"kb_used_data":156320,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94103512,"statfs":{"total":96636764160,"available":96361996288,"internally_reserved":0,"allocated":160071680,"data_stored":159869543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":114683568},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2080000000000002}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215121,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277292,"kb_used_data":177636,"kb_used_omap":4,"kb_used_meta":99643,"kb_avail":94094548,"statfs":{"total":96636764160,"available":96352817152,"internally_reserved":0,"allocated":181899264,"data_stored":181674986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":102035112},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,7,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185978880,"data_stored":185957202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164397056,"data_stored":164397056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:35.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:35.517 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 23 20 47294 47274 8.02782 5.62891 0.00772928 0.00972533 2024-09-16T10:23:35.701 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:35.703 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:35.722 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-16T10:23:34.565030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":177927399,"num_objects":43466,"num_object_clones":0,"num_object_copies":86932,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43493,"num_write_kb":173792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43521,"ondisk_log_size":43521,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":545620,"kb_used_data":333956,"kb_used_omap":8,"kb_used_meta":211639,"kb_avail":188198060,"statfs":{"total":193273528320,"available":192714813440,"internally_reserved":0,"allocated":341970944,"data_stored":341544529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8872,"internal_metadata":216718680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,9,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":93786112,"num_objects":22897,"num_object_clones":0,"num_object_copies":45794,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":22897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22897,"num_write_kb":91588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002997"},"pg_stats":[{"pgid":"2.f","version":"35'2753","reported_seq":2796,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:34.152446+0000","last_change":"2024-09-16T10:23:01.017258+0000","last_active":"2024-09-16T10:23:34.152446+0000","last_peered":"2024-09-16T10:23:34.152446+0000","last_clean":"2024-09-16T10:23:34.152446+0000","last_became_active":"2024-09-16T10:23:01.016994+0000","last_became_peered":"2024-09-16T10:23:01.016994+0000","last_unstale":"2024-09-16T10:23:34.152446+0000","last_undegraded":"2024-09-16T10:23:34.152446+0000","last_fullsized":"2024-09-16T10:23:34.152446+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2753,"log_dups_size":0,"ondisk_log_size":2753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:53.799911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11276288,"num_objects":2753,"num_object_clones":0,"num_object_copies":5506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2753,"num_write_kb":11012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'2491","reported_seq":2545,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.779968+0000","last_change":"2024-09-16T10:23:23.523752+0000","last_active":"2024-09-16T10:23:30.779968+0000","last_peered":"2024-09-16T10:23:30.779968+0000","last_clean":"2024-09-16T10:23:30.779968+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:30.779968+0000","last_undegraded":"2024-09-16T10:23:30.779968+0000","last_fullsized":"2024-09-16T10:23:30.779968+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2491,"log_dups_size":0,"ondisk_log_size":2491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10179526,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2489,"num_write_kb":9944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'2539","reported_seq":2606,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.782271+0000","last_change":"2024-09-16T10:23:24.034506+0000","last_active":"2024-09-16T10:23:30.782271+0000","last_peered":"2024-09-16T10:23:30.782271+0000","last_clean":"2024-09-16T10:23:30.782271+0000","last_became_active":"2024-09-16T10:23:02.013253+0000","last_became_peered":"2024-09-16T10:23:02.013253+0000","last_unstale":"2024-09-16T10:23:30.782271+0000","last_undegraded":"2024-09-16T10:23:30.782271+0000","last_fullsized":"2024-09-16T10:23:30.782271+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2539,"log_dups_size":0,"ondisk_log_size":2539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10383843,"num_objects":2536,"num_object_clones":0,"num_object_copies":5072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":10142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'2515","reported_seq":2559,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.766203+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:30.766203+0000","last_peered":"2024-09-16T10:23:30.766203+0000","last_clean":"2024-09-16T10:23:30.766203+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:30.766203+0000","last_undegraded":"2024-09-16T10:23:30.766203+0000","last_fullsized":"2024-09-16T10:23:30.766203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2515,"log_dups_size":0,"ondisk_log_size":2515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:59.836622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10301440,"num_objects":2515,"num_object_clones":0,"num_object_copies":5030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2515,"num_write_kb":10060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'2813","reported_seq":2871,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.137696+0000","last_change":"2024-09-16T10:23:01.016337+0000","last_active":"2024-09-16T10:23:34.137696+0000","last_peered":"2024-09-16T10:23:34.137696+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:34.137696+0000","last_undegraded":"2024-09-16T10:23:01.010093+0000","last_fullsized":"2024-09-16T10:23:34.137696+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2813,"log_dups_size":0,"ondisk_log_size":2813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11494342,"num_objects":2808,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2808}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'2803","reported_seq":2850,"reported_epoch":35,"state":"active+recovering","last_fresh":"2024-09-16T10:23:34.112185+0000","last_change":"2024-09-16T10:23:02.010889+0000","last_active":"2024-09-16T10:23:34.112185+0000","last_peered":"2024-09-16T10:23:34.112185+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:23:01.014691+0000","last_became_peered":"2024-09-16T10:23:01.014691+0000","last_unstale":"2024-09-16T10:23:34.112185+0000","last_undegraded":"2024-09-16T10:23:34.112185+0000","last_fullsized":"2024-09-16T10:23:34.112185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11461574,"num_objects":2800,"num_object_clones":0,"num_object_copies":5600,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":11195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2800}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'2949","reported_seq":2994,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.142426+0000","last_change":"2024-09-16T10:23:01.018034+0000","last_active":"2024-09-16T10:23:34.142426+0000","last_peered":"2024-09-16T10:23:34.142426+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:23:01.017728+0000","last_became_peered":"2024-09-16T10:23:01.017728+0000","last_unstale":"2024-09-16T10:23:34.142426+0000","last_undegraded":"2024-09-16T10:23:01.011997+0000","last_fullsized":"2024-09-16T10:23:34.142426+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2949,"log_dups_size":0,"ondisk_log_size":2949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055494,"num_objects":2945,"num_object_clones":0,"num_object_copies":5890,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":11776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'2748","reported_seq":2807,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.133871+0000","last_change":"2024-09-16T10:23:01.016059+0000","last_active":"2024-09-16T10:23:34.133871+0000","last_peered":"2024-09-16T10:23:34.133871+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:34.133871+0000","last_undegraded":"2024-09-16T10:23:01.009538+0000","last_fullsized":"2024-09-16T10:23:34.133871+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11248099,"num_objects":2747,"num_object_clones":0,"num_object_copies":5494,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":10985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2747}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017433+0000","last_change":"2024-09-16T10:23:01.016598+0000","last_active":"2024-09-16T10:23:01.017433+0000","last_peered":"2024-09-16T10:23:01.017433+0000","last_clean":"2024-09-16T10:23:01.017433+0000","last_became_active":"2024-09-16T10:23:01.016110+0000","last_became_peered":"2024-09-16T10:23:01.016110+0000","last_unstale":"2024-09-16T10:23:01.017433+0000","last_undegraded":"2024-09-16T10:23:01.017433+0000","last_fullsized":"2024-09-16T10:23:01.017433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:12.204767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"35'2480","reported_seq":2549,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.760321+0000","last_change":"2024-09-16T10:23:26.615524+0000","last_active":"2024-09-16T10:23:30.760321+0000","last_peered":"2024-09-16T10:23:30.760321+0000","last_clean":"2024-09-16T10:23:30.760321+0000","last_became_active":"2024-09-16T10:23:02.014504+0000","last_became_peered":"2024-09-16T10:23:02.014504+0000","last_unstale":"2024-09-16T10:23:30.760321+0000","last_undegraded":"2024-09-16T10:23:30.760321+0000","last_fullsized":"2024-09-16T10:23:30.760321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2480,"log_dups_size":0,"ondisk_log_size":2480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10130374,"num_objects":2475,"num_object_clones":0,"num_object_copies":4950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2478,"num_write_kb":9896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013960+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:02.013960+0000","last_peered":"2024-09-16T10:23:02.013960+0000","last_clean":"2024-09-16T10:23:02.013960+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:02.013960+0000","last_undegraded":"2024-09-16T10:23:02.013960+0000","last_fullsized":"2024-09-16T10:23:02.013960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'2973","reported_seq":3014,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.116724+0000","last_change":"2024-09-16T10:23:01.015673+0000","last_active":"2024-09-16T10:23:34.116724+0000","last_peered":"2024-09-16T10:23:34.116724+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:23:01.015452+0000","last_became_peered":"2024-09-16T10:23:01.015452+0000","last_unstale":"2024-09-16T10:23:34.116724+0000","last_undegraded":"2024-09-16T10:23:01.010375+0000","last_fullsized":"2024-09-16T10:23:34.116724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12145606,"num_objects":2967,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":11864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2967}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":53,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015931+0000","last_change":"2024-09-16T10:23:01.015931+0000","last_active":"2024-09-16T10:23:01.015931+0000","last_peered":"2024-09-16T10:23:01.015931+0000","last_clean":"2024-09-16T10:22:36.125521+0000","last_became_active":"2024-09-16T10:23:01.015674+0000","last_became_peered":"2024-09-16T10:23:01.015674+0000","last_unstale":"2024-09-16T10:23:01.015931+0000","last_undegraded":"2024-09-16T10:23:01.009899+0000","last_fullsized":"2024-09-16T10:23:01.015931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:04.682617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'2508","reported_seq":2567,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.782115+0000","last_change":"2024-09-16T10:23:27.615947+0000","last_active":"2024-09-16T10:23:30.782115+0000","last_peered":"2024-09-16T10:23:30.782115+0000","last_clean":"2024-09-16T10:23:30.782115+0000","last_became_active":"2024-09-16T10:23:02.012723+0000","last_became_peered":"2024-09-16T10:23:02.012723+0000","last_unstale":"2024-09-16T10:23:30.782115+0000","last_undegraded":"2024-09-16T10:23:30.782115+0000","last_fullsized":"2024-09-16T10:23:30.782115+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10253254,"num_objects":2505,"num_object_clones":0,"num_object_copies":5010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2506,"num_write_kb":10015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.015929+0000","last_change":"2024-09-16T10:23:02.014534+0000","last_active":"2024-09-16T10:23:02.015929+0000","last_peered":"2024-09-16T10:23:02.015929+0000","last_clean":"2024-09-16T10:23:02.015929+0000","last_became_active":"2024-09-16T10:23:02.014138+0000","last_became_peered":"2024-09-16T10:23:02.014138+0000","last_unstale":"2024-09-16T10:23:02.015929+0000","last_undegraded":"2024-09-16T10:23:02.015929+0000","last_fullsized":"2024-09-16T10:23:02.015929+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:19:32.676442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"35'2514","reported_seq":2571,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.782911+0000","last_change":"2024-09-16T10:23:28.126723+0000","last_active":"2024-09-16T10:23:30.782911+0000","last_peered":"2024-09-16T10:23:30.782911+0000","last_clean":"2024-09-16T10:23:30.782911+0000","last_became_active":"2024-09-16T10:23:02.012604+0000","last_became_peered":"2024-09-16T10:23:02.012604+0000","last_unstale":"2024-09-16T10:23:30.782911+0000","last_undegraded":"2024-09-16T10:23:30.782911+0000","last_fullsized":"2024-09-16T10:23:30.782911+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2514,"log_dups_size":0,"ondisk_log_size":2514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10273734,"num_objects":2510,"num_object_clones":0,"num_object_copies":5020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2512,"num_write_kb":10035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'2758","reported_seq":2803,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.152638+0000","last_change":"2024-09-16T10:23:01.018002+0000","last_active":"2024-09-16T10:23:34.152638+0000","last_peered":"2024-09-16T10:23:34.152638+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:23:01.017742+0000","last_became_peered":"2024-09-16T10:23:01.017742+0000","last_unstale":"2024-09-16T10:23:34.152638+0000","last_undegraded":"2024-09-16T10:23:01.012206+0000","last_fullsized":"2024-09-16T10:23:34.152638+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2758,"log_dups_size":0,"ondisk_log_size":2758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11289059,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2757,"num_write_kb":11025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2757}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.016349+0000","last_change":"2024-09-16T10:23:01.014951+0000","last_active":"2024-09-16T10:23:01.016349+0000","last_peered":"2024-09-16T10:23:01.016349+0000","last_clean":"2024-09-16T10:23:01.016349+0000","last_became_active":"2024-09-16T10:23:01.014674+0000","last_became_peered":"2024-09-16T10:23:01.014674+0000","last_unstale":"2024-09-16T10:23:01.016349+0000","last_undegraded":"2024-09-16T10:23:01.016349+0000","last_fullsized":"2024-09-16T10:23:01.016349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:05.742015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"35'2899","reported_seq":2944,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.155997+0000","last_change":"2024-09-16T10:23:01.017334+0000","last_active":"2024-09-16T10:23:34.155997+0000","last_peered":"2024-09-16T10:23:34.155997+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:23:01.016901+0000","last_became_peered":"2024-09-16T10:23:01.016901+0000","last_unstale":"2024-09-16T10:23:34.155997+0000","last_undegraded":"2024-09-16T10:23:01.010804+0000","last_fullsized":"2024-09-16T10:23:34.155997+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2899,"log_dups_size":0,"ondisk_log_size":2899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11842985,"num_objects":2894,"num_object_clones":0,"num_object_copies":5788,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013918+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:02.013918+0000","last_peered":"2024-09-16T10:23:02.013918+0000","last_clean":"2024-09-16T10:23:02.013918+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:02.013918+0000","last_undegraded":"2024-09-16T10:23:02.013918+0000","last_fullsized":"2024-09-16T10:23:02.013918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'2815","reported_seq":2860,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.131261+0000","last_change":"2024-09-16T10:23:01.017225+0000","last_active":"2024-09-16T10:23:34.131261+0000","last_peered":"2024-09-16T10:23:34.131261+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:23:01.016945+0000","last_became_peered":"2024-09-16T10:23:01.016945+0000","last_unstale":"2024-09-16T10:23:34.131261+0000","last_undegraded":"2024-09-16T10:23:01.011253+0000","last_fullsized":"2024-09-16T10:23:34.131261+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11514822,"num_objects":2813,"num_object_clones":0,"num_object_copies":5626,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2813,"num_write_kb":11246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017286+0000","last_change":"2024-09-16T10:23:01.015639+0000","last_active":"2024-09-16T10:23:01.017286+0000","last_peered":"2024-09-16T10:23:01.017286+0000","last_clean":"2024-09-16T10:23:01.017286+0000","last_became_active":"2024-09-16T10:23:01.015380+0000","last_became_peered":"2024-09-16T10:23:01.015380+0000","last_unstale":"2024-09-16T10:23:01.017286+0000","last_undegraded":"2024-09-16T10:23:01.017286+0000","last_fullsized":"2024-09-16T10:23:01.017286+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:03.395240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.016743+0000","last_change":"2024-09-16T10:23:02.015813+0000","last_active":"2024-09-16T10:23:02.016743+0000","last_peered":"2024-09-16T10:23:02.016743+0000","last_clean":"2024-09-16T10:23:02.016743+0000","last_became_active":"2024-09-16T10:23:02.015242+0000","last_became_peered":"2024-09-16T10:23:02.015242+0000","last_unstale":"2024-09-16T10:23:02.016743+0000","last_undegraded":"2024-09-16T10:23:02.016743+0000","last_fullsized":"2024-09-16T10:23:02.016743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:34:28.723524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"35'2960","reported_seq":3006,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.153263+0000","last_change":"2024-09-16T10:23:01.015555+0000","last_active":"2024-09-16T10:23:34.153263+0000","last_peered":"2024-09-16T10:23:34.153263+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:23:01.015304+0000","last_became_peered":"2024-09-16T10:23:01.015304+0000","last_unstale":"2024-09-16T10:23:34.153263+0000","last_undegraded":"2024-09-16T10:23:01.009840+0000","last_fullsized":"2024-09-16T10:23:34.153263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2960,"log_dups_size":0,"ondisk_log_size":2960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12076940,"num_objects":2952,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":11799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":177927380,"num_objects":43464,"num_object_clones":0,"num_object_copies":86928,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43491,"num_write_kb":173790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":350490624,"data_stored":350367782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43518,"ondisk_log_size":43518,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018892,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":268328,"kb_used_data":156320,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94103512,"statfs":{"total":96636764160,"available":96361996288,"internally_reserved":0,"allocated":160071680,"data_stored":159869543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":114683568},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2080000000000002}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215121,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277292,"kb_used_data":177636,"kb_used_omap":4,"kb_used_meta":99643,"kb_avail":94094548,"statfs":{"total":96636764160,"available":96352817152,"internally_reserved":0,"allocated":181899264,"data_stored":181674986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":102035112},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,7,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185978880,"data_stored":185957202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164397056,"data_stored":164397056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:35.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:36.259 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:36.261 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:36.282 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-16T10:23:34.565030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":177927399,"num_objects":43466,"num_object_clones":0,"num_object_copies":86932,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43493,"num_write_kb":173792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43521,"ondisk_log_size":43521,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":545620,"kb_used_data":333956,"kb_used_omap":8,"kb_used_meta":211639,"kb_avail":188198060,"statfs":{"total":193273528320,"available":192714813440,"internally_reserved":0,"allocated":341970944,"data_stored":341544529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8872,"internal_metadata":216718680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,2,9,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":93786112,"num_objects":22897,"num_object_clones":0,"num_object_copies":45794,"num_objects_missing_on_primary":-9,"num_objects_missing":-9,"num_objects_degraded":-18,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":22897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22897,"num_write_kb":91588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002997"},"pg_stats":[{"pgid":"2.f","version":"35'2753","reported_seq":2796,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:34.152446+0000","last_change":"2024-09-16T10:23:01.017258+0000","last_active":"2024-09-16T10:23:34.152446+0000","last_peered":"2024-09-16T10:23:34.152446+0000","last_clean":"2024-09-16T10:23:34.152446+0000","last_became_active":"2024-09-16T10:23:01.016994+0000","last_became_peered":"2024-09-16T10:23:01.016994+0000","last_unstale":"2024-09-16T10:23:34.152446+0000","last_undegraded":"2024-09-16T10:23:34.152446+0000","last_fullsized":"2024-09-16T10:23:34.152446+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2753,"log_dups_size":0,"ondisk_log_size":2753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:53.799911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11276288,"num_objects":2753,"num_object_clones":0,"num_object_copies":5506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2753,"num_write_kb":11012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'2491","reported_seq":2545,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.779968+0000","last_change":"2024-09-16T10:23:23.523752+0000","last_active":"2024-09-16T10:23:30.779968+0000","last_peered":"2024-09-16T10:23:30.779968+0000","last_clean":"2024-09-16T10:23:30.779968+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:30.779968+0000","last_undegraded":"2024-09-16T10:23:30.779968+0000","last_fullsized":"2024-09-16T10:23:30.779968+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2491,"log_dups_size":0,"ondisk_log_size":2491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10179526,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2489,"num_write_kb":9944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'2539","reported_seq":2606,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.782271+0000","last_change":"2024-09-16T10:23:24.034506+0000","last_active":"2024-09-16T10:23:30.782271+0000","last_peered":"2024-09-16T10:23:30.782271+0000","last_clean":"2024-09-16T10:23:30.782271+0000","last_became_active":"2024-09-16T10:23:02.013253+0000","last_became_peered":"2024-09-16T10:23:02.013253+0000","last_unstale":"2024-09-16T10:23:30.782271+0000","last_undegraded":"2024-09-16T10:23:30.782271+0000","last_fullsized":"2024-09-16T10:23:30.782271+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2539,"log_dups_size":0,"ondisk_log_size":2539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10383843,"num_objects":2536,"num_object_clones":0,"num_object_copies":5072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2538,"num_write_kb":10142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'2515","reported_seq":2559,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.766203+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:30.766203+0000","last_peered":"2024-09-16T10:23:30.766203+0000","last_clean":"2024-09-16T10:23:30.766203+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:30.766203+0000","last_undegraded":"2024-09-16T10:23:30.766203+0000","last_fullsized":"2024-09-16T10:23:30.766203+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2515,"log_dups_size":0,"ondisk_log_size":2515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:59.836622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10301440,"num_objects":2515,"num_object_clones":0,"num_object_copies":5030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2515,"num_write_kb":10060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'2813","reported_seq":2871,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.137696+0000","last_change":"2024-09-16T10:23:01.016337+0000","last_active":"2024-09-16T10:23:34.137696+0000","last_peered":"2024-09-16T10:23:34.137696+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:34.137696+0000","last_undegraded":"2024-09-16T10:23:01.010093+0000","last_fullsized":"2024-09-16T10:23:34.137696+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2813,"log_dups_size":0,"ondisk_log_size":2813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11494342,"num_objects":2808,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2808}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'2803","reported_seq":2850,"reported_epoch":35,"state":"active+recovering","last_fresh":"2024-09-16T10:23:34.112185+0000","last_change":"2024-09-16T10:23:02.010889+0000","last_active":"2024-09-16T10:23:34.112185+0000","last_peered":"2024-09-16T10:23:34.112185+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:23:01.014691+0000","last_became_peered":"2024-09-16T10:23:01.014691+0000","last_unstale":"2024-09-16T10:23:34.112185+0000","last_undegraded":"2024-09-16T10:23:34.112185+0000","last_fullsized":"2024-09-16T10:23:34.112185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11461574,"num_objects":2800,"num_object_clones":0,"num_object_copies":5600,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":11195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2800}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'2949","reported_seq":2994,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.142426+0000","last_change":"2024-09-16T10:23:01.018034+0000","last_active":"2024-09-16T10:23:34.142426+0000","last_peered":"2024-09-16T10:23:34.142426+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:23:01.017728+0000","last_became_peered":"2024-09-16T10:23:01.017728+0000","last_unstale":"2024-09-16T10:23:34.142426+0000","last_undegraded":"2024-09-16T10:23:01.011997+0000","last_fullsized":"2024-09-16T10:23:34.142426+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2949,"log_dups_size":0,"ondisk_log_size":2949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055494,"num_objects":2945,"num_object_clones":0,"num_object_copies":5890,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":11776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'2748","reported_seq":2807,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.133871+0000","last_change":"2024-09-16T10:23:01.016059+0000","last_active":"2024-09-16T10:23:34.133871+0000","last_peered":"2024-09-16T10:23:34.133871+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:34.133871+0000","last_undegraded":"2024-09-16T10:23:01.009538+0000","last_fullsized":"2024-09-16T10:23:34.133871+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11248099,"num_objects":2747,"num_object_clones":0,"num_object_copies":5494,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":10985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2747}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017433+0000","last_change":"2024-09-16T10:23:01.016598+0000","last_active":"2024-09-16T10:23:01.017433+0000","last_peered":"2024-09-16T10:23:01.017433+0000","last_clean":"2024-09-16T10:23:01.017433+0000","last_became_active":"2024-09-16T10:23:01.016110+0000","last_became_peered":"2024-09-16T10:23:01.016110+0000","last_unstale":"2024-09-16T10:23:01.017433+0000","last_undegraded":"2024-09-16T10:23:01.017433+0000","last_fullsized":"2024-09-16T10:23:01.017433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:12.204767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"35'2480","reported_seq":2549,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.760321+0000","last_change":"2024-09-16T10:23:26.615524+0000","last_active":"2024-09-16T10:23:30.760321+0000","last_peered":"2024-09-16T10:23:30.760321+0000","last_clean":"2024-09-16T10:23:30.760321+0000","last_became_active":"2024-09-16T10:23:02.014504+0000","last_became_peered":"2024-09-16T10:23:02.014504+0000","last_unstale":"2024-09-16T10:23:30.760321+0000","last_undegraded":"2024-09-16T10:23:30.760321+0000","last_fullsized":"2024-09-16T10:23:30.760321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2480,"log_dups_size":0,"ondisk_log_size":2480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10130374,"num_objects":2475,"num_object_clones":0,"num_object_copies":4950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2478,"num_write_kb":9896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013960+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:02.013960+0000","last_peered":"2024-09-16T10:23:02.013960+0000","last_clean":"2024-09-16T10:23:02.013960+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:02.013960+0000","last_undegraded":"2024-09-16T10:23:02.013960+0000","last_fullsized":"2024-09-16T10:23:02.013960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'2973","reported_seq":3014,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.116724+0000","last_change":"2024-09-16T10:23:01.015673+0000","last_active":"2024-09-16T10:23:34.116724+0000","last_peered":"2024-09-16T10:23:34.116724+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:23:01.015452+0000","last_became_peered":"2024-09-16T10:23:01.015452+0000","last_unstale":"2024-09-16T10:23:34.116724+0000","last_undegraded":"2024-09-16T10:23:01.010375+0000","last_fullsized":"2024-09-16T10:23:34.116724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12145606,"num_objects":2967,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":11864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2967}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":53,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015931+0000","last_change":"2024-09-16T10:23:01.015931+0000","last_active":"2024-09-16T10:23:01.015931+0000","last_peered":"2024-09-16T10:23:01.015931+0000","last_clean":"2024-09-16T10:22:36.125521+0000","last_became_active":"2024-09-16T10:23:01.015674+0000","last_became_peered":"2024-09-16T10:23:01.015674+0000","last_unstale":"2024-09-16T10:23:01.015931+0000","last_undegraded":"2024-09-16T10:23:01.009899+0000","last_fullsized":"2024-09-16T10:23:01.015931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:04.682617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'2508","reported_seq":2567,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.782115+0000","last_change":"2024-09-16T10:23:27.615947+0000","last_active":"2024-09-16T10:23:30.782115+0000","last_peered":"2024-09-16T10:23:30.782115+0000","last_clean":"2024-09-16T10:23:30.782115+0000","last_became_active":"2024-09-16T10:23:02.012723+0000","last_became_peered":"2024-09-16T10:23:02.012723+0000","last_unstale":"2024-09-16T10:23:30.782115+0000","last_undegraded":"2024-09-16T10:23:30.782115+0000","last_fullsized":"2024-09-16T10:23:30.782115+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10253254,"num_objects":2505,"num_object_clones":0,"num_object_copies":5010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2506,"num_write_kb":10015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.015929+0000","last_change":"2024-09-16T10:23:02.014534+0000","last_active":"2024-09-16T10:23:02.015929+0000","last_peered":"2024-09-16T10:23:02.015929+0000","last_clean":"2024-09-16T10:23:02.015929+0000","last_became_active":"2024-09-16T10:23:02.014138+0000","last_became_peered":"2024-09-16T10:23:02.014138+0000","last_unstale":"2024-09-16T10:23:02.015929+0000","last_undegraded":"2024-09-16T10:23:02.015929+0000","last_fullsized":"2024-09-16T10:23:02.015929+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:19:32.676442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"35'2514","reported_seq":2571,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:30.782911+0000","last_change":"2024-09-16T10:23:28.126723+0000","last_active":"2024-09-16T10:23:30.782911+0000","last_peered":"2024-09-16T10:23:30.782911+0000","last_clean":"2024-09-16T10:23:30.782911+0000","last_became_active":"2024-09-16T10:23:02.012604+0000","last_became_peered":"2024-09-16T10:23:02.012604+0000","last_unstale":"2024-09-16T10:23:30.782911+0000","last_undegraded":"2024-09-16T10:23:30.782911+0000","last_fullsized":"2024-09-16T10:23:30.782911+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2514,"log_dups_size":0,"ondisk_log_size":2514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10273734,"num_objects":2510,"num_object_clones":0,"num_object_copies":5020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2512,"num_write_kb":10035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'2758","reported_seq":2803,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.152638+0000","last_change":"2024-09-16T10:23:01.018002+0000","last_active":"2024-09-16T10:23:34.152638+0000","last_peered":"2024-09-16T10:23:34.152638+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:23:01.017742+0000","last_became_peered":"2024-09-16T10:23:01.017742+0000","last_unstale":"2024-09-16T10:23:34.152638+0000","last_undegraded":"2024-09-16T10:23:01.012206+0000","last_fullsized":"2024-09-16T10:23:34.152638+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2758,"log_dups_size":0,"ondisk_log_size":2758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11289059,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2757,"num_write_kb":11025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2757}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.016349+0000","last_change":"2024-09-16T10:23:01.014951+0000","last_active":"2024-09-16T10:23:01.016349+0000","last_peered":"2024-09-16T10:23:01.016349+0000","last_clean":"2024-09-16T10:23:01.016349+0000","last_became_active":"2024-09-16T10:23:01.014674+0000","last_became_peered":"2024-09-16T10:23:01.014674+0000","last_unstale":"2024-09-16T10:23:01.016349+0000","last_undegraded":"2024-09-16T10:23:01.016349+0000","last_fullsized":"2024-09-16T10:23:01.016349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:05.742015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"35'2899","reported_seq":2944,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.155997+0000","last_change":"2024-09-16T10:23:01.017334+0000","last_active":"2024-09-16T10:23:34.155997+0000","last_peered":"2024-09-16T10:23:34.155997+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:23:01.016901+0000","last_became_peered":"2024-09-16T10:23:01.016901+0000","last_unstale":"2024-09-16T10:23:34.155997+0000","last_undegraded":"2024-09-16T10:23:01.010804+0000","last_fullsized":"2024-09-16T10:23:34.155997+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2899,"log_dups_size":0,"ondisk_log_size":2899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11842985,"num_objects":2894,"num_object_clones":0,"num_object_copies":5788,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013918+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:02.013918+0000","last_peered":"2024-09-16T10:23:02.013918+0000","last_clean":"2024-09-16T10:23:02.013918+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:02.013918+0000","last_undegraded":"2024-09-16T10:23:02.013918+0000","last_fullsized":"2024-09-16T10:23:02.013918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'2815","reported_seq":2860,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.131261+0000","last_change":"2024-09-16T10:23:01.017225+0000","last_active":"2024-09-16T10:23:34.131261+0000","last_peered":"2024-09-16T10:23:34.131261+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:23:01.016945+0000","last_became_peered":"2024-09-16T10:23:01.016945+0000","last_unstale":"2024-09-16T10:23:34.131261+0000","last_undegraded":"2024-09-16T10:23:01.011253+0000","last_fullsized":"2024-09-16T10:23:34.131261+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11514822,"num_objects":2813,"num_object_clones":0,"num_object_copies":5626,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2813,"num_write_kb":11246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017286+0000","last_change":"2024-09-16T10:23:01.015639+0000","last_active":"2024-09-16T10:23:01.017286+0000","last_peered":"2024-09-16T10:23:01.017286+0000","last_clean":"2024-09-16T10:23:01.017286+0000","last_became_active":"2024-09-16T10:23:01.015380+0000","last_became_peered":"2024-09-16T10:23:01.015380+0000","last_unstale":"2024-09-16T10:23:01.017286+0000","last_undegraded":"2024-09-16T10:23:01.017286+0000","last_fullsized":"2024-09-16T10:23:01.017286+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:03.395240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.016743+0000","last_change":"2024-09-16T10:23:02.015813+0000","last_active":"2024-09-16T10:23:02.016743+0000","last_peered":"2024-09-16T10:23:02.016743+0000","last_clean":"2024-09-16T10:23:02.016743+0000","last_became_active":"2024-09-16T10:23:02.015242+0000","last_became_peered":"2024-09-16T10:23:02.015242+0000","last_unstale":"2024-09-16T10:23:02.016743+0000","last_undegraded":"2024-09-16T10:23:02.016743+0000","last_fullsized":"2024-09-16T10:23:02.016743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:34:28.723524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"35'2960","reported_seq":3006,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.153263+0000","last_change":"2024-09-16T10:23:01.015555+0000","last_active":"2024-09-16T10:23:34.153263+0000","last_peered":"2024-09-16T10:23:34.153263+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:23:01.015304+0000","last_became_peered":"2024-09-16T10:23:01.015304+0000","last_unstale":"2024-09-16T10:23:34.153263+0000","last_undegraded":"2024-09-16T10:23:01.009840+0000","last_fullsized":"2024-09-16T10:23:34.153263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2960,"log_dups_size":0,"ondisk_log_size":2960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12076940,"num_objects":2952,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":11799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":177927380,"num_objects":43464,"num_object_clones":0,"num_object_copies":86928,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43491,"num_write_kb":173790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":350490624,"data_stored":350367782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43518,"ondisk_log_size":43518,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018892,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":268328,"kb_used_data":156320,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94103512,"statfs":{"total":96636764160,"available":96361996288,"internally_reserved":0,"allocated":160071680,"data_stored":159869543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":114683568},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2080000000000002}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215121,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277292,"kb_used_data":177636,"kb_used_omap":4,"kb_used_meta":99643,"kb_avail":94094548,"statfs":{"total":96636764160,"available":96352817152,"internally_reserved":0,"allocated":181899264,"data_stored":181674986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":102035112},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,7,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185978880,"data_stored":185957202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":164397056,"data_stored":164397056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:36.518 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 24 20 48763 48743 7.9324 5.73828 0.0131737 0.0098429 2024-09-16T10:23:37.518 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 25 20 50382 50362 7.86806 6.32422 0.0197469 0.00992439 2024-09-16T10:23:38.518 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 26 20 51813 51793 7.78042 5.58984 0.0130848 0.0100354 2024-09-16T10:23:39.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:39.518 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 27 20 53290 53270 7.70593 5.76953 0.0202805 0.0101326 2024-09-16T10:23:39.817 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:39.820 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:39.840 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-16T10:23:38.565886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188286183,"num_objects":45995,"num_object_clones":0,"num_object_copies":91990,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46022,"num_write_kb":183908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46050,"ondisk_log_size":46050,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":572036,"kb_used_data":364468,"kb_used_omap":8,"kb_used_meta":207543,"kb_avail":188171644,"statfs":{"total":193273528320,"available":192687763456,"internally_reserved":0,"allocated":373215232,"data_stored":372788817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8872,"internal_metadata":212524376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,4,11,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":66220032,"num_objects":16167,"num_object_clones":0,"num_object_copies":32334,"num_objects_missing_on_primary":-6,"num_objects_missing":-6,"num_objects_degraded":-12,"num_objects_misplaced":0,"num_objects_unfound":-24,"num_objects_dirty":16167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16167,"num_write_kb":64668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002803"},"pg_stats":[{"pgid":"2.f","version":"35'2753","reported_seq":2796,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:34.152446+0000","last_change":"2024-09-16T10:23:01.017258+0000","last_active":"2024-09-16T10:23:34.152446+0000","last_peered":"2024-09-16T10:23:34.152446+0000","last_clean":"2024-09-16T10:23:34.152446+0000","last_became_active":"2024-09-16T10:23:01.016994+0000","last_became_peered":"2024-09-16T10:23:01.016994+0000","last_unstale":"2024-09-16T10:23:34.152446+0000","last_undegraded":"2024-09-16T10:23:34.152446+0000","last_fullsized":"2024-09-16T10:23:34.152446+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2753,"log_dups_size":0,"ondisk_log_size":2753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:53.799911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11276288,"num_objects":2753,"num_object_clones":0,"num_object_copies":5506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2753,"num_write_kb":11012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'2904","reported_seq":2958,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.244832+0000","last_change":"2024-09-16T10:23:23.523752+0000","last_active":"2024-09-16T10:23:35.244832+0000","last_peered":"2024-09-16T10:23:35.244832+0000","last_clean":"2024-09-16T10:23:35.244832+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:35.244832+0000","last_undegraded":"2024-09-16T10:23:35.244832+0000","last_fullsized":"2024-09-16T10:23:35.244832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11871174,"num_objects":2900,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'2920","reported_seq":2986,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.242699+0000","last_change":"2024-09-16T10:23:24.034506+0000","last_active":"2024-09-16T10:23:35.242699+0000","last_peered":"2024-09-16T10:23:35.242699+0000","last_clean":"2024-09-16T10:23:35.242699+0000","last_became_active":"2024-09-16T10:23:02.013253+0000","last_became_peered":"2024-09-16T10:23:02.013253+0000","last_unstale":"2024-09-16T10:23:35.242699+0000","last_undegraded":"2024-09-16T10:23:35.242699+0000","last_fullsized":"2024-09-16T10:23:35.242699+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2920,"log_dups_size":0,"ondisk_log_size":2920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11944419,"num_objects":2917,"num_object_clones":0,"num_object_copies":5834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2919,"num_write_kb":11666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'2953","reported_seq":2998,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.254852+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:35.254852+0000","last_peered":"2024-09-16T10:23:35.254852+0000","last_clean":"2024-09-16T10:23:35.254852+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:35.254852+0000","last_undegraded":"2024-09-16T10:23:35.254852+0000","last_fullsized":"2024-09-16T10:23:35.254852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2953,"log_dups_size":0,"ondisk_log_size":2953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:59.836622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12095488,"num_objects":2953,"num_object_clones":0,"num_object_copies":5906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2953,"num_write_kb":11812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'2813","reported_seq":2871,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.137696+0000","last_change":"2024-09-16T10:23:01.016337+0000","last_active":"2024-09-16T10:23:34.137696+0000","last_peered":"2024-09-16T10:23:34.137696+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:34.137696+0000","last_undegraded":"2024-09-16T10:23:01.010093+0000","last_fullsized":"2024-09-16T10:23:34.137696+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2813,"log_dups_size":0,"ondisk_log_size":2813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11494342,"num_objects":2808,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2808}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'2803","reported_seq":2850,"reported_epoch":35,"state":"active+recovering","last_fresh":"2024-09-16T10:23:34.112185+0000","last_change":"2024-09-16T10:23:02.010889+0000","last_active":"2024-09-16T10:23:34.112185+0000","last_peered":"2024-09-16T10:23:34.112185+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:23:01.014691+0000","last_became_peered":"2024-09-16T10:23:01.014691+0000","last_unstale":"2024-09-16T10:23:34.112185+0000","last_undegraded":"2024-09-16T10:23:34.112185+0000","last_fullsized":"2024-09-16T10:23:34.112185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11461574,"num_objects":2800,"num_object_clones":0,"num_object_copies":5600,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":11195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2800}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'2949","reported_seq":2994,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.142426+0000","last_change":"2024-09-16T10:23:01.018034+0000","last_active":"2024-09-16T10:23:34.142426+0000","last_peered":"2024-09-16T10:23:34.142426+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:23:01.017728+0000","last_became_peered":"2024-09-16T10:23:01.017728+0000","last_unstale":"2024-09-16T10:23:34.142426+0000","last_undegraded":"2024-09-16T10:23:01.011997+0000","last_fullsized":"2024-09-16T10:23:34.142426+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2949,"log_dups_size":0,"ondisk_log_size":2949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055494,"num_objects":2945,"num_object_clones":0,"num_object_copies":5890,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":11776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'2748","reported_seq":2807,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.133871+0000","last_change":"2024-09-16T10:23:01.016059+0000","last_active":"2024-09-16T10:23:34.133871+0000","last_peered":"2024-09-16T10:23:34.133871+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:34.133871+0000","last_undegraded":"2024-09-16T10:23:01.009538+0000","last_fullsized":"2024-09-16T10:23:34.133871+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11248099,"num_objects":2747,"num_object_clones":0,"num_object_copies":5494,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":10985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2747}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017433+0000","last_change":"2024-09-16T10:23:01.016598+0000","last_active":"2024-09-16T10:23:01.017433+0000","last_peered":"2024-09-16T10:23:01.017433+0000","last_clean":"2024-09-16T10:23:01.017433+0000","last_became_active":"2024-09-16T10:23:01.016110+0000","last_became_peered":"2024-09-16T10:23:01.016110+0000","last_unstale":"2024-09-16T10:23:01.017433+0000","last_undegraded":"2024-09-16T10:23:01.017433+0000","last_fullsized":"2024-09-16T10:23:01.017433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:12.204767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"35'2924","reported_seq":2991,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.258927+0000","last_change":"2024-09-16T10:23:26.615524+0000","last_active":"2024-09-16T10:23:35.258927+0000","last_peered":"2024-09-16T10:23:35.258927+0000","last_clean":"2024-09-16T10:23:35.258927+0000","last_became_active":"2024-09-16T10:23:02.014504+0000","last_became_peered":"2024-09-16T10:23:02.014504+0000","last_unstale":"2024-09-16T10:23:35.258927+0000","last_undegraded":"2024-09-16T10:23:35.258927+0000","last_fullsized":"2024-09-16T10:23:35.258927+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2924,"log_dups_size":0,"ondisk_log_size":2924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11948998,"num_objects":2919,"num_object_clones":0,"num_object_copies":5838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":11672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013960+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:02.013960+0000","last_peered":"2024-09-16T10:23:02.013960+0000","last_clean":"2024-09-16T10:23:02.013960+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:02.013960+0000","last_undegraded":"2024-09-16T10:23:02.013960+0000","last_fullsized":"2024-09-16T10:23:02.013960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'2973","reported_seq":3014,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.116724+0000","last_change":"2024-09-16T10:23:01.015673+0000","last_active":"2024-09-16T10:23:34.116724+0000","last_peered":"2024-09-16T10:23:34.116724+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:23:01.015452+0000","last_became_peered":"2024-09-16T10:23:01.015452+0000","last_unstale":"2024-09-16T10:23:34.116724+0000","last_undegraded":"2024-09-16T10:23:01.010375+0000","last_fullsized":"2024-09-16T10:23:34.116724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12145606,"num_objects":2967,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":11864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2967}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":53,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015931+0000","last_change":"2024-09-16T10:23:01.015931+0000","last_active":"2024-09-16T10:23:01.015931+0000","last_peered":"2024-09-16T10:23:01.015931+0000","last_clean":"2024-09-16T10:22:36.125521+0000","last_became_active":"2024-09-16T10:23:01.015674+0000","last_became_peered":"2024-09-16T10:23:01.015674+0000","last_unstale":"2024-09-16T10:23:01.015931+0000","last_undegraded":"2024-09-16T10:23:01.009899+0000","last_fullsized":"2024-09-16T10:23:01.015931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:04.682617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'2950","reported_seq":3009,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.195934+0000","last_change":"2024-09-16T10:23:27.615947+0000","last_active":"2024-09-16T10:23:35.195934+0000","last_peered":"2024-09-16T10:23:35.195934+0000","last_clean":"2024-09-16T10:23:35.195934+0000","last_became_active":"2024-09-16T10:23:02.012723+0000","last_became_peered":"2024-09-16T10:23:02.012723+0000","last_unstale":"2024-09-16T10:23:35.195934+0000","last_undegraded":"2024-09-16T10:23:35.195934+0000","last_fullsized":"2024-09-16T10:23:35.195934+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12063686,"num_objects":2947,"num_object_clones":0,"num_object_copies":5894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2948,"num_write_kb":11783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.015929+0000","last_change":"2024-09-16T10:23:02.014534+0000","last_active":"2024-09-16T10:23:02.015929+0000","last_peered":"2024-09-16T10:23:02.015929+0000","last_clean":"2024-09-16T10:23:02.015929+0000","last_became_active":"2024-09-16T10:23:02.014138+0000","last_became_peered":"2024-09-16T10:23:02.014138+0000","last_unstale":"2024-09-16T10:23:02.015929+0000","last_undegraded":"2024-09-16T10:23:02.015929+0000","last_fullsized":"2024-09-16T10:23:02.015929+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:19:32.676442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"35'2925","reported_seq":2984,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.254737+0000","last_change":"2024-09-16T10:23:28.126723+0000","last_active":"2024-09-16T10:23:35.254737+0000","last_peered":"2024-09-16T10:23:35.254737+0000","last_clean":"2024-09-16T10:23:35.254737+0000","last_became_active":"2024-09-16T10:23:02.012604+0000","last_became_peered":"2024-09-16T10:23:02.012604+0000","last_unstale":"2024-09-16T10:23:35.254737+0000","last_undegraded":"2024-09-16T10:23:35.254737+0000","last_fullsized":"2024-09-16T10:23:35.254737+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2925,"log_dups_size":0,"ondisk_log_size":2925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11957190,"num_objects":2921,"num_object_clones":0,"num_object_copies":5842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":11679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'2758","reported_seq":2803,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.152638+0000","last_change":"2024-09-16T10:23:01.018002+0000","last_active":"2024-09-16T10:23:34.152638+0000","last_peered":"2024-09-16T10:23:34.152638+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:23:01.017742+0000","last_became_peered":"2024-09-16T10:23:01.017742+0000","last_unstale":"2024-09-16T10:23:34.152638+0000","last_undegraded":"2024-09-16T10:23:01.012206+0000","last_fullsized":"2024-09-16T10:23:34.152638+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2758,"log_dups_size":0,"ondisk_log_size":2758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11289059,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2757,"num_write_kb":11025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2757}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.016349+0000","last_change":"2024-09-16T10:23:01.014951+0000","last_active":"2024-09-16T10:23:01.016349+0000","last_peered":"2024-09-16T10:23:01.016349+0000","last_clean":"2024-09-16T10:23:01.016349+0000","last_became_active":"2024-09-16T10:23:01.014674+0000","last_became_peered":"2024-09-16T10:23:01.014674+0000","last_unstale":"2024-09-16T10:23:01.016349+0000","last_undegraded":"2024-09-16T10:23:01.016349+0000","last_fullsized":"2024-09-16T10:23:01.016349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:05.742015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"35'2899","reported_seq":2944,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.155997+0000","last_change":"2024-09-16T10:23:01.017334+0000","last_active":"2024-09-16T10:23:34.155997+0000","last_peered":"2024-09-16T10:23:34.155997+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:23:01.016901+0000","last_became_peered":"2024-09-16T10:23:01.016901+0000","last_unstale":"2024-09-16T10:23:34.155997+0000","last_undegraded":"2024-09-16T10:23:01.010804+0000","last_fullsized":"2024-09-16T10:23:34.155997+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2899,"log_dups_size":0,"ondisk_log_size":2899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11842985,"num_objects":2894,"num_object_clones":0,"num_object_copies":5788,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013918+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:02.013918+0000","last_peered":"2024-09-16T10:23:02.013918+0000","last_clean":"2024-09-16T10:23:02.013918+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:02.013918+0000","last_undegraded":"2024-09-16T10:23:02.013918+0000","last_fullsized":"2024-09-16T10:23:02.013918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'2815","reported_seq":2860,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.131261+0000","last_change":"2024-09-16T10:23:01.017225+0000","last_active":"2024-09-16T10:23:34.131261+0000","last_peered":"2024-09-16T10:23:34.131261+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:23:01.016945+0000","last_became_peered":"2024-09-16T10:23:01.016945+0000","last_unstale":"2024-09-16T10:23:34.131261+0000","last_undegraded":"2024-09-16T10:23:01.011253+0000","last_fullsized":"2024-09-16T10:23:34.131261+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11514822,"num_objects":2813,"num_object_clones":0,"num_object_copies":5626,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2813,"num_write_kb":11246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017286+0000","last_change":"2024-09-16T10:23:01.015639+0000","last_active":"2024-09-16T10:23:01.017286+0000","last_peered":"2024-09-16T10:23:01.017286+0000","last_clean":"2024-09-16T10:23:01.017286+0000","last_became_active":"2024-09-16T10:23:01.015380+0000","last_became_peered":"2024-09-16T10:23:01.015380+0000","last_unstale":"2024-09-16T10:23:01.017286+0000","last_undegraded":"2024-09-16T10:23:01.017286+0000","last_fullsized":"2024-09-16T10:23:01.017286+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:03.395240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.016743+0000","last_change":"2024-09-16T10:23:02.015813+0000","last_active":"2024-09-16T10:23:02.016743+0000","last_peered":"2024-09-16T10:23:02.016743+0000","last_clean":"2024-09-16T10:23:02.016743+0000","last_became_active":"2024-09-16T10:23:02.015242+0000","last_became_peered":"2024-09-16T10:23:02.015242+0000","last_unstale":"2024-09-16T10:23:02.016743+0000","last_undegraded":"2024-09-16T10:23:02.016743+0000","last_fullsized":"2024-09-16T10:23:02.016743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:34:28.723524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"35'2960","reported_seq":3006,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.153263+0000","last_change":"2024-09-16T10:23:01.015555+0000","last_active":"2024-09-16T10:23:34.153263+0000","last_peered":"2024-09-16T10:23:34.153263+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:23:01.015304+0000","last_became_peered":"2024-09-16T10:23:01.015304+0000","last_unstale":"2024-09-16T10:23:34.153263+0000","last_undegraded":"2024-09-16T10:23:01.009840+0000","last_fullsized":"2024-09-16T10:23:34.153263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2960,"log_dups_size":0,"ondisk_log_size":2960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12076940,"num_objects":2952,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":11799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":188286164,"num_objects":45993,"num_object_clones":0,"num_object_copies":91986,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46020,"num_write_kb":183906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":378314752,"data_stored":378191910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46047,"ondisk_log_size":46047,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018893,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294744,"kb_used_data":186832,"kb_used_omap":4,"kb_used_meta":107899,"kb_avail":94077096,"statfs":{"total":96636764160,"available":96334946304,"internally_reserved":0,"allocated":191315968,"data_stored":191113831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":110489264},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215121,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277292,"kb_used_data":177636,"kb_used_omap":4,"kb_used_meta":99643,"kb_avail":94094548,"statfs":{"total":96636764160,"available":96352817152,"internally_reserved":0,"allocated":181899264,"data_stored":181674986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":102035112},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,7,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185978880,"data_stored":185957202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192221184,"data_stored":192221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:39.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:40.359 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:40.361 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:40.379 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-16T10:23:38.565886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":188286183,"num_objects":45995,"num_object_clones":0,"num_object_copies":91990,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46022,"num_write_kb":183908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46050,"ondisk_log_size":46050,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":572036,"kb_used_data":364468,"kb_used_omap":8,"kb_used_meta":207543,"kb_avail":188171644,"statfs":{"total":193273528320,"available":192687763456,"internally_reserved":0,"allocated":373215232,"data_stored":372788817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8872,"internal_metadata":212524376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,4,11,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":66220032,"num_objects":16167,"num_object_clones":0,"num_object_copies":32334,"num_objects_missing_on_primary":-6,"num_objects_missing":-6,"num_objects_degraded":-12,"num_objects_misplaced":0,"num_objects_unfound":-24,"num_objects_dirty":16167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16167,"num_write_kb":64668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002803"},"pg_stats":[{"pgid":"2.f","version":"35'2753","reported_seq":2796,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:34.152446+0000","last_change":"2024-09-16T10:23:01.017258+0000","last_active":"2024-09-16T10:23:34.152446+0000","last_peered":"2024-09-16T10:23:34.152446+0000","last_clean":"2024-09-16T10:23:34.152446+0000","last_became_active":"2024-09-16T10:23:01.016994+0000","last_became_peered":"2024-09-16T10:23:01.016994+0000","last_unstale":"2024-09-16T10:23:34.152446+0000","last_undegraded":"2024-09-16T10:23:34.152446+0000","last_fullsized":"2024-09-16T10:23:34.152446+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2753,"log_dups_size":0,"ondisk_log_size":2753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:53.799911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11276288,"num_objects":2753,"num_object_clones":0,"num_object_copies":5506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2753,"num_write_kb":11012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'2904","reported_seq":2958,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.244832+0000","last_change":"2024-09-16T10:23:23.523752+0000","last_active":"2024-09-16T10:23:35.244832+0000","last_peered":"2024-09-16T10:23:35.244832+0000","last_clean":"2024-09-16T10:23:35.244832+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:35.244832+0000","last_undegraded":"2024-09-16T10:23:35.244832+0000","last_fullsized":"2024-09-16T10:23:35.244832+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11871174,"num_objects":2900,"num_object_clones":0,"num_object_copies":5800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2902,"num_write_kb":11596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'2920","reported_seq":2986,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.242699+0000","last_change":"2024-09-16T10:23:24.034506+0000","last_active":"2024-09-16T10:23:35.242699+0000","last_peered":"2024-09-16T10:23:35.242699+0000","last_clean":"2024-09-16T10:23:35.242699+0000","last_became_active":"2024-09-16T10:23:02.013253+0000","last_became_peered":"2024-09-16T10:23:02.013253+0000","last_unstale":"2024-09-16T10:23:35.242699+0000","last_undegraded":"2024-09-16T10:23:35.242699+0000","last_fullsized":"2024-09-16T10:23:35.242699+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2920,"log_dups_size":0,"ondisk_log_size":2920,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11944419,"num_objects":2917,"num_object_clones":0,"num_object_copies":5834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2919,"num_write_kb":11666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'2953","reported_seq":2998,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.254852+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:35.254852+0000","last_peered":"2024-09-16T10:23:35.254852+0000","last_clean":"2024-09-16T10:23:35.254852+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:35.254852+0000","last_undegraded":"2024-09-16T10:23:35.254852+0000","last_fullsized":"2024-09-16T10:23:35.254852+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2953,"log_dups_size":0,"ondisk_log_size":2953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:59.836622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12095488,"num_objects":2953,"num_object_clones":0,"num_object_copies":5906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2953,"num_write_kb":11812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'2813","reported_seq":2871,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.137696+0000","last_change":"2024-09-16T10:23:01.016337+0000","last_active":"2024-09-16T10:23:34.137696+0000","last_peered":"2024-09-16T10:23:34.137696+0000","last_clean":"2024-09-16T10:22:24.933330+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:34.137696+0000","last_undegraded":"2024-09-16T10:23:01.010093+0000","last_fullsized":"2024-09-16T10:23:34.137696+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2813,"log_dups_size":0,"ondisk_log_size":2813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11494342,"num_objects":2808,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2811,"num_write_kb":11228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2808}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'2803","reported_seq":2850,"reported_epoch":35,"state":"active+recovering","last_fresh":"2024-09-16T10:23:34.112185+0000","last_change":"2024-09-16T10:23:02.010889+0000","last_active":"2024-09-16T10:23:34.112185+0000","last_peered":"2024-09-16T10:23:34.112185+0000","last_clean":"2024-09-16T10:22:25.790311+0000","last_became_active":"2024-09-16T10:23:01.014691+0000","last_became_peered":"2024-09-16T10:23:01.014691+0000","last_unstale":"2024-09-16T10:23:34.112185+0000","last_undegraded":"2024-09-16T10:23:34.112185+0000","last_fullsized":"2024-09-16T10:23:34.112185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":0,"ondisk_log_size":2803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11461574,"num_objects":2800,"num_object_clones":0,"num_object_copies":5600,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2801,"num_write_kb":11195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2800}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'2949","reported_seq":2994,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.142426+0000","last_change":"2024-09-16T10:23:01.018034+0000","last_active":"2024-09-16T10:23:34.142426+0000","last_peered":"2024-09-16T10:23:34.142426+0000","last_clean":"2024-09-16T10:22:25.185476+0000","last_became_active":"2024-09-16T10:23:01.017728+0000","last_became_peered":"2024-09-16T10:23:01.017728+0000","last_unstale":"2024-09-16T10:23:34.142426+0000","last_undegraded":"2024-09-16T10:23:01.011997+0000","last_fullsized":"2024-09-16T10:23:34.142426+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2949,"log_dups_size":0,"ondisk_log_size":2949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055494,"num_objects":2945,"num_object_clones":0,"num_object_copies":5890,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2947,"num_write_kb":11776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'2748","reported_seq":2807,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.133871+0000","last_change":"2024-09-16T10:23:01.016059+0000","last_active":"2024-09-16T10:23:34.133871+0000","last_peered":"2024-09-16T10:23:34.133871+0000","last_clean":"2024-09-16T10:22:15.604528+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:34.133871+0000","last_undegraded":"2024-09-16T10:23:01.009538+0000","last_fullsized":"2024-09-16T10:23:34.133871+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11248099,"num_objects":2747,"num_object_clones":0,"num_object_copies":5494,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":10985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2747}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017433+0000","last_change":"2024-09-16T10:23:01.016598+0000","last_active":"2024-09-16T10:23:01.017433+0000","last_peered":"2024-09-16T10:23:01.017433+0000","last_clean":"2024-09-16T10:23:01.017433+0000","last_became_active":"2024-09-16T10:23:01.016110+0000","last_became_peered":"2024-09-16T10:23:01.016110+0000","last_unstale":"2024-09-16T10:23:01.017433+0000","last_undegraded":"2024-09-16T10:23:01.017433+0000","last_fullsized":"2024-09-16T10:23:01.017433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:12.204767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"35'2924","reported_seq":2991,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.258927+0000","last_change":"2024-09-16T10:23:26.615524+0000","last_active":"2024-09-16T10:23:35.258927+0000","last_peered":"2024-09-16T10:23:35.258927+0000","last_clean":"2024-09-16T10:23:35.258927+0000","last_became_active":"2024-09-16T10:23:02.014504+0000","last_became_peered":"2024-09-16T10:23:02.014504+0000","last_unstale":"2024-09-16T10:23:35.258927+0000","last_undegraded":"2024-09-16T10:23:35.258927+0000","last_fullsized":"2024-09-16T10:23:35.258927+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2924,"log_dups_size":0,"ondisk_log_size":2924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11948998,"num_objects":2919,"num_object_clones":0,"num_object_copies":5838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":11672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013960+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:02.013960+0000","last_peered":"2024-09-16T10:23:02.013960+0000","last_clean":"2024-09-16T10:23:02.013960+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:02.013960+0000","last_undegraded":"2024-09-16T10:23:02.013960+0000","last_fullsized":"2024-09-16T10:23:02.013960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'2973","reported_seq":3014,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.116724+0000","last_change":"2024-09-16T10:23:01.015673+0000","last_active":"2024-09-16T10:23:34.116724+0000","last_peered":"2024-09-16T10:23:34.116724+0000","last_clean":"2024-09-16T10:22:25.841098+0000","last_became_active":"2024-09-16T10:23:01.015452+0000","last_became_peered":"2024-09-16T10:23:01.015452+0000","last_unstale":"2024-09-16T10:23:34.116724+0000","last_undegraded":"2024-09-16T10:23:01.010375+0000","last_fullsized":"2024-09-16T10:23:34.116724+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12145606,"num_objects":2967,"num_object_clones":0,"num_object_copies":5934,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":11864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2967}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":53,"reported_epoch":34,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:01.015931+0000","last_change":"2024-09-16T10:23:01.015931+0000","last_active":"2024-09-16T10:23:01.015931+0000","last_peered":"2024-09-16T10:23:01.015931+0000","last_clean":"2024-09-16T10:22:36.125521+0000","last_became_active":"2024-09-16T10:23:01.015674+0000","last_became_peered":"2024-09-16T10:23:01.015674+0000","last_unstale":"2024-09-16T10:23:01.015931+0000","last_undegraded":"2024-09-16T10:23:01.009899+0000","last_fullsized":"2024-09-16T10:23:01.015931+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:04.682617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0"],"object_location_counts":[{"shards":"0","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'2950","reported_seq":3009,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.195934+0000","last_change":"2024-09-16T10:23:27.615947+0000","last_active":"2024-09-16T10:23:35.195934+0000","last_peered":"2024-09-16T10:23:35.195934+0000","last_clean":"2024-09-16T10:23:35.195934+0000","last_became_active":"2024-09-16T10:23:02.012723+0000","last_became_peered":"2024-09-16T10:23:02.012723+0000","last_unstale":"2024-09-16T10:23:35.195934+0000","last_undegraded":"2024-09-16T10:23:35.195934+0000","last_fullsized":"2024-09-16T10:23:35.195934+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2950,"log_dups_size":0,"ondisk_log_size":2950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12063686,"num_objects":2947,"num_object_clones":0,"num_object_copies":5894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2948,"num_write_kb":11783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.015929+0000","last_change":"2024-09-16T10:23:02.014534+0000","last_active":"2024-09-16T10:23:02.015929+0000","last_peered":"2024-09-16T10:23:02.015929+0000","last_clean":"2024-09-16T10:23:02.015929+0000","last_became_active":"2024-09-16T10:23:02.014138+0000","last_became_peered":"2024-09-16T10:23:02.014138+0000","last_unstale":"2024-09-16T10:23:02.015929+0000","last_undegraded":"2024-09-16T10:23:02.015929+0000","last_fullsized":"2024-09-16T10:23:02.015929+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:19:32.676442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"35'2925","reported_seq":2984,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.254737+0000","last_change":"2024-09-16T10:23:28.126723+0000","last_active":"2024-09-16T10:23:35.254737+0000","last_peered":"2024-09-16T10:23:35.254737+0000","last_clean":"2024-09-16T10:23:35.254737+0000","last_became_active":"2024-09-16T10:23:02.012604+0000","last_became_peered":"2024-09-16T10:23:02.012604+0000","last_unstale":"2024-09-16T10:23:35.254737+0000","last_undegraded":"2024-09-16T10:23:35.254737+0000","last_fullsized":"2024-09-16T10:23:35.254737+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2925,"log_dups_size":0,"ondisk_log_size":2925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11957190,"num_objects":2921,"num_object_clones":0,"num_object_copies":5842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":11679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'2758","reported_seq":2803,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.152638+0000","last_change":"2024-09-16T10:23:01.018002+0000","last_active":"2024-09-16T10:23:34.152638+0000","last_peered":"2024-09-16T10:23:34.152638+0000","last_clean":"2024-09-16T10:22:18.022933+0000","last_became_active":"2024-09-16T10:23:01.017742+0000","last_became_peered":"2024-09-16T10:23:01.017742+0000","last_unstale":"2024-09-16T10:23:34.152638+0000","last_undegraded":"2024-09-16T10:23:01.012206+0000","last_fullsized":"2024-09-16T10:23:34.152638+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2758,"log_dups_size":0,"ondisk_log_size":2758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11289059,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2757,"num_write_kb":11025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2757}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.016349+0000","last_change":"2024-09-16T10:23:01.014951+0000","last_active":"2024-09-16T10:23:01.016349+0000","last_peered":"2024-09-16T10:23:01.016349+0000","last_clean":"2024-09-16T10:23:01.016349+0000","last_became_active":"2024-09-16T10:23:01.014674+0000","last_became_peered":"2024-09-16T10:23:01.014674+0000","last_unstale":"2024-09-16T10:23:01.016349+0000","last_undegraded":"2024-09-16T10:23:01.016349+0000","last_fullsized":"2024-09-16T10:23:01.016349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:05.742015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"35'2899","reported_seq":2944,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.155997+0000","last_change":"2024-09-16T10:23:01.017334+0000","last_active":"2024-09-16T10:23:34.155997+0000","last_peered":"2024-09-16T10:23:34.155997+0000","last_clean":"2024-09-16T10:22:25.739680+0000","last_became_active":"2024-09-16T10:23:01.016901+0000","last_became_peered":"2024-09-16T10:23:01.016901+0000","last_unstale":"2024-09-16T10:23:34.155997+0000","last_undegraded":"2024-09-16T10:23:01.010804+0000","last_fullsized":"2024-09-16T10:23:34.155997+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2899,"log_dups_size":0,"ondisk_log_size":2899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11842985,"num_objects":2894,"num_object_clones":0,"num_object_copies":5788,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2896,"num_write_kb":11568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2894}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013918+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:02.013918+0000","last_peered":"2024-09-16T10:23:02.013918+0000","last_clean":"2024-09-16T10:23:02.013918+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:02.013918+0000","last_undegraded":"2024-09-16T10:23:02.013918+0000","last_fullsized":"2024-09-16T10:23:02.013918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'2815","reported_seq":2860,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.131261+0000","last_change":"2024-09-16T10:23:01.017225+0000","last_active":"2024-09-16T10:23:34.131261+0000","last_peered":"2024-09-16T10:23:34.131261+0000","last_clean":"2024-09-16T10:22:18.022503+0000","last_became_active":"2024-09-16T10:23:01.016945+0000","last_became_peered":"2024-09-16T10:23:01.016945+0000","last_unstale":"2024-09-16T10:23:34.131261+0000","last_undegraded":"2024-09-16T10:23:01.011253+0000","last_fullsized":"2024-09-16T10:23:34.131261+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11514822,"num_objects":2813,"num_object_clones":0,"num_object_copies":5626,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2813,"num_write_kb":11246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0,2","objects":2813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017286+0000","last_change":"2024-09-16T10:23:01.015639+0000","last_active":"2024-09-16T10:23:01.017286+0000","last_peered":"2024-09-16T10:23:01.017286+0000","last_clean":"2024-09-16T10:23:01.017286+0000","last_became_active":"2024-09-16T10:23:01.015380+0000","last_became_peered":"2024-09-16T10:23:01.015380+0000","last_unstale":"2024-09-16T10:23:01.017286+0000","last_undegraded":"2024-09-16T10:23:01.017286+0000","last_fullsized":"2024-09-16T10:23:01.017286+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:03.395240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.016743+0000","last_change":"2024-09-16T10:23:02.015813+0000","last_active":"2024-09-16T10:23:02.016743+0000","last_peered":"2024-09-16T10:23:02.016743+0000","last_clean":"2024-09-16T10:23:02.016743+0000","last_became_active":"2024-09-16T10:23:02.015242+0000","last_became_peered":"2024-09-16T10:23:02.015242+0000","last_unstale":"2024-09-16T10:23:02.016743+0000","last_undegraded":"2024-09-16T10:23:02.016743+0000","last_fullsized":"2024-09-16T10:23:02.016743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:34:28.723524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"35'2960","reported_seq":3006,"reported_epoch":35,"state":"active+recovery_wait+degraded","last_fresh":"2024-09-16T10:23:34.153263+0000","last_change":"2024-09-16T10:23:01.015555+0000","last_active":"2024-09-16T10:23:34.153263+0000","last_peered":"2024-09-16T10:23:34.153263+0000","last_clean":"2024-09-16T10:22:25.487628+0000","last_became_active":"2024-09-16T10:23:01.015304+0000","last_became_peered":"2024-09-16T10:23:01.015304+0000","last_unstale":"2024-09-16T10:23:34.153263+0000","last_undegraded":"2024-09-16T10:23:01.009840+0000","last_fullsized":"2024-09-16T10:23:34.153263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":2960,"log_dups_size":0,"ondisk_log_size":2960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12076940,"num_objects":2952,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2957,"num_write_kb":11799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,2","objects":2951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":188286164,"num_objects":45993,"num_object_clones":0,"num_object_copies":91986,"num_objects_missing_on_primary":18,"num_objects_missing":18,"num_objects_degraded":33,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46020,"num_write_kb":183906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":378314752,"data_stored":378191910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46047,"ondisk_log_size":46047,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018893,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294744,"kb_used_data":186832,"kb_used_omap":4,"kb_used_meta":107899,"kb_avail":94077096,"statfs":{"total":96636764160,"available":96334946304,"internally_reserved":0,"allocated":191315968,"data_stored":191113831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":110489264},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215121,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277292,"kb_used_data":177636,"kb_used_omap":4,"kb_used_meta":99643,"kb_avail":94094548,"statfs":{"total":96636764160,"available":96352817152,"internally_reserved":0,"allocated":181899264,"data_stored":181674986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":102035112},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,7,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":185978880,"data_stored":185957202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192221184,"data_stored":192221184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:40.520 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 28 20 54406 54386 7.58581 4.35938 0.0145432 0.0102907 2024-09-16T10:23:41.520 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 29 20 55732 55712 7.50284 5.17969 0.0149479 0.0104068 2024-09-16T10:23:42.520 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 30 20 57155 57135 7.43802 5.55859 0.0117426 0.0104968 2024-09-16T10:23:43.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:43.520 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 31 20 58450 58430 7.36126 5.05859 0.0135886 0.0106064 2024-09-16T10:23:43.900 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:43.900 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:43.923 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-16T10:23:42.566638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218203367,"num_objects":53299,"num_object_clones":0,"num_object_copies":106598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53326,"num_write_kb":213124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53354,"ondisk_log_size":53354,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":662244,"kb_used_data":422676,"kb_used_omap":8,"kb_used_meta":239543,"kb_avail":188081436,"statfs":{"total":193273528320,"available":192595390464,"internally_reserved":0,"allocated":432820224,"data_stored":432404184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8872,"internal_metadata":245292376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,12,13],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":61669376,"num_objects":15056,"num_object_clones":0,"num_object_copies":30112,"num_objects_missing_on_primary":-18,"num_objects_missing":-18,"num_objects_degraded":-34,"num_objects_misplaced":0,"num_objects_unfound":-5,"num_objects_dirty":15056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15056,"num_write_kb":60224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002532"},"pg_stats":[{"pgid":"2.f","version":"35'3252","reported_seq":3295,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:39.155684+0000","last_change":"2024-09-16T10:23:01.017258+0000","last_active":"2024-09-16T10:23:39.155684+0000","last_peered":"2024-09-16T10:23:39.155684+0000","last_clean":"2024-09-16T10:23:39.155684+0000","last_became_active":"2024-09-16T10:23:01.016994+0000","last_became_peered":"2024-09-16T10:23:01.016994+0000","last_unstale":"2024-09-16T10:23:39.155684+0000","last_undegraded":"2024-09-16T10:23:39.155684+0000","last_fullsized":"2024-09-16T10:23:39.155684+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3252,"log_dups_size":0,"ondisk_log_size":3252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:07:53.799911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13320192,"num_objects":3252,"num_object_clones":0,"num_object_copies":6504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3252,"num_write_kb":13008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"35'3351","reported_seq":3404,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:40.274422+0000","last_change":"2024-09-16T10:23:23.523752+0000","last_active":"2024-09-16T10:23:40.274422+0000","last_peered":"2024-09-16T10:23:40.274422+0000","last_clean":"2024-09-16T10:23:40.274422+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:40.274422+0000","last_undegraded":"2024-09-16T10:23:40.274422+0000","last_fullsized":"2024-09-16T10:23:40.274422+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3351,"log_dups_size":0,"ondisk_log_size":3351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13702086,"num_objects":3347,"num_object_clones":0,"num_object_copies":6694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3349,"num_write_kb":13384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"35'3430","reported_seq":3496,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:40.233187+0000","last_change":"2024-09-16T10:23:24.034506+0000","last_active":"2024-09-16T10:23:40.233187+0000","last_peered":"2024-09-16T10:23:40.233187+0000","last_clean":"2024-09-16T10:23:40.233187+0000","last_became_active":"2024-09-16T10:23:02.013253+0000","last_became_peered":"2024-09-16T10:23:02.013253+0000","last_unstale":"2024-09-16T10:23:40.233187+0000","last_undegraded":"2024-09-16T10:23:40.233187+0000","last_fullsized":"2024-09-16T10:23:40.233187+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3430,"log_dups_size":0,"ondisk_log_size":3430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14033379,"num_objects":3427,"num_object_clones":0,"num_object_copies":6854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3429,"num_write_kb":13706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"35'3376","reported_seq":3422,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:40.245769+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:40.245769+0000","last_peered":"2024-09-16T10:23:40.245769+0000","last_clean":"2024-09-16T10:23:40.245769+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:40.245769+0000","last_undegraded":"2024-09-16T10:23:40.245769+0000","last_fullsized":"2024-09-16T10:23:40.245769+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3376,"log_dups_size":0,"ondisk_log_size":3376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:14:59.836622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13828096,"num_objects":3376,"num_object_clones":0,"num_object_copies":6752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3376,"num_write_kb":13504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"35'3280","reported_seq":3348,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:39.137712+0000","last_change":"2024-09-16T10:23:35.751908+0000","last_active":"2024-09-16T10:23:39.137712+0000","last_peered":"2024-09-16T10:23:39.137712+0000","last_clean":"2024-09-16T10:23:39.137712+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:39.137712+0000","last_undegraded":"2024-09-16T10:23:39.137712+0000","last_fullsized":"2024-09-16T10:23:39.137712+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3280,"log_dups_size":0,"ondisk_log_size":3280,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13407174,"num_objects":3275,"num_object_clones":0,"num_object_copies":6550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3278,"num_write_kb":13096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"35'3214","reported_seq":3270,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:39.151302+0000","last_change":"2024-09-16T10:23:35.500228+0000","last_active":"2024-09-16T10:23:39.151302+0000","last_peered":"2024-09-16T10:23:39.151302+0000","last_clean":"2024-09-16T10:23:39.151302+0000","last_became_active":"2024-09-16T10:23:01.014691+0000","last_became_peered":"2024-09-16T10:23:01.014691+0000","last_unstale":"2024-09-16T10:23:39.151302+0000","last_undegraded":"2024-09-16T10:23:39.151302+0000","last_fullsized":"2024-09-16T10:23:39.151302+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13145030,"num_objects":3211,"num_object_clones":0,"num_object_copies":6422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3212,"num_write_kb":12839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"35'3417","reported_seq":3473,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:39.127171+0000","last_change":"2024-09-16T10:23:35.969315+0000","last_active":"2024-09-16T10:23:39.127171+0000","last_peered":"2024-09-16T10:23:39.127171+0000","last_clean":"2024-09-16T10:23:39.127171+0000","last_became_active":"2024-09-16T10:23:01.017728+0000","last_became_peered":"2024-09-16T10:23:01.017728+0000","last_unstale":"2024-09-16T10:23:39.127171+0000","last_undegraded":"2024-09-16T10:23:39.127171+0000","last_fullsized":"2024-09-16T10:23:39.127171+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3417,"log_dups_size":0,"ondisk_log_size":3417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13972422,"num_objects":3413,"num_object_clones":0,"num_object_copies":6826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3415,"num_write_kb":13648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"35'3253","reported_seq":3319,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:39.140431+0000","last_change":"2024-09-16T10:23:35.701688+0000","last_active":"2024-09-16T10:23:39.140431+0000","last_peered":"2024-09-16T10:23:39.140431+0000","last_clean":"2024-09-16T10:23:39.140431+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:39.140431+0000","last_undegraded":"2024-09-16T10:23:39.140431+0000","last_fullsized":"2024-09-16T10:23:39.140431+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3253,"log_dups_size":0,"ondisk_log_size":3253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13316579,"num_objects":3252,"num_object_clones":0,"num_object_copies":6504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3252,"num_write_kb":13005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017433+0000","last_change":"2024-09-16T10:23:01.016598+0000","last_active":"2024-09-16T10:23:01.017433+0000","last_peered":"2024-09-16T10:23:01.017433+0000","last_clean":"2024-09-16T10:23:01.017433+0000","last_became_active":"2024-09-16T10:23:01.016110+0000","last_became_peered":"2024-09-16T10:23:01.016110+0000","last_unstale":"2024-09-16T10:23:01.017433+0000","last_undegraded":"2024-09-16T10:23:01.017433+0000","last_fullsized":"2024-09-16T10:23:01.017433+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:26:12.204767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"35'3358","reported_seq":3427,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:40.221263+0000","last_change":"2024-09-16T10:23:26.615524+0000","last_active":"2024-09-16T10:23:40.221263+0000","last_peered":"2024-09-16T10:23:40.221263+0000","last_clean":"2024-09-16T10:23:40.221263+0000","last_became_active":"2024-09-16T10:23:02.014504+0000","last_became_peered":"2024-09-16T10:23:02.014504+0000","last_unstale":"2024-09-16T10:23:40.221263+0000","last_undegraded":"2024-09-16T10:23:40.221263+0000","last_fullsized":"2024-09-16T10:23:40.221263+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3358,"log_dups_size":0,"ondisk_log_size":3358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13726662,"num_objects":3353,"num_object_clones":0,"num_object_copies":6706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3356,"num_write_kb":13408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013960+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:02.013960+0000","last_peered":"2024-09-16T10:23:02.013960+0000","last_clean":"2024-09-16T10:23:02.013960+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:02.013960+0000","last_undegraded":"2024-09-16T10:23:02.013960+0000","last_fullsized":"2024-09-16T10:23:02.013960+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"35'3447","reported_seq":3497,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:39.145038+0000","last_change":"2024-09-16T10:23:35.651382+0000","last_active":"2024-09-16T10:23:39.145038+0000","last_peered":"2024-09-16T10:23:39.145038+0000","last_clean":"2024-09-16T10:23:39.145038+0000","last_became_active":"2024-09-16T10:23:01.015452+0000","last_became_peered":"2024-09-16T10:23:01.015452+0000","last_unstale":"2024-09-16T10:23:39.145038+0000","last_undegraded":"2024-09-16T10:23:39.145038+0000","last_fullsized":"2024-09-16T10:23:39.145038+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3447,"log_dups_size":0,"ondisk_log_size":3447,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14087110,"num_objects":3441,"num_object_clones":0,"num_object_copies":6882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3445,"num_write_kb":13760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":59,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:35.990641+0000","last_change":"2024-09-16T10:23:35.990641+0000","last_active":"2024-09-16T10:23:35.990641+0000","last_peered":"2024-09-16T10:23:35.990641+0000","last_clean":"2024-09-16T10:23:35.990641+0000","last_became_active":"2024-09-16T10:23:01.015674+0000","last_became_peered":"2024-09-16T10:23:01.015674+0000","last_unstale":"2024-09-16T10:23:35.990641+0000","last_undegraded":"2024-09-16T10:23:35.990641+0000","last_fullsized":"2024-09-16T10:23:35.990641+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:47:04.682617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"35'3377","reported_seq":3434,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:40.277564+0000","last_change":"2024-09-16T10:23:27.615947+0000","last_active":"2024-09-16T10:23:40.277564+0000","last_peered":"2024-09-16T10:23:40.277564+0000","last_clean":"2024-09-16T10:23:40.277564+0000","last_became_active":"2024-09-16T10:23:02.012723+0000","last_became_peered":"2024-09-16T10:23:02.012723+0000","last_unstale":"2024-09-16T10:23:40.277564+0000","last_undegraded":"2024-09-16T10:23:40.277564+0000","last_fullsized":"2024-09-16T10:23:40.277564+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3377,"log_dups_size":0,"ondisk_log_size":3377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13812678,"num_objects":3374,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3375,"num_write_kb":13491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.015929+0000","last_change":"2024-09-16T10:23:02.014534+0000","last_active":"2024-09-16T10:23:02.015929+0000","last_peered":"2024-09-16T10:23:02.015929+0000","last_clean":"2024-09-16T10:23:02.015929+0000","last_became_active":"2024-09-16T10:23:02.014138+0000","last_became_peered":"2024-09-16T10:23:02.014138+0000","last_unstale":"2024-09-16T10:23:02.015929+0000","last_undegraded":"2024-09-16T10:23:02.015929+0000","last_fullsized":"2024-09-16T10:23:02.015929+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:19:32.676442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"35'3324","reported_seq":3383,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:40.261087+0000","last_change":"2024-09-16T10:23:28.126723+0000","last_active":"2024-09-16T10:23:40.261087+0000","last_peered":"2024-09-16T10:23:40.261087+0000","last_clean":"2024-09-16T10:23:40.261087+0000","last_became_active":"2024-09-16T10:23:02.012604+0000","last_became_peered":"2024-09-16T10:23:02.012604+0000","last_unstale":"2024-09-16T10:23:40.261087+0000","last_undegraded":"2024-09-16T10:23:40.261087+0000","last_fullsized":"2024-09-16T10:23:40.261087+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3324,"log_dups_size":0,"ondisk_log_size":3324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13591494,"num_objects":3320,"num_object_clones":0,"num_object_copies":6640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3322,"num_write_kb":13275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"35'3194","reported_seq":3247,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:39.155929+0000","last_change":"2024-09-16T10:23:35.908725+0000","last_active":"2024-09-16T10:23:39.155929+0000","last_peered":"2024-09-16T10:23:39.155929+0000","last_clean":"2024-09-16T10:23:39.155929+0000","last_became_active":"2024-09-16T10:23:01.017742+0000","last_became_peered":"2024-09-16T10:23:01.017742+0000","last_unstale":"2024-09-16T10:23:39.155929+0000","last_undegraded":"2024-09-16T10:23:39.155929+0000","last_fullsized":"2024-09-16T10:23:39.155929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3194,"log_dups_size":0,"ondisk_log_size":3194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13074915,"num_objects":3193,"num_object_clones":0,"num_object_copies":6386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3193,"num_write_kb":12769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.016349+0000","last_change":"2024-09-16T10:23:01.014951+0000","last_active":"2024-09-16T10:23:01.016349+0000","last_peered":"2024-09-16T10:23:01.016349+0000","last_clean":"2024-09-16T10:23:01.016349+0000","last_became_active":"2024-09-16T10:23:01.014674+0000","last_became_peered":"2024-09-16T10:23:01.014674+0000","last_unstale":"2024-09-16T10:23:01.016349+0000","last_undegraded":"2024-09-16T10:23:01.016349+0000","last_fullsized":"2024-09-16T10:23:01.016349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:58:05.742015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"35'3388","reported_seq":3446,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:39.138692+0000","last_change":"2024-09-16T10:23:35.879147+0000","last_active":"2024-09-16T10:23:39.138692+0000","last_peered":"2024-09-16T10:23:39.138692+0000","last_clean":"2024-09-16T10:23:39.138692+0000","last_became_active":"2024-09-16T10:23:01.016901+0000","last_became_peered":"2024-09-16T10:23:01.016901+0000","last_unstale":"2024-09-16T10:23:39.138692+0000","last_undegraded":"2024-09-16T10:23:39.138692+0000","last_fullsized":"2024-09-16T10:23:39.138692+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3388,"log_dups_size":0,"ondisk_log_size":3388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13845929,"num_objects":3383,"num_object_clones":0,"num_object_copies":6766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3385,"num_write_kb":13524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.013918+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:02.013918+0000","last_peered":"2024-09-16T10:23:02.013918+0000","last_clean":"2024-09-16T10:23:02.013918+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:02.013918+0000","last_undegraded":"2024-09-16T10:23:02.013918+0000","last_fullsized":"2024-09-16T10:23:02.013918+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"35'3266","reported_seq":3320,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:39.154311+0000","last_change":"2024-09-16T10:23:35.812620+0000","last_active":"2024-09-16T10:23:39.154311+0000","last_peered":"2024-09-16T10:23:39.154311+0000","last_clean":"2024-09-16T10:23:39.154311+0000","last_became_active":"2024-09-16T10:23:01.016945+0000","last_became_peered":"2024-09-16T10:23:01.016945+0000","last_unstale":"2024-09-16T10:23:39.154311+0000","last_undegraded":"2024-09-16T10:23:39.154311+0000","last_fullsized":"2024-09-16T10:23:39.154311+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3266,"log_dups_size":0,"ondisk_log_size":3266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13362118,"num_objects":3264,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":13050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-16T10:23:01.017286+0000","last_change":"2024-09-16T10:23:01.015639+0000","last_active":"2024-09-16T10:23:01.017286+0000","last_peered":"2024-09-16T10:23:01.017286+0000","last_clean":"2024-09-16T10:23:01.017286+0000","last_became_active":"2024-09-16T10:23:01.015380+0000","last_became_peered":"2024-09-16T10:23:01.015380+0000","last_unstale":"2024-09-16T10:23:01.017286+0000","last_undegraded":"2024-09-16T10:23:01.017286+0000","last_fullsized":"2024-09-16T10:23:01.017286+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:47:03.395240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:02.016743+0000","last_change":"2024-09-16T10:23:02.015813+0000","last_active":"2024-09-16T10:23:02.016743+0000","last_peered":"2024-09-16T10:23:02.016743+0000","last_clean":"2024-09-16T10:23:02.016743+0000","last_became_active":"2024-09-16T10:23:02.015242+0000","last_became_peered":"2024-09-16T10:23:02.015242+0000","last_unstale":"2024-09-16T10:23:02.016743+0000","last_undegraded":"2024-09-16T10:23:02.016743+0000","last_fullsized":"2024-09-16T10:23:02.016743+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:34:28.723524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"35'3424","reported_seq":3485,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-16T10:23:39.114438+0000","last_change":"2024-09-16T10:23:35.593741+0000","last_active":"2024-09-16T10:23:39.114438+0000","last_peered":"2024-09-16T10:23:39.114438+0000","last_clean":"2024-09-16T10:23:39.114438+0000","last_became_active":"2024-09-16T10:23:01.015304+0000","last_became_peered":"2024-09-16T10:23:01.015304+0000","last_unstale":"2024-09-16T10:23:39.114438+0000","last_undegraded":"2024-09-16T10:23:39.114438+0000","last_fullsized":"2024-09-16T10:23:39.114438+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":3424,"log_dups_size":0,"ondisk_log_size":3424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13977484,"num_objects":3416,"num_object_clones":0,"num_object_copies":6832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":13655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":218203348,"num_objects":53297,"num_object_clones":0,"num_object_copies":106594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53324,"num_write_kb":213122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":437985280,"data_stored":437876890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53351,"ondisk_log_size":53351,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018894,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":337324,"kb_used_data":215460,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94034516,"statfs":{"total":96636764160,"available":96291344384,"internally_reserved":0,"allocated":220631040,"data_stored":220421213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":124776112},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,8,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2549999999999999}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215122,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":324920,"kb_used_data":207216,"kb_used_omap":4,"kb_used_meta":117691,"kb_avail":94046920,"statfs":{"total":96636764160,"available":96304046080,"internally_reserved":0,"allocated":212189184,"data_stored":211982971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":120516264},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,9],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2989999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":216301568,"data_stored":216297955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":221569024,"data_stored":221565411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:43.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:recovered! 2024-09-16T10:23:43.925 INFO:tasks.util.rados:rados -p unfounddel_pool get new_1 - 2024-09-16T10:23:43.925 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_1 - 2024-09-16T10:23:44.024 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/new_1: (2) No such file or directory 2024-09-16T10:23:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.036 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_1 - 2024-09-16T10:23:44.037 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_1 - 2024-09-16T10:23:44.146 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existed_1: (2) No such file or directory 2024-09-16T10:23:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.153 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_1 - 2024-09-16T10:23:44.154 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_1 - 2024-09-16T10:23:44.248 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existing_1: (2) No such file or directory 2024-09-16T10:23:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.253 INFO:tasks.util.rados:rados -p unfounddel_pool get new_2 - 2024-09-16T10:23:44.253 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_2 - 2024-09-16T10:23:44.354 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/new_2: (2) No such file or directory 2024-09-16T10:23:44.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.361 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_2 - 2024-09-16T10:23:44.361 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_2 - 2024-09-16T10:23:44.461 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existed_2: (2) No such file or directory 2024-09-16T10:23:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.468 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_2 - 2024-09-16T10:23:44.468 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_2 - 2024-09-16T10:23:44.520 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 32 20 59653 59633 7.27807 4.69922 0.0123701 0.010728 2024-09-16T10:23:44.559 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existing_2: (2) No such file or directory 2024-09-16T10:23:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.566 INFO:tasks.util.rados:rados -p unfounddel_pool get new_3 - 2024-09-16T10:23:44.566 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_3 - 2024-09-16T10:23:44.660 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/new_3: (2) No such file or directory 2024-09-16T10:23:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.674 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_3 - 2024-09-16T10:23:44.674 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_3 - 2024-09-16T10:23:44.770 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existed_3: (2) No such file or directory 2024-09-16T10:23:44.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.775 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_3 - 2024-09-16T10:23:44.775 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_3 - 2024-09-16T10:23:44.871 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existing_3: (2) No such file or directory 2024-09-16T10:23:44.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:44.880 INFO:tasks.util.rados:rados -p unfounddel_pool get new_4 - 2024-09-16T10:23:44.880 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_4 - 2024-09-16T10:23:44.994 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/new_4: (2) No such file or directory 2024-09-16T10:23:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.010 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_4 - 2024-09-16T10:23:45.010 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_4 - 2024-09-16T10:23:45.107 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existed_4: (2) No such file or directory 2024-09-16T10:23:45.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.119 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_4 - 2024-09-16T10:23:45.119 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_4 - 2024-09-16T10:23:45.231 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existing_4: (2) No such file or directory 2024-09-16T10:23:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.242 INFO:tasks.util.rados:rados -p unfounddel_pool get new_5 - 2024-09-16T10:23:45.242 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_5 - 2024-09-16T10:23:45.358 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/new_5: (2) No such file or directory 2024-09-16T10:23:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.368 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_5 - 2024-09-16T10:23:45.368 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_5 - 2024-09-16T10:23:45.480 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existed_5: (2) No such file or directory 2024-09-16T10:23:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.493 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_5 - 2024-09-16T10:23:45.493 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_5 - 2024-09-16T10:23:45.521 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 33 20 60847 60827 7.19885 4.66406 0.0159555 0.0108472 2024-09-16T10:23:45.597 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existing_5: (2) No such file or directory 2024-09-16T10:23:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.609 INFO:tasks.util.rados:rados -p unfounddel_pool get new_6 - 2024-09-16T10:23:45.610 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_6 - 2024-09-16T10:23:45.954 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/new_6: (2) No such file or directory 2024-09-16T10:23:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:45.962 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_6 - 2024-09-16T10:23:45.963 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_6 - 2024-09-16T10:23:46.074 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existed_6: (2) No such file or directory 2024-09-16T10:23:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.081 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_6 - 2024-09-16T10:23:46.081 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_6 - 2024-09-16T10:23:46.189 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existing_6: (2) No such file or directory 2024-09-16T10:23:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.210 INFO:tasks.util.rados:rados -p unfounddel_pool get new_7 - 2024-09-16T10:23:46.210 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_7 - 2024-09-16T10:23:46.318 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/new_7: (2) No such file or directory 2024-09-16T10:23:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.327 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_7 - 2024-09-16T10:23:46.327 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_7 - 2024-09-16T10:23:46.456 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existed_7: (2) No such file or directory 2024-09-16T10:23:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.463 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_7 - 2024-09-16T10:23:46.463 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_7 - 2024-09-16T10:23:46.521 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 34 20 61970 61950 7.11614 4.38672 0.0146769 0.010973 2024-09-16T10:23:46.558 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existing_7: (2) No such file or directory 2024-09-16T10:23:46.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.568 INFO:tasks.util.rados:rados -p unfounddel_pool get new_8 - 2024-09-16T10:23:46.569 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_8 - 2024-09-16T10:23:46.667 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/new_8: (2) No such file or directory 2024-09-16T10:23:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.676 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_8 - 2024-09-16T10:23:46.676 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_8 - 2024-09-16T10:23:46.776 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existed_8: (2) No such file or directory 2024-09-16T10:23:46.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.793 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_8 - 2024-09-16T10:23:46.793 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_8 - 2024-09-16T10:23:46.890 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existing_8: (2) No such file or directory 2024-09-16T10:23:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:46.900 INFO:tasks.util.rados:rados -p unfounddel_pool get new_9 - 2024-09-16T10:23:46.901 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get new_9 - 2024-09-16T10:23:46.999 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/new_9: (2) No such file or directory 2024-09-16T10:23:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:47.007 INFO:tasks.util.rados:rados -p unfounddel_pool get existed_9 - 2024-09-16T10:23:47.007 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existed_9 - 2024-09-16T10:23:47.100 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existed_9: (2) No such file or directory 2024-09-16T10:23:47.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:47.111 INFO:tasks.util.rados:rados -p unfounddel_pool get existing_9 - 2024-09-16T10:23:47.111 DEBUG:teuthology.orchestra.run.smithi050:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unfounddel_pool get existing_9 - 2024-09-16T10:23:47.218 INFO:teuthology.orchestra.run.smithi050.stderr:error getting unfounddel_pool/existing_9: (2) No such file or directory 2024-09-16T10:23:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:23:47.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-09-16T10:23:47.523 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 35 20 63059 63039 7.03391 4.25391 0.0132673 0.0111017 2024-09-16T10:23:47.784 INFO:tasks.ceph.mon.a.smithi050.stderr:2024-09-16T10:23:47.782+0000 7f0322f76640 -1 mon.a@0(leader).osd e35 definitely_dead 0 2024-09-16T10:23:48.317 INFO:teuthology.orchestra.run.smithi050.stderr:marked in osd.1. 2024-09-16T10:23:48.337 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T10:23:48.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T10:23:48.339 INFO:tasks.ceph.osd.1:Started 2024-09-16T10:23:48.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T10:23:48.440 INFO:teuthology.orchestra.run.smithi050.stderr:no valid command found; 10 closest matches: 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:0 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:1 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:2 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:abort 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:assert 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:config diff 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:config diff get 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:config get 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:config help [] 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:config set ... 2024-09-16T10:23:48.441 INFO:teuthology.orchestra.run.smithi050.stderr:admin_socket: invalid command 2024-09-16T10:23:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T10:23:48.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-16T10:23:48.523 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 36 20 64082 64062 6.94952 3.99609 0.0306992 0.0111752 2024-09-16T10:23:48.932 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-16T10:23:48.930+0000 7f644f814640 -1 Falling back to public interface 2024-09-16T10:23:49.523 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 37 20 64082 64062 6.76171 0 - 0.0111752 2024-09-16T10:23:50.523 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 38 20 64293 64273 6.60548 0.412109 0.0104983 0.0118223 2024-09-16T10:23:51.523 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 39 20 65559 65539 6.56291 4.94531 0.0322708 0.0118971 2024-09-16T10:23:52.204 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-16T10:23:52.201+0000 7f644f814640 -1 osd.1 25 log_to_monitors true 2024-09-16T10:23:52.524 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:23:52.523057+0000 min lat: 0.00139595 max lat: 2.06993 avg lat: 0.0119815 2024-09-16T10:23:52.524 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:23:52.524 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 40 20 66766 66746 6.5167 4.71484 0.0152792 0.0119815 2024-09-16T10:23:53.380 INFO:tasks.ceph.osd.1.smithi050.stderr:2024-09-16T10:23:53.377+0000 7f6442fb8640 -1 osd.1 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T10:23:53.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T10:23:53.523 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 41 20 67683 67663 6.44513 3.58203 0.0355169 0.0121142 2024-09-16T10:23:53.629 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:23:53.629 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-09-16T10:23:53.629 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-09-16T10:23:53.629 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:23:53.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for osd.1 to be up 2024-09-16T10:23:53.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:23:54.151 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:54.151 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":39,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:23:52.333548+0000","last_up_change":"2024-09-16T10:22:55.836931+0000","last_in_change":"2024-09-16T10:23:48.304391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T10:22:00.707491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.4900000095367432,"score_stable":1.4900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-16T10:22:12.124014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.8700000047683716,"score_stable":1.8700000047683716,"optimal_score":0.67000001668930054,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":20,"up_from":24,"up_thru":37,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2470025375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":2470025375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":2470025375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":2470025375}]},"public_addr":"172.21.15.50:6804/2470025375","cluster_addr":"172.21.15.50:6805/2470025375","heartbeat_back_addr":"172.21.15.50:6807/2470025375","heartbeat_front_addr":"172.21.15.50:6806/2470025375","state":["exists","up"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":24,"down_at":26,"lost_at":26,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":202723672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":202723672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":202723672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":202723672}]},"public_addr":"172.21.15.50:6800/202723672","cluster_addr":"172.21.15.50:6801/202723672","heartbeat_back_addr":"172.21.15.50:6803/202723672","heartbeat_front_addr":"172.21.15.50:6802/202723672","state":["exists"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":17,"up_from":30,"up_thru":37,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1896614959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1896614959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":1896614959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1896614959}]},"public_addr":"172.21.15.50:6800/1896614959","cluster_addr":"172.21.15.50:6801/1896614959","heartbeat_back_addr":"172.21.15.50:6803/1896614959","heartbeat_front_addr":"172.21.15.50:6802/1896614959","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.244822+0000","dead_epoch":21},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.213317+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.228331+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[0,2]},{"pgid":"1.2","osds":[0,2]},{"pgid":"1.3","osds":[2,0]},{"pgid":"1.4","osds":[0,2]},{"pgid":"1.6","osds":[0,2]},{"pgid":"1.7","osds":[2,0]},{"pgid":"2.0","osds":[2,0]},{"pgid":"2.1","osds":[2,0]},{"pgid":"2.2","osds":[0,2]},{"pgid":"2.3","osds":[2,0]},{"pgid":"2.4","osds":[0,2]},{"pgid":"2.5","osds":[0,2]},{"pgid":"2.6","osds":[0,2]},{"pgid":"2.7","osds":[0,2]},{"pgid":"2.9","osds":[0,2]},{"pgid":"2.a","osds":[0,2]},{"pgid":"2.d","osds":[2,0]},{"pgid":"2.f","osds":[0,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:23:54.524 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 42 20 68402 68382 6.35854 2.80859 0.0034518 0.0122391 2024-09-16T10:23:55.524 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 43 20 68402 68382 6.21068 0 - 0.0122391 2024-09-16T10:23:56.524 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 44 20 68402 68382 6.06954 0 - 0.0122391 2024-09-16T10:23:57.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:23:57.524 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 45 20 68746 68726 5.96453 0.447917 0.0294425 0.0130879 2024-09-16T10:23:57.753 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:57.753 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":42,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:23:56.359809+0000","last_up_change":"2024-09-16T10:23:54.356705+0000","last_in_change":"2024-09-16T10:23:48.304391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T10:22:00.707491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-16T10:22:12.124014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":20,"up_from":24,"up_thru":41,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2470025375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":2470025375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":2470025375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":2470025375}]},"public_addr":"172.21.15.50:6804/2470025375","cluster_addr":"172.21.15.50:6805/2470025375","heartbeat_back_addr":"172.21.15.50:6807/2470025375","heartbeat_front_addr":"172.21.15.50:6806/2470025375","state":["exists","up"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":25,"up_from":40,"up_thru":41,"down_at":26,"lost_at":26,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":832073999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":832073999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":832073999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":832073999}]},"public_addr":"172.21.15.50:6808/832073999","cluster_addr":"172.21.15.50:6809/832073999","heartbeat_back_addr":"172.21.15.50:6811/832073999","heartbeat_front_addr":"172.21.15.50:6810/832073999","state":["exists","up"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":17,"up_from":30,"up_thru":41,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1896614959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1896614959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":1896614959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1896614959}]},"public_addr":"172.21.15.50:6800/1896614959","cluster_addr":"172.21.15.50:6801/1896614959","heartbeat_back_addr":"172.21.15.50:6803/1896614959","heartbeat_front_addr":"172.21.15.50:6802/1896614959","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.244822+0000","dead_epoch":21},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.213317+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.228331+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"2.0","osds":[2]},{"pgid":"2.1","osds":[2]},{"pgid":"2.2","osds":[0]},{"pgid":"2.3","osds":[2]},{"pgid":"2.4","osds":[0]},{"pgid":"2.5","osds":[0]},{"pgid":"2.6","osds":[0]},{"pgid":"2.7","osds":[0]},{"pgid":"2.9","osds":[0]},{"pgid":"2.a","osds":[0]},{"pgid":"2.d","osds":[2]},{"pgid":"2.f","osds":[0]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:23:57.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:osd.1 is up 2024-09-16T10:23:57.781 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for clean 2024-09-16T10:23:57.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:58.328 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:58.330 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:58.354 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-16T10:23:56.570961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":279733228,"num_objects":68313,"num_object_clones":0,"num_object_copies":136626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34236,"num_objects_unfound":0,"num_objects_dirty":68313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50137,"num_write_kb":200427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":4598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68377,"ondisk_log_size":68377,"up":36,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":72,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":856344,"kb_used_data":544952,"kb_used_omap":12,"kb_used_meta":311347,"kb_avail":282259176,"statfs":{"total":289910292480,"available":289033396224,"internally_reserved":0,"allocated":558030848,"data_stored":557316634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":318819330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,7,0,0,0,0,0,11],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17317888,"num_objects":4228,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34236,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4228,"num_write_kb":16912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.231465"},"pg_stats":[{"pgid":"2.f","version":"39'4183","reported_seq":4239,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.162262+0000","last_change":"2024-09-16T10:23:50.351182+0000","last_active":"2024-09-16T10:23:54.162262+0000","last_peered":"2024-09-16T10:23:54.162262+0000","last_clean":"2024-09-16T10:23:54.162262+0000","last_became_active":"2024-09-16T10:23:50.335726+0000","last_became_peered":"2024-09-16T10:23:50.335726+0000","last_unstale":"2024-09-16T10:23:54.162262+0000","last_undegraded":"2024-09-16T10:23:54.162262+0000","last_fullsized":"2024-09-16T10:23:54.162262+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:49.636034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17133568,"num_objects":4183,"num_object_clones":0,"num_object_copies":8366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4183,"num_objects_unfound":0,"num_objects_dirty":4183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":16732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4183}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"40'4278","reported_seq":4430,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T10:23:54.396025+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:23:54.396025+0000","last_peered":"2024-09-16T10:23:54.396025+0000","last_clean":"2024-09-16T10:23:54.396025+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:54.396025+0000","last_undegraded":"2024-09-16T10:23:54.396025+0000","last_fullsized":"2024-09-16T10:23:54.396025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498112,"num_objects":4272,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":1964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'4339","reported_seq":4512,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.396512+0000","last_change":"2024-09-16T10:23:54.396668+0000","last_active":"2024-09-16T10:23:54.396512+0000","last_peered":"2024-09-16T10:23:54.392083+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:50.333009+0000","last_became_peered":"2024-09-16T10:23:50.333009+0000","last_unstale":"2024-09-16T10:23:54.396512+0000","last_undegraded":"2024-09-16T10:23:54.396512+0000","last_fullsized":"2024-09-16T10:23:54.396512+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:13:59.630290+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17756160,"num_objects":4335,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":396,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"40'4310","reported_seq":4361,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T10:23:54.399498+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:54.399498+0000","last_peered":"2024-09-16T10:23:54.399498+0000","last_clean":"2024-09-16T10:23:54.399498+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:54.399498+0000","last_undegraded":"2024-09-16T10:23:54.399498+0000","last_fullsized":"2024-09-16T10:23:54.399498+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17653760,"num_objects":4310,"num_object_clones":0,"num_object_copies":8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":17240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'4262","reported_seq":4412,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T10:23:54.204475+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:23:54.204475+0000","last_peered":"2024-09-16T10:23:54.204475+0000","last_clean":"2024-09-16T10:23:54.204475+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:54.204475+0000","last_undegraded":"2024-09-16T10:23:54.204475+0000","last_fullsized":"2024-09-16T10:23:54.204475+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:31.412867+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17428480,"num_objects":4255,"num_object_clones":0,"num_object_copies":8510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":904,"num_write_kb":3616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'4213","reported_seq":4362,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.185461+0000","last_change":"2024-09-16T10:23:50.339479+0000","last_active":"2024-09-16T10:23:54.185461+0000","last_peered":"2024-09-16T10:23:54.185461+0000","last_clean":"2024-09-16T10:23:54.185461+0000","last_became_active":"2024-09-16T10:23:50.334121+0000","last_became_peered":"2024-09-16T10:23:50.334121+0000","last_unstale":"2024-09-16T10:23:54.185461+0000","last_undegraded":"2024-09-16T10:23:54.185461+0000","last_fullsized":"2024-09-16T10:23:54.185461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4213,"log_dups_size":0,"ondisk_log_size":4213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:13.266606+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":4208,"num_object_clones":0,"num_object_copies":8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4208,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":838,"num_write_kb":3352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4208}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'4401","reported_seq":4560,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.179578+0000","last_change":"2024-09-16T10:23:50.344370+0000","last_active":"2024-09-16T10:23:54.179578+0000","last_peered":"2024-09-16T10:23:54.179578+0000","last_clean":"2024-09-16T10:23:54.179578+0000","last_became_active":"2024-09-16T10:23:50.334013+0000","last_became_peered":"2024-09-16T10:23:50.334013+0000","last_unstale":"2024-09-16T10:23:54.179578+0000","last_undegraded":"2024-09-16T10:23:54.179578+0000","last_fullsized":"2024-09-16T10:23:54.179578+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:22.668041+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18001920,"num_objects":4395,"num_object_clones":0,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4395,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":643,"num_write_kb":2572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'4233","reported_seq":4376,"reported_epoch":39,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:23:54.193511+0000","last_change":"2024-09-16T10:23:52.471903+0000","last_active":"2024-09-16T10:23:54.193511+0000","last_peered":"2024-09-16T10:23:54.193511+0000","last_clean":"2024-09-16T10:23:54.193511+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:54.193511+0000","last_undegraded":"2024-09-16T10:23:54.193511+0000","last_fullsized":"2024-09-16T10:23:54.193511+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":3324,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17330659,"num_objects":4232,"num_object_clones":0,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4232,"num_write_kb":16925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:50.338997+0000","last_change":"2024-09-16T10:23:50.335896+0000","last_active":"2024-09-16T10:23:50.338997+0000","last_peered":"2024-09-16T10:23:50.338997+0000","last_clean":"2024-09-16T10:23:50.338997+0000","last_became_active":"2024-09-16T10:23:50.335445+0000","last_became_peered":"2024-09-16T10:23:50.335445+0000","last_unstale":"2024-09-16T10:23:50.338997+0000","last_undegraded":"2024-09-16T10:23:50.338997+0000","last_fullsized":"2024-09-16T10:23:50.338997+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:50:58.232445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"39'4204","reported_seq":4287,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.374131+0000","last_change":"2024-09-16T10:23:54.374253+0000","last_active":"2024-09-16T10:23:54.374131+0000","last_peered":"2024-09-16T10:23:54.345012+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:50.338673+0000","last_became_peered":"2024-09-16T10:23:50.338673+0000","last_unstale":"2024-09-16T10:23:54.374131+0000","last_undegraded":"2024-09-16T10:23:54.374131+0000","last_fullsized":"2024-09-16T10:23:54.374131+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17191878,"num_objects":4199,"num_object_clones":0,"num_object_copies":8398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":16792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T10:23:52.345559+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:52.345559+0000","last_peered":"2024-09-16T10:23:52.345559+0000","last_clean":"2024-09-16T10:23:52.345559+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:52.345559+0000","last_undegraded":"2024-09-16T10:23:52.345559+0000","last_fullsized":"2024-09-16T10:23:52.345559+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'4426","reported_seq":4490,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.150603+0000","last_change":"2024-09-16T10:23:50.346372+0000","last_active":"2024-09-16T10:23:54.150603+0000","last_peered":"2024-09-16T10:23:54.150603+0000","last_clean":"2024-09-16T10:23:54.150603+0000","last_became_active":"2024-09-16T10:23:50.334044+0000","last_became_peered":"2024-09-16T10:23:50.334044+0000","last_unstale":"2024-09-16T10:23:54.150603+0000","last_undegraded":"2024-09-16T10:23:54.150603+0000","last_fullsized":"2024-09-16T10:23:54.150603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18097094,"num_objects":4420,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4420,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4424,"num_write_kb":17676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":72,"reported_epoch":38,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:50.355920+0000","last_change":"2024-09-16T10:23:50.342443+0000","last_active":"2024-09-16T10:23:50.355920+0000","last_peered":"2024-09-16T10:23:50.355920+0000","last_clean":"2024-09-16T10:23:50.355920+0000","last_became_active":"2024-09-16T10:23:50.339762+0000","last_became_peered":"2024-09-16T10:23:50.339762+0000","last_unstale":"2024-09-16T10:23:50.355920+0000","last_undegraded":"2024-09-16T10:23:50.355920+0000","last_fullsized":"2024-09-16T10:23:50.355920+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:21.709026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'4282","reported_seq":4375,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.391281+0000","last_change":"2024-09-16T10:23:54.391428+0000","last_active":"2024-09-16T10:23:54.391281+0000","last_peered":"2024-09-16T10:23:54.351639+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:50.352466+0000","last_became_peered":"2024-09-16T10:23:50.352466+0000","last_unstale":"2024-09-16T10:23:54.391281+0000","last_undegraded":"2024-09-16T10:23:54.391281+0000","last_fullsized":"2024-09-16T10:23:54.391281+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17519558,"num_objects":4279,"num_object_clones":0,"num_object_copies":8558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":17111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":30,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.373686+0000","last_change":"2024-09-16T10:23:54.373838+0000","last_active":"2024-09-16T10:23:54.373686+0000","last_peered":"2024-09-16T10:23:52.344657+0000","last_clean":"2024-09-16T10:23:52.344657+0000","last_became_active":"2024-09-16T10:23:50.339044+0000","last_became_peered":"2024-09-16T10:23:50.339044+0000","last_unstale":"2024-09-16T10:23:54.373686+0000","last_undegraded":"2024-09-16T10:23:54.373686+0000","last_fullsized":"2024-09-16T10:23:54.373686+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:19:12.734471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"39'4198","reported_seq":4272,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.375299+0000","last_change":"2024-09-16T10:23:54.375453+0000","last_active":"2024-09-16T10:23:54.375299+0000","last_peered":"2024-09-16T10:23:54.357802+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:50.336831+0000","last_became_peered":"2024-09-16T10:23:50.336831+0000","last_unstale":"2024-09-16T10:23:54.375299+0000","last_undegraded":"2024-09-16T10:23:54.375299+0000","last_fullsized":"2024-09-16T10:23:54.375299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4198,"log_dups_size":0,"ondisk_log_size":4198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17171398,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":16771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'4149","reported_seq":4214,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.208090+0000","last_change":"2024-09-16T10:23:50.350102+0000","last_active":"2024-09-16T10:23:54.208090+0000","last_peered":"2024-09-16T10:23:54.208090+0000","last_clean":"2024-09-16T10:23:54.208090+0000","last_became_active":"2024-09-16T10:23:50.340205+0000","last_became_peered":"2024-09-16T10:23:50.340205+0000","last_unstale":"2024-09-16T10:23:54.208090+0000","last_undegraded":"2024-09-16T10:23:54.208090+0000","last_fullsized":"2024-09-16T10:23:54.208090+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16986595,"num_objects":4148,"num_object_clones":0,"num_object_copies":8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4148,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":16589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4148}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:50.341039+0000","last_change":"2024-09-16T10:23:50.339998+0000","last_active":"2024-09-16T10:23:50.341039+0000","last_peered":"2024-09-16T10:23:50.341039+0000","last_clean":"2024-09-16T10:23:50.341039+0000","last_became_active":"2024-09-16T10:23:50.339657+0000","last_became_peered":"2024-09-16T10:23:50.339657+0000","last_unstale":"2024-09-16T10:23:50.341039+0000","last_undegraded":"2024-09-16T10:23:50.341039+0000","last_fullsized":"2024-09-16T10:23:50.341039+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:22:01.061905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"39'4335","reported_seq":4448,"reported_epoch":39,"state":"active+clean+remapped+scrubbing","last_fresh":"2024-09-16T10:23:54.222237+0000","last_change":"2024-09-16T10:23:53.440714+0000","last_active":"2024-09-16T10:23:54.222237+0000","last_peered":"2024-09-16T10:23:54.222237+0000","last_clean":"2024-09-16T10:23:54.222237+0000","last_became_active":"2024-09-16T10:23:50.333769+0000","last_became_peered":"2024-09-16T10:23:50.333769+0000","last_unstale":"2024-09-16T10:23:54.222237+0000","last_undegraded":"2024-09-16T10:23:54.222237+0000","last_fullsized":"2024-09-16T10:23:54.222237+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":1899,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17724841,"num_objects":4330,"num_object_clones":0,"num_object_copies":8660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4330,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":17312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4330}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T10:23:52.344562+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:52.344562+0000","last_peered":"2024-09-16T10:23:52.344562+0000","last_clean":"2024-09-16T10:23:52.344562+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:52.344562+0000","last_undegraded":"2024-09-16T10:23:52.344562+0000","last_fullsized":"2024-09-16T10:23:52.344562+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'4165","reported_seq":4232,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.193029+0000","last_change":"2024-09-16T10:23:50.341146+0000","last_active":"2024-09-16T10:23:54.193029+0000","last_peered":"2024-09-16T10:23:54.193029+0000","last_clean":"2024-09-16T10:23:54.193029+0000","last_became_active":"2024-09-16T10:23:50.335911+0000","last_became_peered":"2024-09-16T10:23:50.335911+0000","last_unstale":"2024-09-16T10:23:54.193029+0000","last_undegraded":"2024-09-16T10:23:54.193029+0000","last_fullsized":"2024-09-16T10:23:54.193029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4165,"log_dups_size":0,"ondisk_log_size":4165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044422,"num_objects":4163,"num_object_clones":0,"num_object_copies":8326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4163,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4163,"num_write_kb":16646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":38,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:50.334816+0000","last_change":"2024-09-16T10:23:50.334067+0000","last_active":"2024-09-16T10:23:50.334816+0000","last_peered":"2024-09-16T10:23:50.334816+0000","last_clean":"2024-09-16T10:23:50.334816+0000","last_became_active":"2024-09-16T10:23:50.333431+0000","last_became_peered":"2024-09-16T10:23:50.333431+0000","last_unstale":"2024-09-16T10:23:50.334816+0000","last_undegraded":"2024-09-16T10:23:50.334816+0000","last_fullsized":"2024-09-16T10:23:50.334816+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:18:05.251217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.389070+0000","last_change":"2024-09-16T10:23:54.390162+0000","last_active":"2024-09-16T10:23:54.389070+0000","last_peered":"2024-09-16T10:23:52.352462+0000","last_clean":"2024-09-16T10:23:52.352462+0000","last_became_active":"2024-09-16T10:23:50.336909+0000","last_became_peered":"2024-09-16T10:23:50.336909+0000","last_unstale":"2024-09-16T10:23:54.389070+0000","last_undegraded":"2024-09-16T10:23:54.389070+0000","last_fullsized":"2024-09-16T10:23:54.389070+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:31:51.284344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"39'4396","reported_seq":4474,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.205431+0000","last_change":"2024-09-16T10:23:50.380060+0000","last_active":"2024-09-16T10:23:54.205431+0000","last_peered":"2024-09-16T10:23:54.205431+0000","last_clean":"2024-09-16T10:23:54.205431+0000","last_became_active":"2024-09-16T10:23:50.356077+0000","last_became_peered":"2024-09-16T10:23:50.356077+0000","last_unstale":"2024-09-16T10:23:54.205431+0000","last_undegraded":"2024-09-16T10:23:54.205431+0000","last_fullsized":"2024-09-16T10:23:54.205431+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4396,"log_dups_size":0,"ondisk_log_size":4396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17958796,"num_objects":4388,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4388,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":17543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4388}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":279733209,"num_objects":68311,"num_object_clones":0,"num_object_copies":136622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34235,"num_objects_unfound":0,"num_objects_dirty":68311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50135,"num_write_kb":200425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":559783936,"data_stored":559675546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68374,"ondisk_log_size":68374,"up":24,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":12,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018897,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422000,"kb_used_data":273892,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93949840,"statfs":{"total":96636764160,"available":96204636160,"internally_reserved":0,"allocated":280465408,"data_stored":280232085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":151645872},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.464}]}]},{"osd":1,"up_from":0,"seq":1,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":331776,"data_stored":75863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":28176042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":24,"seq":103079215125,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406492,"kb_used_data":270736,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":93965348,"statfs":{"total":96636764160,"available":96220516352,"internally_reserved":0,"allocated":277233664,"data_stored":277008686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":138997416},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,11],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279543808,"data_stored":279540195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":13524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280125440,"data_stored":280121827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:58.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:58.524 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 46 20 69299 69279 5.88184 2.16016 0.0392574 0.0132722 2024-09-16T10:23:58.901 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:58.903 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:58.925 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-16T10:23:58.572366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":279733228,"num_objects":68313,"num_object_clones":0,"num_object_copies":136626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34236,"num_objects_unfound":0,"num_objects_dirty":68313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50137,"num_write_kb":200427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":4598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68377,"ondisk_log_size":68377,"up":39,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":879300,"kb_used_data":545744,"kb_used_omap":12,"kb_used_meta":333491,"kb_avail":282236220,"statfs":{"total":289910292480,"available":289009889280,"internally_reserved":0,"allocated":558841856,"data_stored":558060830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":341494786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,10,5,4,1,0,0,11],"upper_bound":2048},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17317888,"num_objects":4228,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34236,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4228,"num_write_kb":16912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003997"},"pg_stats":[{"pgid":"2.f","version":"39'4183","reported_seq":4239,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.162262+0000","last_change":"2024-09-16T10:23:50.351182+0000","last_active":"2024-09-16T10:23:54.162262+0000","last_peered":"2024-09-16T10:23:54.162262+0000","last_clean":"2024-09-16T10:23:54.162262+0000","last_became_active":"2024-09-16T10:23:50.335726+0000","last_became_peered":"2024-09-16T10:23:50.335726+0000","last_unstale":"2024-09-16T10:23:54.162262+0000","last_undegraded":"2024-09-16T10:23:54.162262+0000","last_fullsized":"2024-09-16T10:23:54.162262+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:49.636034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17133568,"num_objects":4183,"num_object_clones":0,"num_object_copies":8366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4183,"num_objects_unfound":0,"num_objects_dirty":4183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":16732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4183}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"40'4278","reported_seq":4430,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T10:23:54.396025+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:23:54.396025+0000","last_peered":"2024-09-16T10:23:54.396025+0000","last_clean":"2024-09-16T10:23:54.396025+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:54.396025+0000","last_undegraded":"2024-09-16T10:23:54.396025+0000","last_fullsized":"2024-09-16T10:23:54.396025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498112,"num_objects":4272,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":1964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'4339","reported_seq":4512,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.396512+0000","last_change":"2024-09-16T10:23:54.396668+0000","last_active":"2024-09-16T10:23:54.396512+0000","last_peered":"2024-09-16T10:23:54.392083+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:50.333009+0000","last_became_peered":"2024-09-16T10:23:50.333009+0000","last_unstale":"2024-09-16T10:23:54.396512+0000","last_undegraded":"2024-09-16T10:23:54.396512+0000","last_fullsized":"2024-09-16T10:23:54.396512+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:13:59.630290+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17756160,"num_objects":4335,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":396,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"40'4310","reported_seq":4361,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T10:23:54.399498+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:54.399498+0000","last_peered":"2024-09-16T10:23:54.399498+0000","last_clean":"2024-09-16T10:23:54.399498+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:54.399498+0000","last_undegraded":"2024-09-16T10:23:54.399498+0000","last_fullsized":"2024-09-16T10:23:54.399498+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17653760,"num_objects":4310,"num_object_clones":0,"num_object_copies":8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":17240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'4262","reported_seq":4412,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T10:23:54.204475+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:23:54.204475+0000","last_peered":"2024-09-16T10:23:54.204475+0000","last_clean":"2024-09-16T10:23:54.204475+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:54.204475+0000","last_undegraded":"2024-09-16T10:23:54.204475+0000","last_fullsized":"2024-09-16T10:23:54.204475+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:31.412867+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17428480,"num_objects":4255,"num_object_clones":0,"num_object_copies":8510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":904,"num_write_kb":3616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'4213","reported_seq":4362,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.185461+0000","last_change":"2024-09-16T10:23:50.339479+0000","last_active":"2024-09-16T10:23:54.185461+0000","last_peered":"2024-09-16T10:23:54.185461+0000","last_clean":"2024-09-16T10:23:54.185461+0000","last_became_active":"2024-09-16T10:23:50.334121+0000","last_became_peered":"2024-09-16T10:23:50.334121+0000","last_unstale":"2024-09-16T10:23:54.185461+0000","last_undegraded":"2024-09-16T10:23:54.185461+0000","last_fullsized":"2024-09-16T10:23:54.185461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4213,"log_dups_size":0,"ondisk_log_size":4213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:13.266606+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":4208,"num_object_clones":0,"num_object_copies":8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4208,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":838,"num_write_kb":3352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4208}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'4401","reported_seq":4560,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.179578+0000","last_change":"2024-09-16T10:23:50.344370+0000","last_active":"2024-09-16T10:23:54.179578+0000","last_peered":"2024-09-16T10:23:54.179578+0000","last_clean":"2024-09-16T10:23:54.179578+0000","last_became_active":"2024-09-16T10:23:50.334013+0000","last_became_peered":"2024-09-16T10:23:50.334013+0000","last_unstale":"2024-09-16T10:23:54.179578+0000","last_undegraded":"2024-09-16T10:23:54.179578+0000","last_fullsized":"2024-09-16T10:23:54.179578+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:22.668041+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18001920,"num_objects":4395,"num_object_clones":0,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4395,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":643,"num_write_kb":2572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'4233","reported_seq":4376,"reported_epoch":39,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:23:54.193511+0000","last_change":"2024-09-16T10:23:52.471903+0000","last_active":"2024-09-16T10:23:54.193511+0000","last_peered":"2024-09-16T10:23:54.193511+0000","last_clean":"2024-09-16T10:23:54.193511+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:54.193511+0000","last_undegraded":"2024-09-16T10:23:54.193511+0000","last_fullsized":"2024-09-16T10:23:54.193511+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":3324,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17330659,"num_objects":4232,"num_object_clones":0,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4232,"num_write_kb":16925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'4204","reported_seq":4287,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.374131+0000","last_change":"2024-09-16T10:23:54.374253+0000","last_active":"2024-09-16T10:23:54.374131+0000","last_peered":"2024-09-16T10:23:54.345012+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:50.338673+0000","last_became_peered":"2024-09-16T10:23:50.338673+0000","last_unstale":"2024-09-16T10:23:54.374131+0000","last_undegraded":"2024-09-16T10:23:54.374131+0000","last_fullsized":"2024-09-16T10:23:54.374131+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17191878,"num_objects":4199,"num_object_clones":0,"num_object_copies":8398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":16792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T10:23:52.345559+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:52.345559+0000","last_peered":"2024-09-16T10:23:52.345559+0000","last_clean":"2024-09-16T10:23:52.345559+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:52.345559+0000","last_undegraded":"2024-09-16T10:23:52.345559+0000","last_fullsized":"2024-09-16T10:23:52.345559+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'4426","reported_seq":4490,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.150603+0000","last_change":"2024-09-16T10:23:50.346372+0000","last_active":"2024-09-16T10:23:54.150603+0000","last_peered":"2024-09-16T10:23:54.150603+0000","last_clean":"2024-09-16T10:23:54.150603+0000","last_became_active":"2024-09-16T10:23:50.334044+0000","last_became_peered":"2024-09-16T10:23:50.334044+0000","last_unstale":"2024-09-16T10:23:54.150603+0000","last_undegraded":"2024-09-16T10:23:54.150603+0000","last_fullsized":"2024-09-16T10:23:54.150603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18097094,"num_objects":4420,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4420,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4424,"num_write_kb":17676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":72,"reported_epoch":38,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:50.355920+0000","last_change":"2024-09-16T10:23:50.342443+0000","last_active":"2024-09-16T10:23:50.355920+0000","last_peered":"2024-09-16T10:23:50.355920+0000","last_clean":"2024-09-16T10:23:50.355920+0000","last_became_active":"2024-09-16T10:23:50.339762+0000","last_became_peered":"2024-09-16T10:23:50.339762+0000","last_unstale":"2024-09-16T10:23:50.355920+0000","last_undegraded":"2024-09-16T10:23:50.355920+0000","last_fullsized":"2024-09-16T10:23:50.355920+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:21.709026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'4282","reported_seq":4375,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.391281+0000","last_change":"2024-09-16T10:23:54.391428+0000","last_active":"2024-09-16T10:23:54.391281+0000","last_peered":"2024-09-16T10:23:54.351639+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:50.352466+0000","last_became_peered":"2024-09-16T10:23:50.352466+0000","last_unstale":"2024-09-16T10:23:54.391281+0000","last_undegraded":"2024-09-16T10:23:54.391281+0000","last_fullsized":"2024-09-16T10:23:54.391281+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17519558,"num_objects":4279,"num_object_clones":0,"num_object_copies":8558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":17111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'4198","reported_seq":4272,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.375299+0000","last_change":"2024-09-16T10:23:54.375453+0000","last_active":"2024-09-16T10:23:54.375299+0000","last_peered":"2024-09-16T10:23:54.357802+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:50.336831+0000","last_became_peered":"2024-09-16T10:23:50.336831+0000","last_unstale":"2024-09-16T10:23:54.375299+0000","last_undegraded":"2024-09-16T10:23:54.375299+0000","last_fullsized":"2024-09-16T10:23:54.375299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4198,"log_dups_size":0,"ondisk_log_size":4198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17171398,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":16771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'4149","reported_seq":4214,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.208090+0000","last_change":"2024-09-16T10:23:50.350102+0000","last_active":"2024-09-16T10:23:54.208090+0000","last_peered":"2024-09-16T10:23:54.208090+0000","last_clean":"2024-09-16T10:23:54.208090+0000","last_became_active":"2024-09-16T10:23:50.340205+0000","last_became_peered":"2024-09-16T10:23:50.340205+0000","last_unstale":"2024-09-16T10:23:54.208090+0000","last_undegraded":"2024-09-16T10:23:54.208090+0000","last_fullsized":"2024-09-16T10:23:54.208090+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16986595,"num_objects":4148,"num_object_clones":0,"num_object_copies":8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4148,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":16589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4148}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'4335","reported_seq":4448,"reported_epoch":39,"state":"active+clean+remapped+scrubbing","last_fresh":"2024-09-16T10:23:54.222237+0000","last_change":"2024-09-16T10:23:53.440714+0000","last_active":"2024-09-16T10:23:54.222237+0000","last_peered":"2024-09-16T10:23:54.222237+0000","last_clean":"2024-09-16T10:23:54.222237+0000","last_became_active":"2024-09-16T10:23:50.333769+0000","last_became_peered":"2024-09-16T10:23:50.333769+0000","last_unstale":"2024-09-16T10:23:54.222237+0000","last_undegraded":"2024-09-16T10:23:54.222237+0000","last_fullsized":"2024-09-16T10:23:54.222237+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":1899,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17724841,"num_objects":4330,"num_object_clones":0,"num_object_copies":8660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4330,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":17312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4330}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T10:23:52.344562+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:52.344562+0000","last_peered":"2024-09-16T10:23:52.344562+0000","last_clean":"2024-09-16T10:23:52.344562+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:52.344562+0000","last_undegraded":"2024-09-16T10:23:52.344562+0000","last_fullsized":"2024-09-16T10:23:52.344562+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'4165","reported_seq":4232,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.193029+0000","last_change":"2024-09-16T10:23:50.341146+0000","last_active":"2024-09-16T10:23:54.193029+0000","last_peered":"2024-09-16T10:23:54.193029+0000","last_clean":"2024-09-16T10:23:54.193029+0000","last_became_active":"2024-09-16T10:23:50.335911+0000","last_became_peered":"2024-09-16T10:23:50.335911+0000","last_unstale":"2024-09-16T10:23:54.193029+0000","last_undegraded":"2024-09-16T10:23:54.193029+0000","last_fullsized":"2024-09-16T10:23:54.193029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4165,"log_dups_size":0,"ondisk_log_size":4165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044422,"num_objects":4163,"num_object_clones":0,"num_object_copies":8326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4163,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4163,"num_write_kb":16646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'4396","reported_seq":4474,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.205431+0000","last_change":"2024-09-16T10:23:50.380060+0000","last_active":"2024-09-16T10:23:54.205431+0000","last_peered":"2024-09-16T10:23:54.205431+0000","last_clean":"2024-09-16T10:23:54.205431+0000","last_became_active":"2024-09-16T10:23:50.356077+0000","last_became_peered":"2024-09-16T10:23:50.356077+0000","last_unstale":"2024-09-16T10:23:54.205431+0000","last_undegraded":"2024-09-16T10:23:54.205431+0000","last_fullsized":"2024-09-16T10:23:54.205431+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4396,"log_dups_size":0,"ondisk_log_size":4396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17958796,"num_objects":4388,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4388,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":17543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4388}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":279733209,"num_objects":68311,"num_object_clones":0,"num_object_copies":136622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34235,"num_objects_unfound":0,"num_objects_dirty":68311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50135,"num_write_kb":200425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":560754688,"data_stored":560664363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68374,"ondisk_log_size":68374,"up":24,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":15,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018897,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422000,"kb_used_data":273892,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93949840,"statfs":{"total":96636764160,"available":96204636160,"internally_reserved":0,"allocated":280465408,"data_stored":280232085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":151645872},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.464}]}]},{"osd":1,"up_from":40,"seq":171798691842,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50808,"kb_used_data":1116,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94321032,"statfs":{"total":96636764160,"available":96584736768,"internally_reserved":0,"allocated":1142784,"data_stored":820059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":50851498},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9630000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9670000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215125,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406492,"kb_used_data":270736,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":93965348,"statfs":{"total":96636764160,"available":96220516352,"internally_reserved":0,"allocated":277233664,"data_stored":277008686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":138997416},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,11],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279543808,"data_stored":279540195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1085440,"data_stored":1002341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280125440,"data_stored":280121827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:58.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:23:59.507 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:23:59.510 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:23:59.524 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 47 20 69869 69849 5.80407 2.22656 0.0244979 0.0134529 2024-09-16T10:23:59.535 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-16T10:23:58.572366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":279733228,"num_objects":68313,"num_object_clones":0,"num_object_copies":136626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34236,"num_objects_unfound":0,"num_objects_dirty":68313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50137,"num_write_kb":200427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":4598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68377,"ondisk_log_size":68377,"up":39,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":879300,"kb_used_data":545744,"kb_used_omap":12,"kb_used_meta":333491,"kb_avail":282236220,"statfs":{"total":289910292480,"available":289009889280,"internally_reserved":0,"allocated":558841856,"data_stored":558060830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":341494786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,10,5,4,1,0,0,11],"upper_bound":2048},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17317888,"num_objects":4228,"num_object_clones":0,"num_object_copies":8456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34236,"num_objects_unfound":0,"num_objects_dirty":4228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4228,"num_write_kb":16912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003997"},"pg_stats":[{"pgid":"2.f","version":"39'4183","reported_seq":4239,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.162262+0000","last_change":"2024-09-16T10:23:50.351182+0000","last_active":"2024-09-16T10:23:54.162262+0000","last_peered":"2024-09-16T10:23:54.162262+0000","last_clean":"2024-09-16T10:23:54.162262+0000","last_became_active":"2024-09-16T10:23:50.335726+0000","last_became_peered":"2024-09-16T10:23:50.335726+0000","last_unstale":"2024-09-16T10:23:54.162262+0000","last_undegraded":"2024-09-16T10:23:54.162262+0000","last_fullsized":"2024-09-16T10:23:54.162262+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:11:49.636034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17133568,"num_objects":4183,"num_object_clones":0,"num_object_copies":8366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4183,"num_objects_unfound":0,"num_objects_dirty":4183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":16732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4183}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"40'4278","reported_seq":4430,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T10:23:54.396025+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:23:54.396025+0000","last_peered":"2024-09-16T10:23:54.396025+0000","last_clean":"2024-09-16T10:23:54.396025+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:23:54.396025+0000","last_undegraded":"2024-09-16T10:23:54.396025+0000","last_fullsized":"2024-09-16T10:23:54.396025+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498112,"num_objects":4272,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":1964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"39'4339","reported_seq":4512,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.396512+0000","last_change":"2024-09-16T10:23:54.396668+0000","last_active":"2024-09-16T10:23:54.396512+0000","last_peered":"2024-09-16T10:23:54.392083+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:50.333009+0000","last_became_peered":"2024-09-16T10:23:50.333009+0000","last_unstale":"2024-09-16T10:23:54.396512+0000","last_undegraded":"2024-09-16T10:23:54.396512+0000","last_fullsized":"2024-09-16T10:23:54.396512+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4339,"log_dups_size":0,"ondisk_log_size":4339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:13:59.630290+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17756160,"num_objects":4335,"num_object_clones":0,"num_object_copies":8670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":396,"num_write_kb":1584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"40'4310","reported_seq":4361,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T10:23:54.399498+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:23:54.399498+0000","last_peered":"2024-09-16T10:23:54.399498+0000","last_clean":"2024-09-16T10:23:54.399498+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:23:54.399498+0000","last_undegraded":"2024-09-16T10:23:54.399498+0000","last_fullsized":"2024-09-16T10:23:54.399498+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17653760,"num_objects":4310,"num_object_clones":0,"num_object_copies":8620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":17240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"39'4262","reported_seq":4412,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T10:23:54.204475+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:23:54.204475+0000","last_peered":"2024-09-16T10:23:54.204475+0000","last_clean":"2024-09-16T10:23:54.204475+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:54.204475+0000","last_undegraded":"2024-09-16T10:23:54.204475+0000","last_fullsized":"2024-09-16T10:23:54.204475+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:25:31.412867+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17428480,"num_objects":4255,"num_object_clones":0,"num_object_copies":8510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":904,"num_write_kb":3616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"39'4213","reported_seq":4362,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.185461+0000","last_change":"2024-09-16T10:23:50.339479+0000","last_active":"2024-09-16T10:23:54.185461+0000","last_peered":"2024-09-16T10:23:54.185461+0000","last_clean":"2024-09-16T10:23:54.185461+0000","last_became_active":"2024-09-16T10:23:50.334121+0000","last_became_peered":"2024-09-16T10:23:50.334121+0000","last_unstale":"2024-09-16T10:23:54.185461+0000","last_undegraded":"2024-09-16T10:23:54.185461+0000","last_fullsized":"2024-09-16T10:23:54.185461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4213,"log_dups_size":0,"ondisk_log_size":4213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:47:13.266606+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":4208,"num_object_clones":0,"num_object_copies":8416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4208,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":838,"num_write_kb":3352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4208}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"39'4401","reported_seq":4560,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.179578+0000","last_change":"2024-09-16T10:23:50.344370+0000","last_active":"2024-09-16T10:23:54.179578+0000","last_peered":"2024-09-16T10:23:54.179578+0000","last_clean":"2024-09-16T10:23:54.179578+0000","last_became_active":"2024-09-16T10:23:50.334013+0000","last_became_peered":"2024-09-16T10:23:50.334013+0000","last_unstale":"2024-09-16T10:23:54.179578+0000","last_undegraded":"2024-09-16T10:23:54.179578+0000","last_fullsized":"2024-09-16T10:23:54.179578+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:12:22.668041+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18001920,"num_objects":4395,"num_object_clones":0,"num_object_copies":8790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4395,"num_objects_unfound":0,"num_objects_dirty":4395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":643,"num_write_kb":2572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"39'4233","reported_seq":4376,"reported_epoch":39,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:23:54.193511+0000","last_change":"2024-09-16T10:23:52.471903+0000","last_active":"2024-09-16T10:23:54.193511+0000","last_peered":"2024-09-16T10:23:54.193511+0000","last_clean":"2024-09-16T10:23:54.193511+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:54.193511+0000","last_undegraded":"2024-09-16T10:23:54.193511+0000","last_fullsized":"2024-09-16T10:23:54.193511+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":3324,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17330659,"num_objects":4232,"num_object_clones":0,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4232,"num_write_kb":16925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"39'4204","reported_seq":4287,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.374131+0000","last_change":"2024-09-16T10:23:54.374253+0000","last_active":"2024-09-16T10:23:54.374131+0000","last_peered":"2024-09-16T10:23:54.345012+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:50.338673+0000","last_became_peered":"2024-09-16T10:23:50.338673+0000","last_unstale":"2024-09-16T10:23:54.374131+0000","last_undegraded":"2024-09-16T10:23:54.374131+0000","last_fullsized":"2024-09-16T10:23:54.374131+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17191878,"num_objects":4199,"num_object_clones":0,"num_object_copies":8398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":16792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":51,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T10:23:52.345559+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:52.345559+0000","last_peered":"2024-09-16T10:23:52.345559+0000","last_clean":"2024-09-16T10:23:52.345559+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:52.345559+0000","last_undegraded":"2024-09-16T10:23:52.345559+0000","last_fullsized":"2024-09-16T10:23:52.345559+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:55:48.971314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"39'4426","reported_seq":4490,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.150603+0000","last_change":"2024-09-16T10:23:50.346372+0000","last_active":"2024-09-16T10:23:54.150603+0000","last_peered":"2024-09-16T10:23:54.150603+0000","last_clean":"2024-09-16T10:23:54.150603+0000","last_became_active":"2024-09-16T10:23:50.334044+0000","last_became_peered":"2024-09-16T10:23:50.334044+0000","last_unstale":"2024-09-16T10:23:54.150603+0000","last_undegraded":"2024-09-16T10:23:54.150603+0000","last_fullsized":"2024-09-16T10:23:54.150603+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4426,"log_dups_size":0,"ondisk_log_size":4426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18097094,"num_objects":4420,"num_object_clones":0,"num_object_copies":8840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4420,"num_objects_unfound":0,"num_objects_dirty":4420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4424,"num_write_kb":17676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":72,"reported_epoch":38,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:50.355920+0000","last_change":"2024-09-16T10:23:50.342443+0000","last_active":"2024-09-16T10:23:50.355920+0000","last_peered":"2024-09-16T10:23:50.355920+0000","last_clean":"2024-09-16T10:23:50.355920+0000","last_became_active":"2024-09-16T10:23:50.339762+0000","last_became_peered":"2024-09-16T10:23:50.339762+0000","last_unstale":"2024-09-16T10:23:50.355920+0000","last_undegraded":"2024-09-16T10:23:50.355920+0000","last_fullsized":"2024-09-16T10:23:50.355920+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:15:21.709026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"39'4282","reported_seq":4375,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.391281+0000","last_change":"2024-09-16T10:23:54.391428+0000","last_active":"2024-09-16T10:23:54.391281+0000","last_peered":"2024-09-16T10:23:54.351639+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:50.352466+0000","last_became_peered":"2024-09-16T10:23:50.352466+0000","last_unstale":"2024-09-16T10:23:54.391281+0000","last_undegraded":"2024-09-16T10:23:54.391281+0000","last_fullsized":"2024-09-16T10:23:54.391281+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17519558,"num_objects":4279,"num_object_clones":0,"num_object_copies":8558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":17111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"39'4198","reported_seq":4272,"reported_epoch":40,"state":"remapped+peering","last_fresh":"2024-09-16T10:23:54.375299+0000","last_change":"2024-09-16T10:23:54.375453+0000","last_active":"2024-09-16T10:23:54.375299+0000","last_peered":"2024-09-16T10:23:54.357802+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:50.336831+0000","last_became_peered":"2024-09-16T10:23:50.336831+0000","last_unstale":"2024-09-16T10:23:54.375299+0000","last_undegraded":"2024-09-16T10:23:54.375299+0000","last_fullsized":"2024-09-16T10:23:54.375299+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4198,"log_dups_size":0,"ondisk_log_size":4198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17171398,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":16771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"39'4149","reported_seq":4214,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.208090+0000","last_change":"2024-09-16T10:23:50.350102+0000","last_active":"2024-09-16T10:23:54.208090+0000","last_peered":"2024-09-16T10:23:54.208090+0000","last_clean":"2024-09-16T10:23:54.208090+0000","last_became_active":"2024-09-16T10:23:50.340205+0000","last_became_peered":"2024-09-16T10:23:50.340205+0000","last_unstale":"2024-09-16T10:23:54.208090+0000","last_undegraded":"2024-09-16T10:23:54.208090+0000","last_fullsized":"2024-09-16T10:23:54.208090+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16986595,"num_objects":4148,"num_object_clones":0,"num_object_copies":8296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4148,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":16589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4148}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"39'4335","reported_seq":4448,"reported_epoch":39,"state":"active+clean+remapped+scrubbing","last_fresh":"2024-09-16T10:23:54.222237+0000","last_change":"2024-09-16T10:23:53.440714+0000","last_active":"2024-09-16T10:23:54.222237+0000","last_peered":"2024-09-16T10:23:54.222237+0000","last_clean":"2024-09-16T10:23:54.222237+0000","last_became_active":"2024-09-16T10:23:50.333769+0000","last_became_peered":"2024-09-16T10:23:50.333769+0000","last_unstale":"2024-09-16T10:23:54.222237+0000","last_undegraded":"2024-09-16T10:23:54.222237+0000","last_fullsized":"2024-09-16T10:23:54.222237+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":1899,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17724841,"num_objects":4330,"num_object_clones":0,"num_object_copies":8660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4330,"num_objects_unfound":0,"num_objects_dirty":4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4332,"num_write_kb":17312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4330}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":51,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T10:23:52.344562+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:52.344562+0000","last_peered":"2024-09-16T10:23:52.344562+0000","last_clean":"2024-09-16T10:23:52.344562+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:52.344562+0000","last_undegraded":"2024-09-16T10:23:52.344562+0000","last_fullsized":"2024-09-16T10:23:52.344562+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:03:37.053930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"39'4165","reported_seq":4232,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.193029+0000","last_change":"2024-09-16T10:23:50.341146+0000","last_active":"2024-09-16T10:23:54.193029+0000","last_peered":"2024-09-16T10:23:54.193029+0000","last_clean":"2024-09-16T10:23:54.193029+0000","last_became_active":"2024-09-16T10:23:50.335911+0000","last_became_peered":"2024-09-16T10:23:50.335911+0000","last_unstale":"2024-09-16T10:23:54.193029+0000","last_undegraded":"2024-09-16T10:23:54.193029+0000","last_fullsized":"2024-09-16T10:23:54.193029+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4165,"log_dups_size":0,"ondisk_log_size":4165,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17044422,"num_objects":4163,"num_object_clones":0,"num_object_copies":8326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4163,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4163,"num_write_kb":16646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"39'4396","reported_seq":4474,"reported_epoch":39,"state":"active+clean+remapped","last_fresh":"2024-09-16T10:23:54.205431+0000","last_change":"2024-09-16T10:23:50.380060+0000","last_active":"2024-09-16T10:23:54.205431+0000","last_peered":"2024-09-16T10:23:54.205431+0000","last_clean":"2024-09-16T10:23:54.205431+0000","last_became_active":"2024-09-16T10:23:50.356077+0000","last_became_peered":"2024-09-16T10:23:50.356077+0000","last_unstale":"2024-09-16T10:23:54.205431+0000","last_undegraded":"2024-09-16T10:23:54.205431+0000","last_fullsized":"2024-09-16T10:23:54.205431+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4396,"log_dups_size":0,"ondisk_log_size":4396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17958796,"num_objects":4388,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4388,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":17543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":4388}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":279733209,"num_objects":68311,"num_object_clones":0,"num_object_copies":136622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34235,"num_objects_unfound":0,"num_objects_dirty":68311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50135,"num_write_kb":200425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":560754688,"data_stored":560664363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68374,"ondisk_log_size":68374,"up":24,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":15,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018897,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422000,"kb_used_data":273892,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93949840,"statfs":{"total":96636764160,"available":96204636160,"internally_reserved":0,"allocated":280465408,"data_stored":280232085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":151645872},"hb_peers":[0],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.464}]}]},{"osd":1,"up_from":40,"seq":171798691842,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50808,"kb_used_data":1116,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94321032,"statfs":{"total":96636764160,"available":96584736768,"internally_reserved":0,"allocated":1142784,"data_stored":820059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":50851498},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9630000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9670000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215125,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406492,"kb_used_data":270736,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":93965348,"statfs":{"total":96636764160,"available":96220516352,"internally_reserved":0,"allocated":277233664,"data_stored":277008686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":138997416},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,11],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":279543808,"data_stored":279540195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1085440,"data_stored":1002341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280125440,"data_stored":280121827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:23:59.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:23:59.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "39'4183", 'reported_seq': 4239, 'reported_epoch': 39, 'state': 'active+clean+remapped', 'last_fresh': '2024-09-16T10:23:54.162262+0000', 'last_change': '2024-09-16T10:23:50.351182+0000', 'last_active': '2024-09-16T10:23:54.162262+0000', 'last_peered': '2024-09-16T10:23:54.162262+0000', 'last_clean': '2024-09-16T10:23:54.162262+0000', 'last_became_active': '2024-09-16T10:23:50.335726+0000', 'last_became_peered': '2024-09-16T10:23:50.335726+0000', 'last_unstale': '2024-09-16T10:23:54.162262+0000', 'last_undegraded': '2024-09-16T10:23:54.162262+0000', 'last_fullsized': '2024-09-16T10:23:54.162262+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4183, 'log_dups_size': 0, 'ondisk_log_size': 4183, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T12:11:49.636034+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17133568, 'num_objects': 4183, 'num_object_clones': 0, 'num_object_copies': 8366, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4183, 'num_objects_unfound': 0, 'num_objects_dirty': 4183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4183, 'num_write_kb': 16732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 4183}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:23:59.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:23:59.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "39'4339", 'reported_seq': 4512, 'reported_epoch': 40, 'state': 'remapped+peering', 'last_fresh': '2024-09-16T10:23:54.396512+0000', 'last_change': '2024-09-16T10:23:54.396668+0000', 'last_active': '2024-09-16T10:23:54.396512+0000', 'last_peered': '2024-09-16T10:23:54.392083+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:50.333009+0000', 'last_became_peered': '2024-09-16T10:23:50.333009+0000', 'last_unstale': '2024-09-16T10:23:54.396512+0000', 'last_undegraded': '2024-09-16T10:23:54.396512+0000', 'last_fullsized': '2024-09-16T10:23:54.396512+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 4339, 'log_dups_size': 0, 'ondisk_log_size': 4339, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:13:59.630290+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17756160, 'num_objects': 4335, 'num_object_clones': 0, 'num_object_copies': 8670, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 396, 'num_write_kb': 1584, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:23:59.536 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:23:59.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "39'4213", 'reported_seq': 4362, 'reported_epoch': 39, 'state': 'active+clean+remapped', 'last_fresh': '2024-09-16T10:23:54.185461+0000', 'last_change': '2024-09-16T10:23:50.339479+0000', 'last_active': '2024-09-16T10:23:54.185461+0000', 'last_peered': '2024-09-16T10:23:54.185461+0000', 'last_clean': '2024-09-16T10:23:54.185461+0000', 'last_became_active': '2024-09-16T10:23:50.334121+0000', 'last_became_peered': '2024-09-16T10:23:50.334121+0000', 'last_unstale': '2024-09-16T10:23:54.185461+0000', 'last_undegraded': '2024-09-16T10:23:54.185461+0000', 'last_fullsized': '2024-09-16T10:23:54.185461+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 4213, 'log_dups_size': 0, 'ondisk_log_size': 4213, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:47:13.266606+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17235968, 'num_objects': 4208, 'num_object_clones': 0, 'num_object_copies': 8416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4208, 'num_objects_unfound': 0, 'num_objects_dirty': 4208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 838, 'num_write_kb': 3352, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 4208}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:23:59.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:23:59.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "39'4401", 'reported_seq': 4560, 'reported_epoch': 39, 'state': 'active+clean+remapped', 'last_fresh': '2024-09-16T10:23:54.179578+0000', 'last_change': '2024-09-16T10:23:50.344370+0000', 'last_active': '2024-09-16T10:23:54.179578+0000', 'last_peered': '2024-09-16T10:23:54.179578+0000', 'last_clean': '2024-09-16T10:23:54.179578+0000', 'last_became_active': '2024-09-16T10:23:50.334013+0000', 'last_became_peered': '2024-09-16T10:23:50.334013+0000', 'last_unstale': '2024-09-16T10:23:54.179578+0000', 'last_undegraded': '2024-09-16T10:23:54.179578+0000', 'last_fullsized': '2024-09-16T10:23:54.179578+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 4401, 'log_dups_size': 0, 'ondisk_log_size': 4401, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:12:22.668041+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18001920, 'num_objects': 4395, 'num_object_clones': 0, 'num_object_copies': 8790, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4395, 'num_objects_unfound': 0, 'num_objects_dirty': 4395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 643, 'num_write_kb': 2572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:23:59.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.8 is not active+clean 2024-09-16T10:23:59.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.8', 'version': "39'4233", 'reported_seq': 4376, 'reported_epoch': 39, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:23:54.193511+0000', 'last_change': '2024-09-16T10:23:52.471903+0000', 'last_active': '2024-09-16T10:23:54.193511+0000', 'last_peered': '2024-09-16T10:23:54.193511+0000', 'last_clean': '2024-09-16T10:23:54.193511+0000', 'last_became_active': '2024-09-16T10:23:01.015829+0000', 'last_became_peered': '2024-09-16T10:23:01.015829+0000', 'last_unstale': '2024-09-16T10:23:54.193511+0000', 'last_undegraded': '2024-09-16T10:23:54.193511+0000', 'last_fullsized': '2024-09-16T10:23:54.193511+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 35, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 3324, 'log_size': 4233, 'log_dups_size': 0, 'ondisk_log_size': 4233, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 1s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17330659, 'num_objects': 4232, 'num_object_clones': 0, 'num_object_copies': 8464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4232, 'num_write_kb': 16925, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:23:59.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:23:59.537 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "39'4204", 'reported_seq': 4287, 'reported_epoch': 40, 'state': 'remapped+peering', 'last_fresh': '2024-09-16T10:23:54.374131+0000', 'last_change': '2024-09-16T10:23:54.374253+0000', 'last_active': '2024-09-16T10:23:54.374131+0000', 'last_peered': '2024-09-16T10:23:54.345012+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:50.338673+0000', 'last_became_peered': '2024-09-16T10:23:50.338673+0000', 'last_unstale': '2024-09-16T10:23:54.374131+0000', 'last_undegraded': '2024-09-16T10:23:54.374131+0000', 'last_fullsized': '2024-09-16T10:23:54.374131+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4204, 'log_dups_size': 0, 'ondisk_log_size': 4204, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17191878, 'num_objects': 4199, 'num_object_clones': 0, 'num_object_copies': 8398, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4199, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4202, 'num_write_kb': 16792, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "39'4426", 'reported_seq': 4490, 'reported_epoch': 39, 'state': 'active+clean+remapped', 'last_fresh': '2024-09-16T10:23:54.150603+0000', 'last_change': '2024-09-16T10:23:50.346372+0000', 'last_active': '2024-09-16T10:23:54.150603+0000', 'last_peered': '2024-09-16T10:23:54.150603+0000', 'last_clean': '2024-09-16T10:23:54.150603+0000', 'last_became_active': '2024-09-16T10:23:50.334044+0000', 'last_became_peered': '2024-09-16T10:23:50.334044+0000', 'last_unstale': '2024-09-16T10:23:54.150603+0000', 'last_undegraded': '2024-09-16T10:23:54.150603+0000', 'last_fullsized': '2024-09-16T10:23:54.150603+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4426, 'log_dups_size': 0, 'ondisk_log_size': 4426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18097094, 'num_objects': 4420, 'num_object_clones': 0, 'num_object_copies': 8840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4420, 'num_objects_unfound': 0, 'num_objects_dirty': 4420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4424, 'num_write_kb': 17676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 4420}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 1.2 is not active+clean 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '1.2', 'version': "13'2", 'reported_seq': 72, 'reported_epoch': 38, 'state': 'active+clean+remapped', 'last_fresh': '2024-09-16T10:23:50.355920+0000', 'last_change': '2024-09-16T10:23:50.342443+0000', 'last_active': '2024-09-16T10:23:50.355920+0000', 'last_peered': '2024-09-16T10:23:50.355920+0000', 'last_clean': '2024-09-16T10:23:50.355920+0000', 'last_became_active': '2024-09-16T10:23:50.339762+0000', 'last_became_peered': '2024-09-16T10:23:50.339762+0000', 'last_unstale': '2024-09-16T10:23:50.355920+0000', 'last_undegraded': '2024-09-16T10:23:50.355920+0000', 'last_fullsized': '2024-09-16T10:23:50.355920+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:01.610622+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:01.610622+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:01.610622+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T12:15:21.709026+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.002132092, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "39'4282", 'reported_seq': 4375, 'reported_epoch': 40, 'state': 'remapped+peering', 'last_fresh': '2024-09-16T10:23:54.391281+0000', 'last_change': '2024-09-16T10:23:54.391428+0000', 'last_active': '2024-09-16T10:23:54.391281+0000', 'last_peered': '2024-09-16T10:23:54.351639+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:50.352466+0000', 'last_became_peered': '2024-09-16T10:23:50.352466+0000', 'last_unstale': '2024-09-16T10:23:54.391281+0000', 'last_undegraded': '2024-09-16T10:23:54.391281+0000', 'last_fullsized': '2024-09-16T10:23:54.391281+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 4282, 'log_dups_size': 0, 'ondisk_log_size': 4282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17519558, 'num_objects': 4279, 'num_object_clones': 0, 'num_object_copies': 8558, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4280, 'num_write_kb': 17111, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "39'4198", 'reported_seq': 4272, 'reported_epoch': 40, 'state': 'remapped+peering', 'last_fresh': '2024-09-16T10:23:54.375299+0000', 'last_change': '2024-09-16T10:23:54.375453+0000', 'last_active': '2024-09-16T10:23:54.375299+0000', 'last_peered': '2024-09-16T10:23:54.357802+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:50.336831+0000', 'last_became_peered': '2024-09-16T10:23:50.336831+0000', 'last_unstale': '2024-09-16T10:23:54.375299+0000', 'last_undegraded': '2024-09-16T10:23:54.375299+0000', 'last_fullsized': '2024-09-16T10:23:54.375299+0000', 'mapping_epoch': 40, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4198, 'log_dups_size': 0, 'ondisk_log_size': 4198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17171398, 'num_objects': 4194, 'num_object_clones': 0, 'num_object_copies': 8388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4196, 'num_write_kb': 16771, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "39'4149", 'reported_seq': 4214, 'reported_epoch': 39, 'state': 'active+clean+remapped', 'last_fresh': '2024-09-16T10:23:54.208090+0000', 'last_change': '2024-09-16T10:23:50.350102+0000', 'last_active': '2024-09-16T10:23:54.208090+0000', 'last_peered': '2024-09-16T10:23:54.208090+0000', 'last_clean': '2024-09-16T10:23:54.208090+0000', 'last_became_active': '2024-09-16T10:23:50.340205+0000', 'last_became_peered': '2024-09-16T10:23:50.340205+0000', 'last_unstale': '2024-09-16T10:23:54.208090+0000', 'last_undegraded': '2024-09-16T10:23:54.208090+0000', 'last_fullsized': '2024-09-16T10:23:54.208090+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4149, 'log_dups_size': 0, 'ondisk_log_size': 4149, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16986595, 'num_objects': 4148, 'num_object_clones': 0, 'num_object_copies': 8296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4148, 'num_objects_unfound': 0, 'num_objects_dirty': 4148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4148, 'num_write_kb': 16589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 4148}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:23:59.538 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:23:59.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "39'4335", 'reported_seq': 4448, 'reported_epoch': 39, 'state': 'active+clean+remapped+scrubbing', 'last_fresh': '2024-09-16T10:23:54.222237+0000', 'last_change': '2024-09-16T10:23:53.440714+0000', 'last_active': '2024-09-16T10:23:54.222237+0000', 'last_peered': '2024-09-16T10:23:54.222237+0000', 'last_clean': '2024-09-16T10:23:54.222237+0000', 'last_became_active': '2024-09-16T10:23:50.333769+0000', 'last_became_peered': '2024-09-16T10:23:50.333769+0000', 'last_unstale': '2024-09-16T10:23:54.222237+0000', 'last_undegraded': '2024-09-16T10:23:54.222237+0000', 'last_fullsized': '2024-09-16T10:23:54.222237+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 1899, 'log_size': 4335, 'log_dups_size': 0, 'ondisk_log_size': 4335, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 0s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17724841, 'num_objects': 4330, 'num_object_clones': 0, 'num_object_copies': 8660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4330, 'num_objects_unfound': 0, 'num_objects_dirty': 4330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4332, 'num_write_kb': 17312, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 4330}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:23:59.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:23:59.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "39'4165", 'reported_seq': 4232, 'reported_epoch': 39, 'state': 'active+clean+remapped', 'last_fresh': '2024-09-16T10:23:54.193029+0000', 'last_change': '2024-09-16T10:23:50.341146+0000', 'last_active': '2024-09-16T10:23:54.193029+0000', 'last_peered': '2024-09-16T10:23:54.193029+0000', 'last_clean': '2024-09-16T10:23:54.193029+0000', 'last_became_active': '2024-09-16T10:23:50.335911+0000', 'last_became_peered': '2024-09-16T10:23:50.335911+0000', 'last_unstale': '2024-09-16T10:23:54.193029+0000', 'last_undegraded': '2024-09-16T10:23:54.193029+0000', 'last_fullsized': '2024-09-16T10:23:54.193029+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4165, 'log_dups_size': 0, 'ondisk_log_size': 4165, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17044422, 'num_objects': 4163, 'num_object_clones': 0, 'num_object_copies': 8326, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4163, 'num_objects_unfound': 0, 'num_objects_dirty': 4163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4163, 'num_write_kb': 16646, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 4163}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:23:59.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:23:59.539 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "39'4396", 'reported_seq': 4474, 'reported_epoch': 39, 'state': 'active+clean+remapped', 'last_fresh': '2024-09-16T10:23:54.205431+0000', 'last_change': '2024-09-16T10:23:50.380060+0000', 'last_active': '2024-09-16T10:23:54.205431+0000', 'last_peered': '2024-09-16T10:23:54.205431+0000', 'last_clean': '2024-09-16T10:23:54.205431+0000', 'last_became_active': '2024-09-16T10:23:50.356077+0000', 'last_became_peered': '2024-09-16T10:23:50.356077+0000', 'last_unstale': '2024-09-16T10:23:54.205431+0000', 'last_undegraded': '2024-09-16T10:23:54.205431+0000', 'last_fullsized': '2024-09-16T10:23:54.205431+0000', 'mapping_epoch': 37, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4396, 'log_dups_size': 0, 'ondisk_log_size': 4396, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17958796, 'num_objects': 4388, 'num_object_clones': 0, 'num_object_copies': 8776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4388, 'num_objects_unfound': 0, 'num_objects_dirty': 4388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4393, 'num_write_kb': 17543, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0, 2], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0,2', 'objects': 4388}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:23:59.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:00.220 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:00.220 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":129,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+clean+remapped","count":8},{"state_name":"remapped+peering","count":4},{"state_name":"active+clean+remapped+scrubbing","count":1},{"state_name":"active+clean+scrubbing","count":1}],"num_pgs":24,"num_pools":2,"num_objects":68313,"data_bytes":279733228,"bytes_used":900403200,"bytes_avail":289009889280,"bytes_total":289910292480,"inactive_pgs_ratio":0.1666666716337204,"misplaced_objects":34236,"misplaced_total":136626,"misplaced_ratio":0.25058188046199115,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":682,"recovering_keys_per_sec":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"write_bytes_sec":2884393,"read_op_per_sec":0,"write_op_per_sec":704},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T10:23:56.572050+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:24:00.243 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:00.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:00.524 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 48 20 70426 70406 5.72849 2.17578 0.0415674 0.0136305 2024-09-16T10:24:00.791 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:00.793 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:00.815 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2024-09-16T10:24:00.572769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286458377,"num_objects":69954,"num_object_clones":0,"num_object_copies":139908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":69954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47583,"num_write_kb":190214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":356854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70019,"ondisk_log_size":70019,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":880440,"kb_used_data":555084,"kb_used_omap":12,"kb_used_meta":325299,"kb_avail":282235080,"statfs":{"total":289910292480,"available":289008721920,"internally_reserved":0,"allocated":568406016,"data_stored":567600427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":333106178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,11,14,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13397533,"num_objects":3270,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":-30125,"num_objects_unfound":0,"num_objects_dirty":3270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-925,"num_write_kb":-3697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.210364"},"pg_stats":[{"pgid":"2.f","version":"42'4270","reported_seq":4335,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.218497+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:23:59.218497+0000","last_peered":"2024-09-16T10:23:59.218497+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:23:59.218497+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17489920,"num_objects":4270,"num_object_clones":0,"num_object_copies":8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4270,"num_write_kb":17080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":78},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'4394","reported_seq":4548,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:00.237868+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:00.237868+0000","last_peered":"2024-09-16T10:24:00.237868+0000","last_clean":"2024-09-16T10:24:00.237868+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:00.237868+0000","last_undegraded":"2024-09-16T10:24:00.237868+0000","last_fullsized":"2024-09-16T10:24:00.237868+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17973248,"num_objects":4388,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":607,"num_write_kb":2428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'4455","reported_seq":4637,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.295107+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:00.295107+0000","last_peered":"2024-09-16T10:24:00.295107+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:00.295107+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4455,"log_dups_size":0,"ondisk_log_size":4455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18231296,"num_objects":4451,"num_object_clones":0,"num_object_copies":8902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'4425","reported_seq":4478,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:00.276024+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:00.276024+0000","last_peered":"2024-09-16T10:24:00.276024+0000","last_clean":"2024-09-16T10:24:00.276024+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:00.276024+0000","last_undegraded":"2024-09-16T10:24:00.276024+0000","last_fullsized":"2024-09-16T10:24:00.276024+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18124800,"num_objects":4425,"num_object_clones":0,"num_object_copies":8850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4425,"num_write_kb":17700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'4338","reported_seq":4491,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:59.181848+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:23:59.181848+0000","last_peered":"2024-09-16T10:23:59.181848+0000","last_clean":"2024-09-16T10:23:59.181848+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:59.181848+0000","last_undegraded":"2024-09-16T10:23:59.181848+0000","last_fullsized":"2024-09-16T10:23:59.181848+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17739776,"num_objects":4331,"num_object_clones":0,"num_object_copies":8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":980,"num_write_kb":3920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'4303","reported_seq":4463,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.176984+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:23:59.176984+0000","last_peered":"2024-09-16T10:23:59.176984+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:23:59.176984+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17604608,"num_objects":4298,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":928,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":81},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'4494","reported_seq":4662,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.181989+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:23:59.181989+0000","last_peered":"2024-09-16T10:23:59.181989+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:23:59.181989+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18382848,"num_objects":4488,"num_object_clones":0,"num_object_copies":8976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":736,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":85},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'4327","reported_seq":4499,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:59.232051+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:23:59.232051+0000","last_peered":"2024-09-16T10:23:59.232051+0000","last_clean":"2024-09-16T10:23:59.232051+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:59.232051+0000","last_undegraded":"2024-09-16T10:23:59.232051+0000","last_fullsized":"2024-09-16T10:23:59.232051+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":4327,"log_dups_size":0,"ondisk_log_size":4327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17715200,"num_objects":4325,"num_object_clones":0,"num_object_copies":8650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130,"num_write_kb":520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'4332","reported_seq":4595,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:00.300303+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:00.300303+0000","last_peered":"2024-09-16T10:24:00.300303+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:00.300303+0000","last_undegraded":"2024-09-16T10:24:00.300303+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17716166,"num_objects":4327,"num_object_clones":0,"num_object_copies":8654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4330,"num_write_kb":17304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4111},{"shards":"2","objects":216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'4552","reported_seq":4625,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.244260+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:23:59.244260+0000","last_peered":"2024-09-16T10:23:59.244260+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:23:59.244260+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4552,"log_dups_size":0,"ondisk_log_size":4552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18613190,"num_objects":4546,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4550,"num_write_kb":18180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":112},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'4402","reported_seq":4504,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.192662+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:00.192662+0000","last_peered":"2024-09-16T10:24:00.192662+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:00.192662+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18011078,"num_objects":4399,"num_object_clones":0,"num_object_copies":8798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":17591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'4304","reported_seq":4387,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.242617+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:00.242617+0000","last_peered":"2024-09-16T10:24:00.242617+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:00.242617+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17605574,"num_objects":4300,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":17195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4247","reported_seq":4323,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.232884+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:23:59.232884+0000","last_peered":"2024-09-16T10:23:59.232884+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:23:59.232884+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17388003,"num_objects":4246,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4246,"num_write_kb":16981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":87},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'4433","reported_seq":4563,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.243824+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:23:59.243824+0000","last_peered":"2024-09-16T10:23:59.243824+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:23:59.243824+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18126249,"num_objects":4428,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4430,"num_write_kb":17704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":91},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'4250","reported_seq":4327,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.236371+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:23:59.236371+0000","last_peered":"2024-09-16T10:23:59.236371+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:23:59.236371+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17392582,"num_objects":4248,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4248,"num_write_kb":16986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":81},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'4490","reported_seq":4579,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.223227+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:23:59.223227+0000","last_peered":"2024-09-16T10:23:59.223227+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:23:59.223227+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18343820,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4487,"num_write_kb":17919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":87},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":286458358,"num_objects":69952,"num_object_clones":0,"num_object_copies":139904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":69952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47581,"num_write_kb":190212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":356835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569290752,"data_stored":569200427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70016,"ondisk_log_size":70016,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018898,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413192,"kb_used_data":277376,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93958648,"statfs":{"total":96636764160,"available":96213655552,"internally_reserved":0,"allocated":284033024,"data_stored":283789785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":139062960},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.712}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4369999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691842,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50808,"kb_used_data":1116,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94321032,"statfs":{"total":96636764160,"available":96584736768,"internally_reserved":0,"allocated":1142784,"data_stored":820059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":50851498},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.9630000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9670000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215126,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416440,"kb_used_data":276592,"kb_used_omap":4,"kb_used_meta":139835,"kb_avail":93955400,"statfs":{"total":96636764160,"available":96210329600,"internally_reserved":0,"allocated":283230208,"data_stored":282990583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":143191720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.272}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":8.5030000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284270592,"data_stored":284266979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1085440,"data_stored":1002341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283934720,"data_stored":283931107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:01.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 49 20 71040 71020 5.66053 2.39844 0.0233767 0.0137923 2024-09-16T10:24:02.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 50 20 71679 71659 5.59725 2.49609 0.0126546 0.0139492 2024-09-16T10:24:03.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 51 20 72309 72289 5.53575 2.46094 0.0792781 0.0141029 2024-09-16T10:24:03.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:04.393 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:04.394 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:04.415 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-16T10:24:02.573303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286458377,"num_objects":69954,"num_object_clones":0,"num_object_copies":139908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":69954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47583,"num_write_kb":190214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":356854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70019,"ondisk_log_size":70019,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893788,"kb_used_data":564356,"kb_used_omap":12,"kb_used_meta":329395,"kb_avail":282221732,"statfs":{"total":289910292480,"available":288995053568,"internally_reserved":0,"allocated":577900544,"data_stored":577094955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":337300482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,7,10,14,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13397533,"num_objects":3270,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":-30125,"num_objects_unfound":0,"num_objects_dirty":3270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-925,"num_write_kb":-3697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004542"},"pg_stats":[{"pgid":"2.f","version":"42'4270","reported_seq":4335,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.218497+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:23:59.218497+0000","last_peered":"2024-09-16T10:23:59.218497+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:23:59.218497+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17489920,"num_objects":4270,"num_object_clones":0,"num_object_copies":8540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4270,"num_write_kb":17080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":78},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'4394","reported_seq":4548,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:00.237868+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:00.237868+0000","last_peered":"2024-09-16T10:24:00.237868+0000","last_clean":"2024-09-16T10:24:00.237868+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:00.237868+0000","last_undegraded":"2024-09-16T10:24:00.237868+0000","last_fullsized":"2024-09-16T10:24:00.237868+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17973248,"num_objects":4388,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":607,"num_write_kb":2428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'4455","reported_seq":4637,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.295107+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:00.295107+0000","last_peered":"2024-09-16T10:24:00.295107+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:00.295107+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4455,"log_dups_size":0,"ondisk_log_size":4455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18231296,"num_objects":4451,"num_object_clones":0,"num_object_copies":8902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'4425","reported_seq":4478,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:00.276024+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:00.276024+0000","last_peered":"2024-09-16T10:24:00.276024+0000","last_clean":"2024-09-16T10:24:00.276024+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:00.276024+0000","last_undegraded":"2024-09-16T10:24:00.276024+0000","last_fullsized":"2024-09-16T10:24:00.276024+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18124800,"num_objects":4425,"num_object_clones":0,"num_object_copies":8850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4425,"num_write_kb":17700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'4338","reported_seq":4491,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:59.181848+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:23:59.181848+0000","last_peered":"2024-09-16T10:23:59.181848+0000","last_clean":"2024-09-16T10:23:59.181848+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:23:59.181848+0000","last_undegraded":"2024-09-16T10:23:59.181848+0000","last_fullsized":"2024-09-16T10:23:59.181848+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4338,"log_dups_size":0,"ondisk_log_size":4338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17739776,"num_objects":4331,"num_object_clones":0,"num_object_copies":8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":980,"num_write_kb":3920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'4303","reported_seq":4463,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.176984+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:23:59.176984+0000","last_peered":"2024-09-16T10:23:59.176984+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:23:59.176984+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17604608,"num_objects":4298,"num_object_clones":0,"num_object_copies":8596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":928,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":81},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'4494","reported_seq":4662,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.181989+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:23:59.181989+0000","last_peered":"2024-09-16T10:23:59.181989+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:23:59.181989+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18382848,"num_objects":4488,"num_object_clones":0,"num_object_copies":8976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":736,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":85},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'4327","reported_seq":4499,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:59.232051+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:23:59.232051+0000","last_peered":"2024-09-16T10:23:59.232051+0000","last_clean":"2024-09-16T10:23:59.232051+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:23:59.232051+0000","last_undegraded":"2024-09-16T10:23:59.232051+0000","last_fullsized":"2024-09-16T10:23:59.232051+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":4327,"log_dups_size":0,"ondisk_log_size":4327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17715200,"num_objects":4325,"num_object_clones":0,"num_object_copies":8650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130,"num_write_kb":520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'4332","reported_seq":4595,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:00.300303+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:00.300303+0000","last_peered":"2024-09-16T10:24:00.300303+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:00.300303+0000","last_undegraded":"2024-09-16T10:24:00.300303+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17716166,"num_objects":4327,"num_object_clones":0,"num_object_copies":8654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4330,"num_write_kb":17304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4111},{"shards":"2","objects":216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'4552","reported_seq":4625,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.244260+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:23:59.244260+0000","last_peered":"2024-09-16T10:23:59.244260+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:23:59.244260+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4552,"log_dups_size":0,"ondisk_log_size":4552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18613190,"num_objects":4546,"num_object_clones":0,"num_object_copies":9092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4550,"num_write_kb":18180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":112},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'4402","reported_seq":4504,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.192662+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:00.192662+0000","last_peered":"2024-09-16T10:24:00.192662+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:00.192662+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18011078,"num_objects":4399,"num_object_clones":0,"num_object_copies":8798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":17591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'4304","reported_seq":4387,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.242617+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:00.242617+0000","last_peered":"2024-09-16T10:24:00.242617+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:00.242617+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17605574,"num_objects":4300,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":17195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4247","reported_seq":4323,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.232884+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:23:59.232884+0000","last_peered":"2024-09-16T10:23:59.232884+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:23:59.232884+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17388003,"num_objects":4246,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4246,"num_write_kb":16981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":87},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'4433","reported_seq":4563,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.243824+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:23:59.243824+0000","last_peered":"2024-09-16T10:23:59.243824+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:23:59.243824+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18126249,"num_objects":4428,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4430,"num_write_kb":17704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":91},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'4250","reported_seq":4327,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.236371+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:23:59.236371+0000","last_peered":"2024-09-16T10:23:59.236371+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:23:59.236371+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17392582,"num_objects":4248,"num_object_clones":0,"num_object_copies":8496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4248,"num_write_kb":16986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":81},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'4490","reported_seq":4579,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:23:59.223227+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:23:59.223227+0000","last_peered":"2024-09-16T10:23:59.223227+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:23:59.223227+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18343820,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4487,"num_write_kb":17919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":87},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":286458358,"num_objects":69952,"num_object_clones":0,"num_object_copies":139904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":69952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47581,"num_write_kb":190212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":356835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":579010560,"data_stored":578920235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70016,"ondisk_log_size":70016,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018898,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413192,"kb_used_data":277376,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93958648,"statfs":{"total":96636764160,"available":96213655552,"internally_reserved":0,"allocated":284033024,"data_stored":283789785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":139062960},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.712}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4369999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691843,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64156,"kb_used_data":10388,"kb_used_omap":4,"kb_used_meta":53755,"kb_avail":94307684,"statfs":{"total":96636764160,"available":96571068416,"internally_reserved":0,"allocated":10637312,"data_stored":10314587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":55045802},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8159999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]}]},{"osd":0,"up_from":24,"seq":103079215126,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":416440,"kb_used_data":276592,"kb_used_omap":4,"kb_used_meta":139835,"kb_avail":93955400,"statfs":{"total":96636764160,"available":96210329600,"internally_reserved":0,"allocated":283230208,"data_stored":282990583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":143191720},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.272}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":8.5030000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284270592,"data_stored":284266979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10805248,"data_stored":10722149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283934720,"data_stored":283931107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:04.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:04.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 52 20 72889 72869 5.47287 2.26562 0.0115461 0.0142658 2024-09-16T10:24:04.957 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:04.959 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:04.985 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-16T10:24:04.573727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":294208009,"num_objects":71846,"num_object_clones":0,"num_object_copies":143692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":71846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49475,"num_write_kb":197782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":356854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71911,"ondisk_log_size":71911,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":906828,"kb_used_data":573296,"kb_used_omap":12,"kb_used_meta":333491,"kb_avail":282208692,"statfs":{"total":289910292480,"available":288981700608,"internally_reserved":0,"allocated":587055104,"data_stored":586249515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":341494786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,10,4,12,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14474781,"num_objects":3533,"num_object_clones":0,"num_object_copies":7066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":-30125,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-662,"num_write_kb":-2645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.213031"},"pg_stats":[{"pgid":"2.f","version":"42'4477","reported_seq":4542,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.269313+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:04.269313+0000","last_peered":"2024-09-16T10:24:04.269313+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:04.269313+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4477,"log_dups_size":0,"ondisk_log_size":4477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18337792,"num_objects":4477,"num_object_clones":0,"num_object_copies":8954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4477,"num_write_kb":17908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":285},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'4394","reported_seq":4548,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:00.237868+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:00.237868+0000","last_peered":"2024-09-16T10:24:00.237868+0000","last_clean":"2024-09-16T10:24:00.237868+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:00.237868+0000","last_undegraded":"2024-09-16T10:24:00.237868+0000","last_fullsized":"2024-09-16T10:24:00.237868+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17973248,"num_objects":4388,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":607,"num_write_kb":2428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'4455","reported_seq":4637,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.295107+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:00.295107+0000","last_peered":"2024-09-16T10:24:00.295107+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:00.295107+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4455,"log_dups_size":0,"ondisk_log_size":4455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18231296,"num_objects":4451,"num_object_clones":0,"num_object_copies":8902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'4425","reported_seq":4478,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:00.276024+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:00.276024+0000","last_peered":"2024-09-16T10:24:00.276024+0000","last_clean":"2024-09-16T10:24:00.276024+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:00.276024+0000","last_undegraded":"2024-09-16T10:24:00.276024+0000","last_fullsized":"2024-09-16T10:24:00.276024+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18124800,"num_objects":4425,"num_object_clones":0,"num_object_copies":8850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4425,"num_write_kb":17700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'4513","reported_seq":4666,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:04.271078+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:04.271078+0000","last_peered":"2024-09-16T10:24:04.271078+0000","last_clean":"2024-09-16T10:24:04.271078+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:04.271078+0000","last_undegraded":"2024-09-16T10:24:04.271078+0000","last_fullsized":"2024-09-16T10:24:04.271078+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18456576,"num_objects":4506,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1155,"num_write_kb":4620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'4497","reported_seq":4657,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.253100+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:04.253100+0000","last_peered":"2024-09-16T10:24:04.253100+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:04.253100+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18399232,"num_objects":4492,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1122,"num_write_kb":4488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":275},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'4687","reported_seq":4857,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.273691+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:04.273691+0000","last_peered":"2024-09-16T10:24:04.273691+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:04.273691+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":4687,"log_dups_size":0,"ondisk_log_size":4687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19173376,"num_objects":4681,"num_object_clones":0,"num_object_copies":9362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":929,"num_write_kb":3716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":278},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'4501","reported_seq":4672,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:04.226573+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:04.226573+0000","last_peered":"2024-09-16T10:24:04.226573+0000","last_clean":"2024-09-16T10:24:04.226573+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:04.226573+0000","last_undegraded":"2024-09-16T10:24:04.226573+0000","last_fullsized":"2024-09-16T10:24:04.226573+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18427904,"num_objects":4499,"num_object_clones":0,"num_object_copies":8998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'4332","reported_seq":4595,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:00.300303+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:00.300303+0000","last_peered":"2024-09-16T10:24:00.300303+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:00.300303+0000","last_undegraded":"2024-09-16T10:24:00.300303+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17716166,"num_objects":4327,"num_object_clones":0,"num_object_copies":8654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4330,"num_write_kb":17304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4111},{"shards":"2","objects":216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'4720","reported_seq":4795,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.179590+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:04.179590+0000","last_peered":"2024-09-16T10:24:04.179590+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:04.179590+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19301318,"num_objects":4714,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4718,"num_write_kb":18852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":280},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'4402","reported_seq":4504,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.192662+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:00.192662+0000","last_peered":"2024-09-16T10:24:00.192662+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:00.192662+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18011078,"num_objects":4399,"num_object_clones":0,"num_object_copies":8798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":17591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'4304","reported_seq":4387,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.242617+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:00.242617+0000","last_peered":"2024-09-16T10:24:00.242617+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:00.242617+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17605574,"num_objects":4300,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":17195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4413","reported_seq":4489,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.281202+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:04.281202+0000","last_peered":"2024-09-16T10:24:04.281202+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:04.281202+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18067939,"num_objects":4412,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4412,"num_write_kb":17645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":253},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'4633","reported_seq":4763,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.206401+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:04.206401+0000","last_peered":"2024-09-16T10:24:04.206401+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:04.206401+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":4633,"log_dups_size":0,"ondisk_log_size":4633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945449,"num_objects":4628,"num_object_clones":0,"num_object_copies":9256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":18504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":291},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'4453","reported_seq":4527,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.242736+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:04.242736+0000","last_peered":"2024-09-16T10:24:04.242736+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:04.242736+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18224070,"num_objects":4451,"num_object_clones":0,"num_object_copies":8902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4451,"num_write_kb":17798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":284},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'4702","reported_seq":4791,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.277094+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:04.277094+0000","last_peered":"2024-09-16T10:24:04.277094+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:04.277094+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19212172,"num_objects":4694,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":299},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":294207990,"num_objects":71844,"num_object_clones":0,"num_object_copies":143688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":71844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49473,"num_write_kb":197780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":356835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":588455936,"data_stored":588365611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71908,"ondisk_log_size":71908,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018898,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413192,"kb_used_data":277376,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93958648,"statfs":{"total":96636764160,"available":96213655552,"internally_reserved":0,"allocated":284033024,"data_stored":283789785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":139062960},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.712}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4369999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691843,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64156,"kb_used_data":10388,"kb_used_omap":4,"kb_used_meta":53755,"kb_avail":94307684,"statfs":{"total":96636764160,"available":96571068416,"internally_reserved":0,"allocated":10637312,"data_stored":10314587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":55045802},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8159999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]}]},{"osd":0,"up_from":24,"seq":103079215127,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429480,"kb_used_data":285532,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93942360,"statfs":{"total":96636764160,"available":96196976640,"internally_reserved":0,"allocated":292384768,"data_stored":292145143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":147386024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7510000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":6.7000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293715968,"data_stored":293712355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10805248,"data_stored":10722149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283934720,"data_stored":283931107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:04.986 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:04.986 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'4477", 'reported_seq': 4542, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.269313+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:04.269313+0000', 'last_peered': '2024-09-16T10:24:04.269313+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:04.269313+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4477, 'log_dups_size': 0, 'ondisk_log_size': 4477, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18337792, 'num_objects': 4477, 'num_object_clones': 0, 'num_object_copies': 8954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4477, 'num_write_kb': 17908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 285}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:04.986 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:04.986 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'4455", 'reported_seq': 4637, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:00.295107+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:00.295107+0000', 'last_peered': '2024-09-16T10:24:00.295107+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:00.295107+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 4455, 'log_dups_size': 0, 'ondisk_log_size': 4455, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18231296, 'num_objects': 4451, 'num_object_clones': 0, 'num_object_copies': 8902, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 512, 'num_write_kb': 2048, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 116}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:04.987 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:04.987 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'4497", 'reported_seq': 4657, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.253100+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:04.253100+0000', 'last_peered': '2024-09-16T10:24:04.253100+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:04.253100+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 4497, 'log_dups_size': 0, 'ondisk_log_size': 4497, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18399232, 'num_objects': 4492, 'num_object_clones': 0, 'num_object_copies': 8984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1122, 'num_write_kb': 4488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 275}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:04.987 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:04.987 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'4687", 'reported_seq': 4857, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.273691+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:04.273691+0000', 'last_peered': '2024-09-16T10:24:04.273691+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:04.273691+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 4687, 'log_dups_size': 0, 'ondisk_log_size': 4687, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19173376, 'num_objects': 4681, 'num_object_clones': 0, 'num_object_copies': 9362, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 929, 'num_write_kb': 3716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 278}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:04.987 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:04.987 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'4332", 'reported_seq': 4595, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:00.300303+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:00.300303+0000', 'last_peered': '2024-09-16T10:24:00.300303+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:00.300303+0000', 'last_undegraded': '2024-09-16T10:24:00.300303+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4332, 'log_dups_size': 0, 'ondisk_log_size': 4332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17716166, 'num_objects': 4327, 'num_object_clones': 0, 'num_object_copies': 8654, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4111, 'num_objects_unfound': 0, 'num_objects_dirty': 4327, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4330, 'num_write_kb': 17304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 88, 'num_bytes_recovered': 352256, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4111}, {'shards': '2', 'objects': 216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:04.987 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:04.987 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'4720", 'reported_seq': 4795, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.179590+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:04.179590+0000', 'last_peered': '2024-09-16T10:24:04.179590+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:04.179590+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4720, 'log_dups_size': 0, 'ondisk_log_size': 4720, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19301318, 'num_objects': 4714, 'num_object_clones': 0, 'num_object_copies': 9428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4714, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4718, 'num_write_kb': 18852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 280}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:04.987 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:04.988 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'4402", 'reported_seq': 4504, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:00.192662+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:00.192662+0000', 'last_peered': '2024-09-16T10:24:00.192662+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:00.192662+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 4402, 'log_dups_size': 0, 'ondisk_log_size': 4402, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18011078, 'num_objects': 4399, 'num_object_clones': 0, 'num_object_copies': 8798, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4400, 'num_write_kb': 17591, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 122}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:04.988 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:04.988 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'4304", 'reported_seq': 4387, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:00.242617+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:00.242617+0000', 'last_peered': '2024-09-16T10:24:00.242617+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:00.242617+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4304, 'log_dups_size': 0, 'ondisk_log_size': 4304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17605574, 'num_objects': 4300, 'num_object_clones': 0, 'num_object_copies': 8600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4302, 'num_write_kb': 17195, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 108}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:04.988 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:04.988 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'4413", 'reported_seq': 4489, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.281202+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:04.281202+0000', 'last_peered': '2024-09-16T10:24:04.281202+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:04.281202+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4413, 'log_dups_size': 0, 'ondisk_log_size': 4413, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18067939, 'num_objects': 4412, 'num_object_clones': 0, 'num_object_copies': 8824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4412, 'num_write_kb': 17645, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 253}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:04.988 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:04.988 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'4633", 'reported_seq': 4763, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.206401+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:04.206401+0000', 'last_peered': '2024-09-16T10:24:04.206401+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:04.206401+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 4633, 'log_dups_size': 0, 'ondisk_log_size': 4633, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18945449, 'num_objects': 4628, 'num_object_clones': 0, 'num_object_copies': 9256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4630, 'num_write_kb': 18504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 291}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:04.988 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:04.989 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'4453", 'reported_seq': 4527, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.242736+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:04.242736+0000', 'last_peered': '2024-09-16T10:24:04.242736+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:04.242736+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18224070, 'num_objects': 4451, 'num_object_clones': 0, 'num_object_copies': 8902, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4451, 'num_write_kb': 17798, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 284}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:04.989 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:04.989 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'4702", 'reported_seq': 4791, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.277094+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:04.277094+0000', 'last_peered': '2024-09-16T10:24:04.277094+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:04.277094+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4702, 'log_dups_size': 0, 'ondisk_log_size': 4702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19212172, 'num_objects': 4694, 'num_object_clones': 0, 'num_object_copies': 9388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4699, 'num_write_kb': 18767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 299}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:04.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:05.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 53 20 73436 73416 5.40993 2.13672 0.038746 0.0144293 2024-09-16T10:24:05.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:05.679 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/143692 objects degraded (32.798%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":134,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":71846,"data_bytes":294208009,"bytes_used":928591872,"bytes_avail":288981700608,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":143692,"degraded_ratio":0.32797928903488016,"misplaced_objects":4111,"misplaced_total":143692,"misplaced_ratio":0.028609804303649473,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":38234,"recovering_keys_per_sec":0,"num_objects_recovered":85,"num_bytes_recovered":352256,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T10:23:56.572050+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:24:05.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:05.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:06.261 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:06.264 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:06.289 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-16T10:24:04.573727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":294208009,"num_objects":71846,"num_object_clones":0,"num_object_copies":143692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":71846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49475,"num_write_kb":197782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":356854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71911,"ondisk_log_size":71911,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":906828,"kb_used_data":573296,"kb_used_omap":12,"kb_used_meta":333491,"kb_avail":282208692,"statfs":{"total":289910292480,"available":288981700608,"internally_reserved":0,"allocated":587055104,"data_stored":586249515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":341494786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,10,4,12,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14474781,"num_objects":3533,"num_object_clones":0,"num_object_copies":7066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":-30125,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-662,"num_write_kb":-2645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.213031"},"pg_stats":[{"pgid":"2.f","version":"42'4477","reported_seq":4542,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.269313+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:04.269313+0000","last_peered":"2024-09-16T10:24:04.269313+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:04.269313+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4477,"log_dups_size":0,"ondisk_log_size":4477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18337792,"num_objects":4477,"num_object_clones":0,"num_object_copies":8954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4477,"num_write_kb":17908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":285},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'4394","reported_seq":4548,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:00.237868+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:00.237868+0000","last_peered":"2024-09-16T10:24:00.237868+0000","last_clean":"2024-09-16T10:24:00.237868+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:00.237868+0000","last_undegraded":"2024-09-16T10:24:00.237868+0000","last_fullsized":"2024-09-16T10:24:00.237868+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4394,"log_dups_size":0,"ondisk_log_size":4394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17973248,"num_objects":4388,"num_object_clones":0,"num_object_copies":8776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":607,"num_write_kb":2428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'4455","reported_seq":4637,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.295107+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:00.295107+0000","last_peered":"2024-09-16T10:24:00.295107+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:00.295107+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4455,"log_dups_size":0,"ondisk_log_size":4455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18231296,"num_objects":4451,"num_object_clones":0,"num_object_copies":8902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":116}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'4425","reported_seq":4478,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:00.276024+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:00.276024+0000","last_peered":"2024-09-16T10:24:00.276024+0000","last_clean":"2024-09-16T10:24:00.276024+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:00.276024+0000","last_undegraded":"2024-09-16T10:24:00.276024+0000","last_fullsized":"2024-09-16T10:24:00.276024+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18124800,"num_objects":4425,"num_object_clones":0,"num_object_copies":8850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4425,"num_write_kb":17700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'4513","reported_seq":4666,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:04.271078+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:04.271078+0000","last_peered":"2024-09-16T10:24:04.271078+0000","last_clean":"2024-09-16T10:24:04.271078+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:04.271078+0000","last_undegraded":"2024-09-16T10:24:04.271078+0000","last_fullsized":"2024-09-16T10:24:04.271078+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18456576,"num_objects":4506,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1155,"num_write_kb":4620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'4497","reported_seq":4657,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.253100+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:04.253100+0000","last_peered":"2024-09-16T10:24:04.253100+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:04.253100+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18399232,"num_objects":4492,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1122,"num_write_kb":4488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":275},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'4687","reported_seq":4857,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.273691+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:04.273691+0000","last_peered":"2024-09-16T10:24:04.273691+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:04.273691+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":4687,"log_dups_size":0,"ondisk_log_size":4687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19173376,"num_objects":4681,"num_object_clones":0,"num_object_copies":9362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":929,"num_write_kb":3716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":278},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'4501","reported_seq":4672,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:04.226573+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:04.226573+0000","last_peered":"2024-09-16T10:24:04.226573+0000","last_clean":"2024-09-16T10:24:04.226573+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:04.226573+0000","last_undegraded":"2024-09-16T10:24:04.226573+0000","last_fullsized":"2024-09-16T10:24:04.226573+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18427904,"num_objects":4499,"num_object_clones":0,"num_object_copies":8998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'4332","reported_seq":4595,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:00.300303+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:00.300303+0000","last_peered":"2024-09-16T10:24:00.300303+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:00.300303+0000","last_undegraded":"2024-09-16T10:24:00.300303+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4332,"log_dups_size":0,"ondisk_log_size":4332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17716166,"num_objects":4327,"num_object_clones":0,"num_object_copies":8654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4330,"num_write_kb":17304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4111},{"shards":"2","objects":216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'4720","reported_seq":4795,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.179590+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:04.179590+0000","last_peered":"2024-09-16T10:24:04.179590+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:04.179590+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19301318,"num_objects":4714,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4718,"num_write_kb":18852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":280},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'4402","reported_seq":4504,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.192662+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:00.192662+0000","last_peered":"2024-09-16T10:24:00.192662+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:00.192662+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4402,"log_dups_size":0,"ondisk_log_size":4402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18011078,"num_objects":4399,"num_object_clones":0,"num_object_copies":8798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":17591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'4304","reported_seq":4387,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:00.242617+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:00.242617+0000","last_peered":"2024-09-16T10:24:00.242617+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:00.242617+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17605574,"num_objects":4300,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4302,"num_write_kb":17195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4413","reported_seq":4489,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.281202+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:04.281202+0000","last_peered":"2024-09-16T10:24:04.281202+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:04.281202+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18067939,"num_objects":4412,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4412,"num_write_kb":17645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":253},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'4633","reported_seq":4763,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.206401+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:04.206401+0000","last_peered":"2024-09-16T10:24:04.206401+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:04.206401+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":4633,"log_dups_size":0,"ondisk_log_size":4633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945449,"num_objects":4628,"num_object_clones":0,"num_object_copies":9256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":18504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":291},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'4453","reported_seq":4527,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.242736+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:04.242736+0000","last_peered":"2024-09-16T10:24:04.242736+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:04.242736+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18224070,"num_objects":4451,"num_object_clones":0,"num_object_copies":8902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4451,"num_write_kb":17798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":284},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'4702","reported_seq":4791,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.277094+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:04.277094+0000","last_peered":"2024-09-16T10:24:04.277094+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:04.277094+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19212172,"num_objects":4694,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":299},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":294207990,"num_objects":71844,"num_object_clones":0,"num_object_copies":143688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":4111,"num_objects_unfound":0,"num_objects_dirty":71844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49473,"num_write_kb":197780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":356835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":588455936,"data_stored":588365611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71908,"ondisk_log_size":71908,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018898,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":413192,"kb_used_data":277376,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":93958648,"statfs":{"total":96636764160,"available":96213655552,"internally_reserved":0,"allocated":284033024,"data_stored":283789785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":139062960},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.712}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4369999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691843,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64156,"kb_used_data":10388,"kb_used_omap":4,"kb_used_meta":53755,"kb_avail":94307684,"statfs":{"total":96636764160,"available":96571068416,"internally_reserved":0,"allocated":10637312,"data_stored":10314587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":55045802},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.8159999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]}]},{"osd":0,"up_from":24,"seq":103079215127,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429480,"kb_used_data":285532,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93942360,"statfs":{"total":96636764160,"available":96196976640,"internally_reserved":0,"allocated":292384768,"data_stored":292145143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":147386024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7510000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":6.7000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293715968,"data_stored":293712355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10805248,"data_stored":10722149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283934720,"data_stored":283931107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:06.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 54 19 74009 73990 5.35127 2.24219 0.0175441 0.01459 2024-09-16T10:24:07.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 55 20 74643 74623 5.29893 2.47266 0.00966232 0.0147347 2024-09-16T10:24:08.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 56 20 75270 75250 5.24805 2.44922 0.0329736 0.0148778 2024-09-16T10:24:09.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:09.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 57 20 75869 75849 5.19702 2.33984 0.0223553 0.0150195 2024-09-16T10:24:09.828 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:09.829 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:09.859 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-16T10:24:08.574660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298824201,"num_objects":72973,"num_object_clones":0,"num_object_copies":145946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3981,"num_objects_unfound":0,"num_objects_dirty":72973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50602,"num_write_kb":202290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":889334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73038,"ondisk_log_size":73038,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":935952,"kb_used_data":588468,"kb_used_omap":12,"kb_used_meta":347443,"kb_avail":282179568,"statfs":{"total":289910292480,"available":288951877632,"internally_reserved":0,"allocated":602591232,"data_stored":601785643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":355781634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,8,6,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19090973,"num_objects":4660,"num_object_clones":0,"num_object_copies":9320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":-30255,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":465,"num_write_kb":1863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003698"},"pg_stats":[{"pgid":"2.f","version":"42'4477","reported_seq":4542,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.269313+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:04.269313+0000","last_peered":"2024-09-16T10:24:04.269313+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:04.269313+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4477,"log_dups_size":0,"ondisk_log_size":4477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18337792,"num_objects":4477,"num_object_clones":0,"num_object_copies":8954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4477,"num_write_kb":17908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":285},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'4603","reported_seq":4757,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:05.302927+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:05.302927+0000","last_peered":"2024-09-16T10:24:05.302927+0000","last_clean":"2024-09-16T10:24:05.302927+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:05.302927+0000","last_undegraded":"2024-09-16T10:24:05.302927+0000","last_fullsized":"2024-09-16T10:24:05.302927+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4603,"log_dups_size":0,"ondisk_log_size":4603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18829312,"num_objects":4597,"num_object_clones":0,"num_object_copies":9194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":816,"num_write_kb":3264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'4638","reported_seq":4820,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:05.307526+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:05.307526+0000","last_peered":"2024-09-16T10:24:05.307526+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:05.307526+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4638,"log_dups_size":0,"ondisk_log_size":4638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18980864,"num_objects":4634,"num_object_clones":0,"num_object_copies":9268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":695,"num_write_kb":2780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'4611","reported_seq":4664,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:05.168746+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:05.168746+0000","last_peered":"2024-09-16T10:24:05.168746+0000","last_clean":"2024-09-16T10:24:05.168746+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:05.168746+0000","last_undegraded":"2024-09-16T10:24:05.168746+0000","last_fullsized":"2024-09-16T10:24:05.168746+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18886656,"num_objects":4611,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4611,"num_write_kb":18444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'4513","reported_seq":4666,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:04.271078+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:04.271078+0000","last_peered":"2024-09-16T10:24:04.271078+0000","last_clean":"2024-09-16T10:24:04.271078+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:04.271078+0000","last_undegraded":"2024-09-16T10:24:04.271078+0000","last_fullsized":"2024-09-16T10:24:04.271078+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18456576,"num_objects":4506,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1155,"num_write_kb":4620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'4497","reported_seq":4657,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.253100+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:04.253100+0000","last_peered":"2024-09-16T10:24:04.253100+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:04.253100+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18399232,"num_objects":4492,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1122,"num_write_kb":4488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":275},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'4687","reported_seq":4857,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.273691+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:04.273691+0000","last_peered":"2024-09-16T10:24:04.273691+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:04.273691+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":4687,"log_dups_size":0,"ondisk_log_size":4687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19173376,"num_objects":4681,"num_object_clones":0,"num_object_copies":9362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":929,"num_write_kb":3716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":278},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'4501","reported_seq":4672,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:04.226573+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:04.226573+0000","last_peered":"2024-09-16T10:24:04.226573+0000","last_clean":"2024-09-16T10:24:04.226573+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:04.226573+0000","last_undegraded":"2024-09-16T10:24:04.226573+0000","last_fullsized":"2024-09-16T10:24:04.226573+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18427904,"num_objects":4499,"num_object_clones":0,"num_object_copies":8998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'4526","reported_seq":5051,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:05.312700+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:05.312700+0000","last_peered":"2024-09-16T10:24:05.312700+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:05.312700+0000","last_undegraded":"2024-09-16T10:24:05.312700+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18510790,"num_objects":4521,"num_object_clones":0,"num_object_copies":9042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3981,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4524,"num_write_kb":18080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3981},{"shards":"2","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'4720","reported_seq":4795,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.179590+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:04.179590+0000","last_peered":"2024-09-16T10:24:04.179590+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:04.179590+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19301318,"num_objects":4714,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4718,"num_write_kb":18852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":280},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'4571","reported_seq":4673,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:05.227950+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:05.227950+0000","last_peered":"2024-09-16T10:24:05.227950+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:05.227950+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18703302,"num_objects":4568,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4569,"num_write_kb":18267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":291}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'4490","reported_seq":4573,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:05.227333+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:05.227333+0000","last_peered":"2024-09-16T10:24:05.227333+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:05.227333+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18367430,"num_objects":4486,"num_object_clones":0,"num_object_copies":8972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":17939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4413","reported_seq":4489,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.281202+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:04.281202+0000","last_peered":"2024-09-16T10:24:04.281202+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:04.281202+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18067939,"num_objects":4412,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4412,"num_write_kb":17645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":253},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'4633","reported_seq":4763,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.206401+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:04.206401+0000","last_peered":"2024-09-16T10:24:04.206401+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:04.206401+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":4633,"log_dups_size":0,"ondisk_log_size":4633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945449,"num_objects":4628,"num_object_clones":0,"num_object_copies":9256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":18504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":291},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'4453","reported_seq":4527,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.242736+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:04.242736+0000","last_peered":"2024-09-16T10:24:04.242736+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:04.242736+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18224070,"num_objects":4451,"num_object_clones":0,"num_object_copies":8902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4451,"num_write_kb":17798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":284},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'4702","reported_seq":4791,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.277094+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:04.277094+0000","last_peered":"2024-09-16T10:24:04.277094+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:04.277094+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19212172,"num_objects":4694,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":299},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":298824182,"num_objects":72971,"num_object_clones":0,"num_object_copies":145942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3981,"num_objects_unfound":0,"num_objects_dirty":72971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50600,"num_write_kb":202288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":889315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":604213248,"data_stored":604122923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73035,"ondisk_log_size":73035,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018899,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":423188,"kb_used_data":283276,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":93948652,"statfs":{"total":96636764160,"available":96203419648,"internally_reserved":0,"allocated":290074624,"data_stored":289831385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":143257264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":1.343},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":0.254}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6420000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691844,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83284,"kb_used_data":19660,"kb_used_omap":4,"kb_used_meta":63611,"kb_avail":94288556,"statfs":{"total":96636764160,"available":96551481344,"internally_reserved":0,"allocated":20131840,"data_stored":19809115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":65138346},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.038}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215127,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429480,"kb_used_data":285532,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93942360,"statfs":{"total":96636764160,"available":96196976640,"internally_reserved":0,"allocated":292384768,"data_stored":292145143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":147386024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7510000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":6.7000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293715968,"data_stored":293712355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20529152,"data_stored":20446053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289968128,"data_stored":289964515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:09.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:10.387 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:10.388 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:10.417 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-16T10:24:08.574660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":298824201,"num_objects":72973,"num_object_clones":0,"num_object_copies":145946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3981,"num_objects_unfound":0,"num_objects_dirty":72973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50602,"num_write_kb":202290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":889334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73038,"ondisk_log_size":73038,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":935952,"kb_used_data":588468,"kb_used_omap":12,"kb_used_meta":347443,"kb_avail":282179568,"statfs":{"total":289910292480,"available":288951877632,"internally_reserved":0,"allocated":602591232,"data_stored":601785643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":355781634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,8,6,5,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19090973,"num_objects":4660,"num_object_clones":0,"num_object_copies":9320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":-30255,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":465,"num_write_kb":1863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003698"},"pg_stats":[{"pgid":"2.f","version":"42'4477","reported_seq":4542,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.269313+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:04.269313+0000","last_peered":"2024-09-16T10:24:04.269313+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:04.269313+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4477,"log_dups_size":0,"ondisk_log_size":4477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18337792,"num_objects":4477,"num_object_clones":0,"num_object_copies":8954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4477,"num_write_kb":17908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":285},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'4603","reported_seq":4757,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:05.302927+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:05.302927+0000","last_peered":"2024-09-16T10:24:05.302927+0000","last_clean":"2024-09-16T10:24:05.302927+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:05.302927+0000","last_undegraded":"2024-09-16T10:24:05.302927+0000","last_fullsized":"2024-09-16T10:24:05.302927+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4603,"log_dups_size":0,"ondisk_log_size":4603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18829312,"num_objects":4597,"num_object_clones":0,"num_object_copies":9194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":816,"num_write_kb":3264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'4638","reported_seq":4820,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:05.307526+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:05.307526+0000","last_peered":"2024-09-16T10:24:05.307526+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:05.307526+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4638,"log_dups_size":0,"ondisk_log_size":4638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18980864,"num_objects":4634,"num_object_clones":0,"num_object_copies":9268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":695,"num_write_kb":2780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'4611","reported_seq":4664,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:05.168746+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:05.168746+0000","last_peered":"2024-09-16T10:24:05.168746+0000","last_clean":"2024-09-16T10:24:05.168746+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:05.168746+0000","last_undegraded":"2024-09-16T10:24:05.168746+0000","last_fullsized":"2024-09-16T10:24:05.168746+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18886656,"num_objects":4611,"num_object_clones":0,"num_object_copies":9222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4611,"num_write_kb":18444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'4513","reported_seq":4666,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:04.271078+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:04.271078+0000","last_peered":"2024-09-16T10:24:04.271078+0000","last_clean":"2024-09-16T10:24:04.271078+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:04.271078+0000","last_undegraded":"2024-09-16T10:24:04.271078+0000","last_fullsized":"2024-09-16T10:24:04.271078+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4513,"log_dups_size":0,"ondisk_log_size":4513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18456576,"num_objects":4506,"num_object_clones":0,"num_object_copies":9012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1155,"num_write_kb":4620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'4497","reported_seq":4657,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.253100+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:04.253100+0000","last_peered":"2024-09-16T10:24:04.253100+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:04.253100+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18399232,"num_objects":4492,"num_object_clones":0,"num_object_copies":8984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1122,"num_write_kb":4488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":275},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'4687","reported_seq":4857,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.273691+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:04.273691+0000","last_peered":"2024-09-16T10:24:04.273691+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:04.273691+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":4687,"log_dups_size":0,"ondisk_log_size":4687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19173376,"num_objects":4681,"num_object_clones":0,"num_object_copies":9362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":929,"num_write_kb":3716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":278},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'4501","reported_seq":4672,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:04.226573+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:04.226573+0000","last_peered":"2024-09-16T10:24:04.226573+0000","last_clean":"2024-09-16T10:24:04.226573+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:04.226573+0000","last_undegraded":"2024-09-16T10:24:04.226573+0000","last_fullsized":"2024-09-16T10:24:04.226573+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18427904,"num_objects":4499,"num_object_clones":0,"num_object_copies":8998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":1216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'4526","reported_seq":5051,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:05.312700+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:05.312700+0000","last_peered":"2024-09-16T10:24:05.312700+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:05.312700+0000","last_undegraded":"2024-09-16T10:24:05.312700+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4526,"log_dups_size":0,"ondisk_log_size":4526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18510790,"num_objects":4521,"num_object_clones":0,"num_object_copies":9042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3981,"num_objects_unfound":0,"num_objects_dirty":4521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4524,"num_write_kb":18080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3981},{"shards":"2","objects":540}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'4720","reported_seq":4795,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.179590+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:04.179590+0000","last_peered":"2024-09-16T10:24:04.179590+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:04.179590+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19301318,"num_objects":4714,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4718,"num_write_kb":18852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":280},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'4571","reported_seq":4673,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:05.227950+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:05.227950+0000","last_peered":"2024-09-16T10:24:05.227950+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:05.227950+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18703302,"num_objects":4568,"num_object_clones":0,"num_object_copies":9136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4569,"num_write_kb":18267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":291}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'4490","reported_seq":4573,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:05.227333+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:05.227333+0000","last_peered":"2024-09-16T10:24:05.227333+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:05.227333+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18367430,"num_objects":4486,"num_object_clones":0,"num_object_copies":8972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":17939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4413","reported_seq":4489,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.281202+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:04.281202+0000","last_peered":"2024-09-16T10:24:04.281202+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:04.281202+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18067939,"num_objects":4412,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4412,"num_write_kb":17645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":253},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'4633","reported_seq":4763,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.206401+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:04.206401+0000","last_peered":"2024-09-16T10:24:04.206401+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:04.206401+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":4633,"log_dups_size":0,"ondisk_log_size":4633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945449,"num_objects":4628,"num_object_clones":0,"num_object_copies":9256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":18504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":291},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'4453","reported_seq":4527,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.242736+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:04.242736+0000","last_peered":"2024-09-16T10:24:04.242736+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:04.242736+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18224070,"num_objects":4451,"num_object_clones":0,"num_object_copies":8902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4451,"num_write_kb":17798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":284},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'4702","reported_seq":4791,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:04.277094+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:04.277094+0000","last_peered":"2024-09-16T10:24:04.277094+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:04.277094+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19212172,"num_objects":4694,"num_object_clones":0,"num_object_copies":9388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":18767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":299},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":298824182,"num_objects":72971,"num_object_clones":0,"num_object_copies":145942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3981,"num_objects_unfound":0,"num_objects_dirty":72971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50600,"num_write_kb":202288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":235,"num_bytes_recovered":889315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":604213248,"data_stored":604122923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73035,"ondisk_log_size":73035,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018899,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":423188,"kb_used_data":283276,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":93948652,"statfs":{"total":96636764160,"available":96203419648,"internally_reserved":0,"allocated":290074624,"data_stored":289831385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":143257264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":1.343},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":0.254}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6420000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691844,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83284,"kb_used_data":19660,"kb_used_omap":4,"kb_used_meta":63611,"kb_avail":94288556,"statfs":{"total":96636764160,"available":96551481344,"internally_reserved":0,"allocated":20131840,"data_stored":19809115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":65138346},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.038}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215127,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429480,"kb_used_data":285532,"kb_used_omap":4,"kb_used_meta":143931,"kb_avail":93942360,"statfs":{"total":96636764160,"available":96196976640,"internally_reserved":0,"allocated":292384768,"data_stored":292145143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":147386024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.7510000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":6.7000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":293715968,"data_stored":293712355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20529152,"data_stored":20446053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289968128,"data_stored":289964515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:10.418 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:10.418 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'4477", 'reported_seq': 4542, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.269313+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:04.269313+0000', 'last_peered': '2024-09-16T10:24:04.269313+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:04.269313+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4477, 'log_dups_size': 0, 'ondisk_log_size': 4477, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18337792, 'num_objects': 4477, 'num_object_clones': 0, 'num_object_copies': 8954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4477, 'num_write_kb': 17908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 285}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:10.418 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:10.418 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'4638", 'reported_seq': 4820, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:05.307526+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:05.307526+0000', 'last_peered': '2024-09-16T10:24:05.307526+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:05.307526+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 4638, 'log_dups_size': 0, 'ondisk_log_size': 4638, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18980864, 'num_objects': 4634, 'num_object_clones': 0, 'num_object_copies': 9268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 695, 'num_write_kb': 2780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:10.418 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'4497", 'reported_seq': 4657, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.253100+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:04.253100+0000', 'last_peered': '2024-09-16T10:24:04.253100+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:04.253100+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 4497, 'log_dups_size': 0, 'ondisk_log_size': 4497, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18399232, 'num_objects': 4492, 'num_object_clones': 0, 'num_object_copies': 8984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1122, 'num_write_kb': 4488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 275}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'4687", 'reported_seq': 4857, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.273691+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:04.273691+0000', 'last_peered': '2024-09-16T10:24:04.273691+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:04.273691+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 4687, 'log_dups_size': 0, 'ondisk_log_size': 4687, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19173376, 'num_objects': 4681, 'num_object_clones': 0, 'num_object_copies': 9362, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 929, 'num_write_kb': 3716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 278}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'4526", 'reported_seq': 5051, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:05.312700+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:05.312700+0000', 'last_peered': '2024-09-16T10:24:05.312700+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:05.312700+0000', 'last_undegraded': '2024-09-16T10:24:05.312700+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4526, 'log_dups_size': 0, 'ondisk_log_size': 4526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18510790, 'num_objects': 4521, 'num_object_clones': 0, 'num_object_copies': 9042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3981, 'num_objects_unfound': 0, 'num_objects_dirty': 4521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4524, 'num_write_kb': 18080, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 218, 'num_bytes_recovered': 884736, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3981}, {'shards': '2', 'objects': 540}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'4720", 'reported_seq': 4795, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.179590+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:04.179590+0000', 'last_peered': '2024-09-16T10:24:04.179590+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:04.179590+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4720, 'log_dups_size': 0, 'ondisk_log_size': 4720, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19301318, 'num_objects': 4714, 'num_object_clones': 0, 'num_object_copies': 9428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4714, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4718, 'num_write_kb': 18852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 280}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'4571", 'reported_seq': 4673, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:05.227950+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:05.227950+0000', 'last_peered': '2024-09-16T10:24:05.227950+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:05.227950+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 4571, 'log_dups_size': 0, 'ondisk_log_size': 4571, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18703302, 'num_objects': 4568, 'num_object_clones': 0, 'num_object_copies': 9136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4568, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4569, 'num_write_kb': 18267, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 291}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:10.419 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'4490", 'reported_seq': 4573, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:05.227333+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:05.227333+0000', 'last_peered': '2024-09-16T10:24:05.227333+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:05.227333+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18367430, 'num_objects': 4486, 'num_object_clones': 0, 'num_object_copies': 8972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4488, 'num_write_kb': 17939, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 294}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:10.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:10.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'4413", 'reported_seq': 4489, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.281202+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:04.281202+0000', 'last_peered': '2024-09-16T10:24:04.281202+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:04.281202+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4413, 'log_dups_size': 0, 'ondisk_log_size': 4413, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18067939, 'num_objects': 4412, 'num_object_clones': 0, 'num_object_copies': 8824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4412, 'num_write_kb': 17645, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 253}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:10.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:10.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'4633", 'reported_seq': 4763, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.206401+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:04.206401+0000', 'last_peered': '2024-09-16T10:24:04.206401+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:04.206401+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 4633, 'log_dups_size': 0, 'ondisk_log_size': 4633, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18945449, 'num_objects': 4628, 'num_object_clones': 0, 'num_object_copies': 9256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4628, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4630, 'num_write_kb': 18504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 291}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:10.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:10.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'4453", 'reported_seq': 4527, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.242736+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:04.242736+0000', 'last_peered': '2024-09-16T10:24:04.242736+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:04.242736+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18224070, 'num_objects': 4451, 'num_object_clones': 0, 'num_object_copies': 8902, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4451, 'num_write_kb': 17798, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 284}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:10.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:10.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'4702", 'reported_seq': 4791, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:04.277094+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:04.277094+0000', 'last_peered': '2024-09-16T10:24:04.277094+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:04.277094+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4702, 'log_dups_size': 0, 'ondisk_log_size': 4702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19212172, 'num_objects': 4694, 'num_object_clones': 0, 'num_object_copies': 9388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4699, 'num_write_kb': 18767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 299}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:10.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:10.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 58 20 76409 76389 5.14379 2.10938 0.00906613 0.015178 2024-09-16T10:24:11.105 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:11.105 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/151960 objects degraded (31.013%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":139,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":75980,"data_bytes":311140873,"bytes_used":990605312,"bytes_avail":288919687168,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":151960,"degraded_ratio":0.31013424585417215,"misplaced_objects":3845,"misplaced_total":151960,"misplaced_ratio":0.025302711239799946,"recovering_objects_per_sec":29,"recovering_bytes_per_sec":120121,"recovering_keys_per_sec":0,"num_objects_recovered":351,"num_bytes_recovered":1441792,"num_keys_recovered":0,"write_bytes_sec":1185094,"read_op_per_sec":0,"write_op_per_sec":289},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T10:23:56.572050+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:24:11.126 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:11.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:11.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 59 20 76948 76928 5.0923 2.10547 0.0654847 0.0153322 2024-09-16T10:24:11.656 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:11.656 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:11.679 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-16T10:24:10.575111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":311140873,"num_objects":75980,"num_object_clones":0,"num_object_copies":151960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3845,"num_objects_unfound":0,"num_objects_dirty":75980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53609,"num_write_kb":214318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":1446390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76045,"ondisk_log_size":76045,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":967388,"kb_used_data":603524,"kb_used_omap":12,"kb_used_meta":363827,"kb_avail":282148132,"statfs":{"total":289910292480,"available":288919687168,"internally_reserved":0,"allocated":618008576,"data_stored":617202987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":372558850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,4,7,9,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31407645,"num_objects":7667,"num_object_clones":0,"num_object_copies":15334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":-30391,"num_objects_unfound":0,"num_objects_dirty":7667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3472,"num_write_kb":13891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002745"},"pg_stats":[{"pgid":"2.f","version":"42'4669","reported_seq":4734,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:09.348689+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:09.348689+0000","last_peered":"2024-09-16T10:24:09.348689+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:09.348689+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19124224,"num_objects":4669,"num_object_clones":0,"num_object_copies":9338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4669,"num_write_kb":18676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":477},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'4799","reported_seq":4953,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:10.298366+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:10.298366+0000","last_peered":"2024-09-16T10:24:10.298366+0000","last_clean":"2024-09-16T10:24:10.298366+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:10.298366+0000","last_undegraded":"2024-09-16T10:24:10.298366+0000","last_fullsized":"2024-09-16T10:24:10.298366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19632128,"num_objects":4793,"num_object_clones":0,"num_object_copies":9586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1012,"num_write_kb":4048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'4821","reported_seq":5003,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:10.278646+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:10.278646+0000","last_peered":"2024-09-16T10:24:10.278646+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:10.278646+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4821,"log_dups_size":0,"ondisk_log_size":4821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19730432,"num_objects":4817,"num_object_clones":0,"num_object_copies":9634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":878,"num_write_kb":3512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'4803","reported_seq":4856,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:10.325394+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:10.325394+0000","last_peered":"2024-09-16T10:24:10.325394+0000","last_clean":"2024-09-16T10:24:10.325394+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:10.325394+0000","last_undegraded":"2024-09-16T10:24:10.325394+0000","last_fullsized":"2024-09-16T10:24:10.325394+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19673088,"num_objects":4803,"num_object_clones":0,"num_object_copies":9606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4803,"num_write_kb":19212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'4702","reported_seq":4855,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:09.308890+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:09.308890+0000","last_peered":"2024-09-16T10:24:09.308890+0000","last_clean":"2024-09-16T10:24:09.308890+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:09.308890+0000","last_undegraded":"2024-09-16T10:24:09.308890+0000","last_fullsized":"2024-09-16T10:24:09.308890+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19230720,"num_objects":4695,"num_object_clones":0,"num_object_copies":9390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1344,"num_write_kb":5376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'4688","reported_seq":4848,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:09.274827+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:09.274827+0000","last_peered":"2024-09-16T10:24:09.274827+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:09.274827+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19181568,"num_objects":4683,"num_object_clones":0,"num_object_copies":9366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1313,"num_write_kb":5252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":466},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'4867","reported_seq":5037,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:09.296881+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:09.296881+0000","last_peered":"2024-09-16T10:24:09.296881+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:09.296881+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19910656,"num_objects":4861,"num_object_clones":0,"num_object_copies":9722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1109,"num_write_kb":4436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":458},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'4695","reported_seq":4867,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:09.212746+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:09.212746+0000","last_peered":"2024-09-16T10:24:09.212746+0000","last_clean":"2024-09-16T10:24:09.212746+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:09.212746+0000","last_undegraded":"2024-09-16T10:24:09.212746+0000","last_fullsized":"2024-09-16T10:24:09.212746+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":4695,"log_dups_size":0,"ondisk_log_size":4695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19222528,"num_objects":4693,"num_object_clones":0,"num_object_copies":9386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":498,"num_write_kb":1992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'4711","reported_seq":5508,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:10.305011+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:10.305011+0000","last_peered":"2024-09-16T10:24:10.305011+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:10.305011+0000","last_undegraded":"2024-09-16T10:24:10.305011+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19268550,"num_objects":4706,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3845,"num_objects_unfound":0,"num_objects_dirty":4706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":18820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3845},{"shards":"2","objects":861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'4901","reported_seq":4970,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:09.273356+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:09.273356+0000","last_peered":"2024-09-16T10:24:09.273356+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:09.273356+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4901,"log_dups_size":0,"ondisk_log_size":4901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20042694,"num_objects":4895,"num_object_clones":0,"num_object_copies":9790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4899,"num_write_kb":19576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":461},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'4751","reported_seq":4853,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:10.184582+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:10.184582+0000","last_peered":"2024-09-16T10:24:10.184582+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:10.184582+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19440582,"num_objects":4748,"num_object_clones":0,"num_object_copies":9496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":18987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":471}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'4680","reported_seq":4762,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:10.287228+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:10.287228+0000","last_peered":"2024-09-16T10:24:10.287228+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:10.287228+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4680,"log_dups_size":0,"ondisk_log_size":4680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19145670,"num_objects":4676,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4678,"num_write_kb":18699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":484}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4599","reported_seq":4675,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:09.346190+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:09.346190+0000","last_peered":"2024-09-16T10:24:09.346190+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:09.346190+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4599,"log_dups_size":0,"ondisk_log_size":4599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18829795,"num_objects":4598,"num_object_clones":0,"num_object_copies":9196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4598,"num_write_kb":18389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":439},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'4807","reported_seq":4937,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:09.319710+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:09.319710+0000","last_peered":"2024-09-16T10:24:09.319710+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:09.319710+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19658153,"num_objects":4802,"num_object_clones":0,"num_object_copies":9604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4804,"num_write_kb":19200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":465},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'4645","reported_seq":4722,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:09.328920+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:09.328920+0000","last_peered":"2024-09-16T10:24:09.328920+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:09.328920+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4645,"log_dups_size":0,"ondisk_log_size":4645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19010502,"num_objects":4643,"num_object_clones":0,"num_object_copies":9286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4643,"num_write_kb":18566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":476},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'4904","reported_seq":4993,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:09.256462+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:09.256462+0000","last_peered":"2024-09-16T10:24:09.256462+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:09.256462+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4904,"log_dups_size":0,"ondisk_log_size":4904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20039564,"num_objects":4896,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4901,"num_write_kb":19575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":501},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":311140854,"num_objects":75978,"num_object_clones":0,"num_object_copies":151956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3845,"num_objects_unfound":0,"num_objects_dirty":75978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53607,"num_write_kb":214316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":1446371,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":619655168,"data_stored":619564843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76042,"ondisk_log_size":76042,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018900,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437344,"kb_used_data":289240,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93934496,"statfs":{"total":96636764160,"available":96188923904,"internally_reserved":0,"allocated":296181760,"data_stored":295938521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":151645872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.1499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.8879999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691844,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83284,"kb_used_data":19660,"kb_used_omap":4,"kb_used_meta":63611,"kb_avail":94288556,"statfs":{"total":96636764160,"available":96551481344,"internally_reserved":0,"allocated":20131840,"data_stored":19809115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":65138346},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.038}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26200000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215128,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446760,"kb_used_data":294624,"kb_used_omap":4,"kb_used_meta":152123,"kb_avail":93925080,"statfs":{"total":96636764160,"available":96179281920,"internally_reserved":0,"allocated":301694976,"data_stored":301455351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":155774632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,4,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.4990000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":2.536},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.149}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":302997504,"data_stored":302993891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20529152,"data_stored":20446053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":296128512,"data_stored":296124899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:12.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:24:12.524879+0000 min lat: 0.00139595 max lat: 2.61294 avg lat: 0.0154739 2024-09-16T10:24:12.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:24:12.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 60 20 77546 77526 5.04636 2.33594 0.0302225 0.0154739 2024-09-16T10:24:13.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 61 20 78160 78140 5.00295 2.39844 0.0185531 0.0156081 2024-09-16T10:24:14.525 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 62 20 78781 78761 4.96138 2.42578 0.0476035 0.0157387 2024-09-16T10:24:14.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:15.217 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:15.219 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:15.239 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-16T10:24:14.577487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":318657033,"num_objects":77815,"num_object_clones":0,"num_object_copies":155630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3845,"num_objects_unfound":0,"num_objects_dirty":77815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55444,"num_write_kb":221658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":1446390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77880,"ondisk_log_size":77880,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":999492,"kb_used_data":621676,"kb_used_omap":12,"kb_used_meta":377779,"kb_avail":282116028,"statfs":{"total":289910292480,"available":288886812672,"internally_reserved":0,"allocated":636596224,"data_stored":635790635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":386845698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,2,8,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32198656,"num_objects":7861,"num_object_clones":0,"num_object_copies":15722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-266,"num_objects_unfound":0,"num_objects_dirty":7861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":31444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004184"},"pg_stats":[{"pgid":"2.f","version":"42'4855","reported_seq":4920,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.340450+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:14.340450+0000","last_peered":"2024-09-16T10:24:14.340450+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:14.340450+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19886080,"num_objects":4855,"num_object_clones":0,"num_object_copies":9710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":19420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":663},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'4799","reported_seq":4953,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:10.298366+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:10.298366+0000","last_peered":"2024-09-16T10:24:10.298366+0000","last_clean":"2024-09-16T10:24:10.298366+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:10.298366+0000","last_undegraded":"2024-09-16T10:24:10.298366+0000","last_fullsized":"2024-09-16T10:24:10.298366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19632128,"num_objects":4793,"num_object_clones":0,"num_object_copies":9586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1012,"num_write_kb":4048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'4821","reported_seq":5003,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:10.278646+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:10.278646+0000","last_peered":"2024-09-16T10:24:10.278646+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:10.278646+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4821,"log_dups_size":0,"ondisk_log_size":4821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19730432,"num_objects":4817,"num_object_clones":0,"num_object_copies":9634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":878,"num_write_kb":3512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'4803","reported_seq":4856,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:10.325394+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:10.325394+0000","last_peered":"2024-09-16T10:24:10.325394+0000","last_clean":"2024-09-16T10:24:10.325394+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:10.325394+0000","last_undegraded":"2024-09-16T10:24:10.325394+0000","last_fullsized":"2024-09-16T10:24:10.325394+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19673088,"num_objects":4803,"num_object_clones":0,"num_object_copies":9606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4803,"num_write_kb":19212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'4871","reported_seq":5024,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:14.325068+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:14.325068+0000","last_peered":"2024-09-16T10:24:14.325068+0000","last_clean":"2024-09-16T10:24:14.325068+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:14.325068+0000","last_undegraded":"2024-09-16T10:24:14.325068+0000","last_fullsized":"2024-09-16T10:24:14.325068+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19922944,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":6052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'4880","reported_seq":5040,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.243935+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:14.243935+0000","last_peered":"2024-09-16T10:24:14.243935+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:14.243935+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19968000,"num_objects":4875,"num_object_clones":0,"num_object_copies":9750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1505,"num_write_kb":6020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":658},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5053","reported_seq":5223,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.298615+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:14.298615+0000","last_peered":"2024-09-16T10:24:14.298615+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:14.298615+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5053,"log_dups_size":0,"ondisk_log_size":5053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20672512,"num_objects":5047,"num_object_clones":0,"num_object_copies":10094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1295,"num_write_kb":5180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":644},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'4866","reported_seq":5038,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:14.353503+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:14.353503+0000","last_peered":"2024-09-16T10:24:14.353503+0000","last_clean":"2024-09-16T10:24:14.353503+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:14.353503+0000","last_undegraded":"2024-09-16T10:24:14.353503+0000","last_fullsized":"2024-09-16T10:24:14.353503+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19922944,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":669,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'4711","reported_seq":5508,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:10.305011+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:10.305011+0000","last_peered":"2024-09-16T10:24:10.305011+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:10.305011+0000","last_undegraded":"2024-09-16T10:24:10.305011+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19268550,"num_objects":4706,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3845,"num_objects_unfound":0,"num_objects_dirty":4706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":18820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3845},{"shards":"2","objects":861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5094","reported_seq":5166,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.366213+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:14.366213+0000","last_peered":"2024-09-16T10:24:14.366213+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:14.366213+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20833222,"num_objects":5088,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":20348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":654},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'4751","reported_seq":4853,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:10.184582+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:10.184582+0000","last_peered":"2024-09-16T10:24:10.184582+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:10.184582+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19440582,"num_objects":4748,"num_object_clones":0,"num_object_copies":9496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":18987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":471}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'4680","reported_seq":4762,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:10.287228+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:10.287228+0000","last_peered":"2024-09-16T10:24:10.287228+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:10.287228+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4680,"log_dups_size":0,"ondisk_log_size":4680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19145670,"num_objects":4676,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4678,"num_write_kb":18699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":484}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4787","reported_seq":4863,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.314182+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:14.314182+0000","last_peered":"2024-09-16T10:24:14.314182+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:14.314182+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19599843,"num_objects":4786,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4786,"num_write_kb":19141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":627},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5008","reported_seq":5138,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.343950+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:14.343950+0000","last_peered":"2024-09-16T10:24:14.343950+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:14.343950+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20481449,"num_objects":5003,"num_object_clones":0,"num_object_copies":10006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":20004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":666},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'4826","reported_seq":4900,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.308580+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:14.308580+0000","last_peered":"2024-09-16T10:24:14.308580+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:14.308580+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4826,"log_dups_size":0,"ondisk_log_size":4826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19751878,"num_objects":4824,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":19290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":657},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5072","reported_seq":5161,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.340293+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:14.340293+0000","last_peered":"2024-09-16T10:24:14.340293+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:14.340293+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20727692,"num_objects":5064,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5069,"num_write_kb":20247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":669},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":318657014,"num_objects":77813,"num_object_clones":0,"num_object_copies":155626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3845,"num_objects_unfound":0,"num_objects_dirty":77813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55442,"num_write_kb":221656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":1446371,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638230528,"data_stored":638140203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77877,"ondisk_log_size":77877,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018900,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437344,"kb_used_data":289240,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93934496,"statfs":{"total":96636764160,"available":96188923904,"internally_reserved":0,"allocated":296181760,"data_stored":295938521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":151645872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.1499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.8879999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691845,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100720,"kb_used_data":28904,"kb_used_omap":4,"kb_used_meta":71803,"kb_avail":94271120,"statfs":{"total":96636764160,"available":96533626880,"internally_reserved":0,"allocated":29597696,"data_stored":29274971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":73526954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.871}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215129,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461428,"kb_used_data":303532,"kb_used_omap":4,"kb_used_meta":157883,"kb_avail":93910412,"statfs":{"total":96636764160,"available":96164261888,"internally_reserved":0,"allocated":310816768,"data_stored":310577143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":161672872},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312037376,"data_stored":312033763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30064640,"data_stored":29981541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":296128512,"data_stored":296124899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:15.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:15.526 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 63 20 79346 79326 4.91765 2.20703 0.0835001 0.0158787 2024-09-16T10:24:15.795 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:15.797 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:15.821 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-16T10:24:14.577487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":318657033,"num_objects":77815,"num_object_clones":0,"num_object_copies":155630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3845,"num_objects_unfound":0,"num_objects_dirty":77815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55444,"num_write_kb":221658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":372,"num_bytes_recovered":1446390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77880,"ondisk_log_size":77880,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":999492,"kb_used_data":621676,"kb_used_omap":12,"kb_used_meta":377779,"kb_avail":282116028,"statfs":{"total":289910292480,"available":288886812672,"internally_reserved":0,"allocated":636596224,"data_stored":635790635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":386845698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,2,8,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32198656,"num_objects":7861,"num_object_clones":0,"num_object_copies":15722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-266,"num_objects_unfound":0,"num_objects_dirty":7861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7861,"num_write_kb":31444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004184"},"pg_stats":[{"pgid":"2.f","version":"42'4855","reported_seq":4920,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.340450+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:14.340450+0000","last_peered":"2024-09-16T10:24:14.340450+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:14.340450+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19886080,"num_objects":4855,"num_object_clones":0,"num_object_copies":9710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":19420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":663},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'4799","reported_seq":4953,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:10.298366+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:10.298366+0000","last_peered":"2024-09-16T10:24:10.298366+0000","last_clean":"2024-09-16T10:24:10.298366+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:10.298366+0000","last_undegraded":"2024-09-16T10:24:10.298366+0000","last_fullsized":"2024-09-16T10:24:10.298366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19632128,"num_objects":4793,"num_object_clones":0,"num_object_copies":9586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1012,"num_write_kb":4048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'4821","reported_seq":5003,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:10.278646+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:10.278646+0000","last_peered":"2024-09-16T10:24:10.278646+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:10.278646+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":4821,"log_dups_size":0,"ondisk_log_size":4821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19730432,"num_objects":4817,"num_object_clones":0,"num_object_copies":9634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":878,"num_write_kb":3512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'4803","reported_seq":4856,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:10.325394+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:10.325394+0000","last_peered":"2024-09-16T10:24:10.325394+0000","last_clean":"2024-09-16T10:24:10.325394+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:10.325394+0000","last_undegraded":"2024-09-16T10:24:10.325394+0000","last_fullsized":"2024-09-16T10:24:10.325394+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19673088,"num_objects":4803,"num_object_clones":0,"num_object_copies":9606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4803,"num_write_kb":19212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'4871","reported_seq":5024,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:14.325068+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:14.325068+0000","last_peered":"2024-09-16T10:24:14.325068+0000","last_clean":"2024-09-16T10:24:14.325068+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:14.325068+0000","last_undegraded":"2024-09-16T10:24:14.325068+0000","last_fullsized":"2024-09-16T10:24:14.325068+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19922944,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":6052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'4880","reported_seq":5040,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.243935+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:14.243935+0000","last_peered":"2024-09-16T10:24:14.243935+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:14.243935+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19968000,"num_objects":4875,"num_object_clones":0,"num_object_copies":9750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1505,"num_write_kb":6020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":658},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5053","reported_seq":5223,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.298615+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:14.298615+0000","last_peered":"2024-09-16T10:24:14.298615+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:14.298615+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5053,"log_dups_size":0,"ondisk_log_size":5053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20672512,"num_objects":5047,"num_object_clones":0,"num_object_copies":10094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1295,"num_write_kb":5180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":644},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'4866","reported_seq":5038,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:14.353503+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:14.353503+0000","last_peered":"2024-09-16T10:24:14.353503+0000","last_clean":"2024-09-16T10:24:14.353503+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:14.353503+0000","last_undegraded":"2024-09-16T10:24:14.353503+0000","last_fullsized":"2024-09-16T10:24:14.353503+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19922944,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":669,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'4711","reported_seq":5508,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:10.305011+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:10.305011+0000","last_peered":"2024-09-16T10:24:10.305011+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:10.305011+0000","last_undegraded":"2024-09-16T10:24:10.305011+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19268550,"num_objects":4706,"num_object_clones":0,"num_object_copies":9412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3845,"num_objects_unfound":0,"num_objects_dirty":4706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":18820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":1441792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3845},{"shards":"2","objects":861}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5094","reported_seq":5166,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.366213+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:14.366213+0000","last_peered":"2024-09-16T10:24:14.366213+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:14.366213+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20833222,"num_objects":5088,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":20348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":654},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'4751","reported_seq":4853,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:10.184582+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:10.184582+0000","last_peered":"2024-09-16T10:24:10.184582+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:10.184582+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19440582,"num_objects":4748,"num_object_clones":0,"num_object_copies":9496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":18987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":471}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'4680","reported_seq":4762,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:10.287228+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:10.287228+0000","last_peered":"2024-09-16T10:24:10.287228+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:10.287228+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4680,"log_dups_size":0,"ondisk_log_size":4680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19145670,"num_objects":4676,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4678,"num_write_kb":18699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":484}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4787","reported_seq":4863,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.314182+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:14.314182+0000","last_peered":"2024-09-16T10:24:14.314182+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:14.314182+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19599843,"num_objects":4786,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4786,"num_write_kb":19141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":627},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5008","reported_seq":5138,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.343950+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:14.343950+0000","last_peered":"2024-09-16T10:24:14.343950+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:14.343950+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20481449,"num_objects":5003,"num_object_clones":0,"num_object_copies":10006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":20004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":666},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'4826","reported_seq":4900,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.308580+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:14.308580+0000","last_peered":"2024-09-16T10:24:14.308580+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:14.308580+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4826,"log_dups_size":0,"ondisk_log_size":4826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19751878,"num_objects":4824,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":19290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":657},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5072","reported_seq":5161,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.340293+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:14.340293+0000","last_peered":"2024-09-16T10:24:14.340293+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:14.340293+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20727692,"num_objects":5064,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5069,"num_write_kb":20247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":669},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":318657014,"num_objects":77813,"num_object_clones":0,"num_object_copies":155626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3845,"num_objects_unfound":0,"num_objects_dirty":77813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55442,"num_write_kb":221656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":1446371,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638230528,"data_stored":638140203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77877,"ondisk_log_size":77877,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018900,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437344,"kb_used_data":289240,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":93934496,"statfs":{"total":96636764160,"available":96188923904,"internally_reserved":0,"allocated":296181760,"data_stored":295938521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":151645872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.1499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.8879999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691845,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100720,"kb_used_data":28904,"kb_used_omap":4,"kb_used_meta":71803,"kb_avail":94271120,"statfs":{"total":96636764160,"available":96533626880,"internally_reserved":0,"allocated":29597696,"data_stored":29274971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":73526954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.871}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215129,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461428,"kb_used_data":303532,"kb_used_omap":4,"kb_used_meta":157883,"kb_avail":93910412,"statfs":{"total":96636764160,"available":96164261888,"internally_reserved":0,"allocated":310816768,"data_stored":310577143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":161672872},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312037376,"data_stored":312033763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30064640,"data_stored":29981541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":296128512,"data_stored":296124899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:15.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:15.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'4855", 'reported_seq': 4920, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:14.340450+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:14.340450+0000', 'last_peered': '2024-09-16T10:24:14.340450+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:14.340450+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4855, 'log_dups_size': 0, 'ondisk_log_size': 4855, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19886080, 'num_objects': 4855, 'num_object_clones': 0, 'num_object_copies': 9710, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4855, 'num_write_kb': 19420, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 663}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:15.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:15.822 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'4821", 'reported_seq': 5003, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:10.278646+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:10.278646+0000', 'last_peered': '2024-09-16T10:24:10.278646+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:10.278646+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 4821, 'log_dups_size': 0, 'ondisk_log_size': 4821, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19730432, 'num_objects': 4817, 'num_object_clones': 0, 'num_object_copies': 9634, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 878, 'num_write_kb': 3512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'4880", 'reported_seq': 5040, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:14.243935+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:14.243935+0000', 'last_peered': '2024-09-16T10:24:14.243935+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:14.243935+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 4880, 'log_dups_size': 0, 'ondisk_log_size': 4880, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19968000, 'num_objects': 4875, 'num_object_clones': 0, 'num_object_copies': 9750, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1505, 'num_write_kb': 6020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 658}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'5053", 'reported_seq': 5223, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:14.298615+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:14.298615+0000', 'last_peered': '2024-09-16T10:24:14.298615+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:14.298615+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 5053, 'log_dups_size': 0, 'ondisk_log_size': 5053, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20672512, 'num_objects': 5047, 'num_object_clones': 0, 'num_object_copies': 10094, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1295, 'num_write_kb': 5180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 644}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'4711", 'reported_seq': 5508, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:10.305011+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:10.305011+0000', 'last_peered': '2024-09-16T10:24:10.305011+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:10.305011+0000', 'last_undegraded': '2024-09-16T10:24:10.305011+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4711, 'log_dups_size': 0, 'ondisk_log_size': 4711, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19268550, 'num_objects': 4706, 'num_object_clones': 0, 'num_object_copies': 9412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3845, 'num_objects_unfound': 0, 'num_objects_dirty': 4706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4709, 'num_write_kb': 18820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 354, 'num_bytes_recovered': 1441792, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3845}, {'shards': '2', 'objects': 861}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'5094", 'reported_seq': 5166, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:14.366213+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:14.366213+0000', 'last_peered': '2024-09-16T10:24:14.366213+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:14.366213+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5094, 'log_dups_size': 0, 'ondisk_log_size': 5094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20833222, 'num_objects': 5088, 'num_object_clones': 0, 'num_object_copies': 10176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5092, 'num_write_kb': 20348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 654}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'4751", 'reported_seq': 4853, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:10.184582+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:10.184582+0000', 'last_peered': '2024-09-16T10:24:10.184582+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:10.184582+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 4751, 'log_dups_size': 0, 'ondisk_log_size': 4751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19440582, 'num_objects': 4748, 'num_object_clones': 0, 'num_object_copies': 9496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4749, 'num_write_kb': 18987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 471}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:15.823 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:15.824 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'4680", 'reported_seq': 4762, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:10.287228+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:10.287228+0000', 'last_peered': '2024-09-16T10:24:10.287228+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:10.287228+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4680, 'log_dups_size': 0, 'ondisk_log_size': 4680, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19145670, 'num_objects': 4676, 'num_object_clones': 0, 'num_object_copies': 9352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4678, 'num_write_kb': 18699, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 484}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:15.824 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:15.824 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'4787", 'reported_seq': 4863, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:14.314182+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:14.314182+0000', 'last_peered': '2024-09-16T10:24:14.314182+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:14.314182+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4787, 'log_dups_size': 0, 'ondisk_log_size': 4787, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19599843, 'num_objects': 4786, 'num_object_clones': 0, 'num_object_copies': 9572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4786, 'num_write_kb': 19141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 627}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:15.824 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:15.824 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'5008", 'reported_seq': 5138, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:14.343950+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:14.343950+0000', 'last_peered': '2024-09-16T10:24:14.343950+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:14.343950+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 5008, 'log_dups_size': 0, 'ondisk_log_size': 5008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20481449, 'num_objects': 5003, 'num_object_clones': 0, 'num_object_copies': 10006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5005, 'num_write_kb': 20004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 666}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:15.824 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:15.825 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'4826", 'reported_seq': 4900, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:14.308580+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:14.308580+0000', 'last_peered': '2024-09-16T10:24:14.308580+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:14.308580+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4826, 'log_dups_size': 0, 'ondisk_log_size': 4826, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19751878, 'num_objects': 4824, 'num_object_clones': 0, 'num_object_copies': 9648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4824, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4824, 'num_write_kb': 19290, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 657}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:15.825 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:15.825 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'5072", 'reported_seq': 5161, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:14.340293+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:14.340293+0000', 'last_peered': '2024-09-16T10:24:14.340293+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:14.340293+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5072, 'log_dups_size': 0, 'ondisk_log_size': 5072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20727692, 'num_objects': 5064, 'num_object_clones': 0, 'num_object_copies': 10128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5064, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5069, 'num_write_kb': 20247, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 669}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:15.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:16.503 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:16.504 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/155630 objects degraded (30.282%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":145,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":77815,"data_bytes":318657033,"bytes_used":1023479808,"bytes_avail":288886812672,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":155630,"degraded_ratio":0.30282079290625202,"misplaced_objects":3845,"misplaced_total":155630,"misplaced_ratio":0.024706033541091049,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":90763,"recovering_keys_per_sec":0,"num_objects_recovered":266,"num_bytes_recovered":1089536,"num_keys_recovered":0,"write_bytes_sec":2682286,"read_op_per_sec":0,"write_op_per_sec":654},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T10:23:56.572050+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:24:16.527 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 64 20 79868 79848 4.87267 2.03906 0.0638245 0.016025 2024-09-16T10:24:16.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:16.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:17.093 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:17.095 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:17.130 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-09-16T10:24:16.577928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":323166729,"num_objects":78916,"num_object_clones":0,"num_object_copies":157832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":78916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56545,"num_write_kb":226062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":2015734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78981,"ondisk_log_size":78981,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1009396,"kb_used_data":627484,"kb_used_omap":12,"kb_used_meta":381875,"kb_avail":282106124,"statfs":{"total":289910292480,"available":288876670976,"internally_reserved":0,"allocated":642543616,"data_stored":641738027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":391040002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,8,8,9,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28958720,"num_objects":7070,"num_object_clones":0,"num_object_copies":14140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-405,"num_objects_unfound":0,"num_objects_dirty":7070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7070,"num_write_kb":28280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":405,"num_bytes_recovered":1658880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004200"},"pg_stats":[{"pgid":"2.f","version":"42'4855","reported_seq":4920,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.340450+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:14.340450+0000","last_peered":"2024-09-16T10:24:14.340450+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:14.340450+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4855,"log_dups_size":0,"ondisk_log_size":4855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19886080,"num_objects":4855,"num_object_clones":0,"num_object_copies":9710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4855,"num_write_kb":19420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":663},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'4996","reported_seq":5150,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:15.278976+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:15.278976+0000","last_peered":"2024-09-16T10:24:15.278976+0000","last_clean":"2024-09-16T10:24:15.278976+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:15.278976+0000","last_undegraded":"2024-09-16T10:24:15.278976+0000","last_fullsized":"2024-09-16T10:24:15.278976+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":4996,"log_dups_size":0,"ondisk_log_size":4996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20439040,"num_objects":4990,"num_object_clones":0,"num_object_copies":9980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1209,"num_write_kb":4836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5007","reported_seq":5189,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:15.278262+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:15.278262+0000","last_peered":"2024-09-16T10:24:15.278262+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:15.278262+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20492288,"num_objects":5003,"num_object_clones":0,"num_object_copies":10006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1064,"num_write_kb":4256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":668}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5004","reported_seq":5057,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:15.317288+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:15.317288+0000","last_peered":"2024-09-16T10:24:15.317288+0000","last_clean":"2024-09-16T10:24:15.317288+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:15.317288+0000","last_undegraded":"2024-09-16T10:24:15.317288+0000","last_fullsized":"2024-09-16T10:24:15.317288+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20496384,"num_objects":5004,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5004,"num_write_kb":20016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'4871","reported_seq":5024,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:14.325068+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:14.325068+0000","last_peered":"2024-09-16T10:24:14.325068+0000","last_clean":"2024-09-16T10:24:14.325068+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:14.325068+0000","last_undegraded":"2024-09-16T10:24:14.325068+0000","last_fullsized":"2024-09-16T10:24:14.325068+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19922944,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":6052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'4880","reported_seq":5040,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.243935+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:14.243935+0000","last_peered":"2024-09-16T10:24:14.243935+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:14.243935+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19968000,"num_objects":4875,"num_object_clones":0,"num_object_copies":9750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1505,"num_write_kb":6020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":658},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5053","reported_seq":5223,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.298615+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:14.298615+0000","last_peered":"2024-09-16T10:24:14.298615+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:14.298615+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5053,"log_dups_size":0,"ondisk_log_size":5053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20672512,"num_objects":5047,"num_object_clones":0,"num_object_copies":10094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1295,"num_write_kb":5180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":644},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'4866","reported_seq":5038,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:14.353503+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:14.353503+0000","last_peered":"2024-09-16T10:24:14.353503+0000","last_clean":"2024-09-16T10:24:14.353503+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:14.353503+0000","last_undegraded":"2024-09-16T10:24:14.353503+0000","last_fullsized":"2024-09-16T10:24:14.353503+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19922944,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":669,"num_write_kb":2676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'4881","reported_seq":5952,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:15.327523+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:15.327523+0000","last_peered":"2024-09-16T10:24:15.327523+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:15.327523+0000","last_undegraded":"2024-09-16T10:24:15.327523+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19964870,"num_objects":4876,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4879,"num_write_kb":19500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":2011136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3706},{"shards":"2","objects":1170}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5094","reported_seq":5166,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.366213+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:14.366213+0000","last_peered":"2024-09-16T10:24:14.366213+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:14.366213+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20833222,"num_objects":5088,"num_object_clones":0,"num_object_copies":10176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":20348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":654},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'4925","reported_seq":5027,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:15.276238+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:15.276238+0000","last_peered":"2024-09-16T10:24:15.276238+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:15.276238+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20153286,"num_objects":4922,"num_object_clones":0,"num_object_copies":9844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4923,"num_write_kb":19683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":645}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'4853","reported_seq":4936,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:15.288681+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:15.288681+0000","last_peered":"2024-09-16T10:24:15.288681+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:15.288681+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4853,"log_dups_size":0,"ondisk_log_size":4853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19854278,"num_objects":4849,"num_object_clones":0,"num_object_copies":9698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4851,"num_write_kb":19391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4787","reported_seq":4863,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.314182+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:14.314182+0000","last_peered":"2024-09-16T10:24:14.314182+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:14.314182+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19599843,"num_objects":4786,"num_object_clones":0,"num_object_copies":9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4786,"num_write_kb":19141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":627},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5008","reported_seq":5138,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.343950+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:14.343950+0000","last_peered":"2024-09-16T10:24:14.343950+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:14.343950+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20481449,"num_objects":5003,"num_object_clones":0,"num_object_copies":10006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":20004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":666},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'4826","reported_seq":4900,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.308580+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:14.308580+0000","last_peered":"2024-09-16T10:24:14.308580+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:14.308580+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4826,"log_dups_size":0,"ondisk_log_size":4826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19751878,"num_objects":4824,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":19290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":657},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5072","reported_seq":5161,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:14.340293+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:14.340293+0000","last_peered":"2024-09-16T10:24:14.340293+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:14.340293+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20727692,"num_objects":5064,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5069,"num_write_kb":20247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":669},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":323166710,"num_objects":78914,"num_object_clones":0,"num_object_copies":157828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":78914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56543,"num_write_kb":226060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":510,"num_bytes_recovered":2015715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":644202496,"data_stored":644112171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78978,"ondisk_log_size":78978,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018901,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447248,"kb_used_data":295048,"kb_used_omap":4,"kb_used_meta":152187,"kb_avail":93924592,"statfs":{"total":96636764160,"available":96178782208,"internally_reserved":0,"allocated":302129152,"data_stored":301885913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":155840176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":3.2109999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2530000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691845,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100720,"kb_used_data":28904,"kb_used_omap":4,"kb_used_meta":71803,"kb_avail":94271120,"statfs":{"total":96636764160,"available":96533626880,"internally_reserved":0,"allocated":29597696,"data_stored":29274971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":73526954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.871}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215129,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461428,"kb_used_data":303532,"kb_used_omap":4,"kb_used_meta":157883,"kb_avail":93910412,"statfs":{"total":96636764160,"available":96164261888,"internally_reserved":0,"allocated":310816768,"data_stored":310577143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":161672872},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.443}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312037376,"data_stored":312033763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":30064640,"data_stored":29981541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":302100480,"data_stored":302096867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:17.526 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 65 20 80432 80412 4.83161 2.20312 0.0164884 0.0161633 2024-09-16T10:24:18.526 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 66 20 81026 81006 4.79356 2.32031 0.0278795 0.0162881 2024-09-16T10:24:19.526 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 67 20 81631 81611 4.75728 2.36328 0.018547 0.0164141 2024-09-16T10:24:20.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:20.526 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 68 20 82186 82166 4.71921 2.16797 0.0678316 0.0165473 2024-09-16T10:24:20.686 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:20.688 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:20.710 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-16T10:24:20.579153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334893577,"num_objects":81779,"num_object_clones":0,"num_object_copies":163558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":81779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59408,"num_write_kb":237514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":645,"num_bytes_recovered":2564598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81844,"ondisk_log_size":81844,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1050804,"kb_used_data":650844,"kb_used_omap":12,"kb_used_meta":399923,"kb_avail":282064716,"statfs":{"total":289910292480,"available":288834269184,"internally_reserved":0,"allocated":666464256,"data_stored":665658667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":409521154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,7,5,4,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36069376,"num_objects":8806,"num_object_clones":0,"num_object_copies":17612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-409,"num_objects_unfound":0,"num_objects_dirty":8806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8806,"num_write_kb":35224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":1675264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004493"},"pg_stats":[{"pgid":"2.f","version":"42'5034","reported_seq":5099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.345564+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:19.345564+0000","last_peered":"2024-09-16T10:24:19.345564+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:19.345564+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20619264,"num_objects":5034,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":20136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":842},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5159","reported_seq":5313,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:20.307763+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:20.307763+0000","last_peered":"2024-09-16T10:24:20.307763+0000","last_clean":"2024-09-16T10:24:20.307763+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:20.307763+0000","last_undegraded":"2024-09-16T10:24:20.307763+0000","last_fullsized":"2024-09-16T10:24:20.307763+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21106688,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1372,"num_write_kb":5488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5207","reported_seq":5388,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.337057+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:20.337057+0000","last_peered":"2024-09-16T10:24:20.337057+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:20.337057+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21311488,"num_objects":5203,"num_object_clones":0,"num_object_copies":10406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":868}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5155","reported_seq":5208,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:20.309520+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:20.309520+0000","last_peered":"2024-09-16T10:24:20.309520+0000","last_clean":"2024-09-16T10:24:20.309520+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:20.309520+0000","last_undegraded":"2024-09-16T10:24:20.309520+0000","last_fullsized":"2024-09-16T10:24:20.309520+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21114880,"num_objects":5155,"num_object_clones":0,"num_object_copies":10310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":20620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5052","reported_seq":5205,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:19.393782+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:19.393782+0000","last_peered":"2024-09-16T10:24:19.393782+0000","last_clean":"2024-09-16T10:24:19.393782+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:19.393782+0000","last_undegraded":"2024-09-16T10:24:19.393782+0000","last_fullsized":"2024-09-16T10:24:19.393782+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20664320,"num_objects":5045,"num_object_clones":0,"num_object_copies":10090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1694,"num_write_kb":6776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5039","reported_seq":5199,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.393010+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:19.393010+0000","last_peered":"2024-09-16T10:24:19.393010+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:19.393010+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20619264,"num_objects":5034,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1664,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":817},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5243","reported_seq":5413,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.377714+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:19.377714+0000","last_peered":"2024-09-16T10:24:19.377714+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:19.377714+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5243,"log_dups_size":0,"ondisk_log_size":5243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21450752,"num_objects":5237,"num_object_clones":0,"num_object_copies":10474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1485,"num_write_kb":5940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":834},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5051","reported_seq":5223,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:19.359060+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:19.359060+0000","last_peered":"2024-09-16T10:24:19.359060+0000","last_clean":"2024-09-16T10:24:19.359060+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:19.359060+0000","last_undegraded":"2024-09-16T10:24:19.359060+0000","last_fullsized":"2024-09-16T10:24:19.359060+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5051,"log_dups_size":0,"ondisk_log_size":5051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20680704,"num_objects":5049,"num_object_clones":0,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":854,"num_write_kb":3416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5073","reported_seq":6416,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:20.369098+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:20.369098+0000","last_peered":"2024-09-16T10:24:20.369098+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:20.369098+0000","last_undegraded":"2024-09-16T10:24:20.369098+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20751302,"num_objects":5068,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5071,"num_write_kb":20268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":2560000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3572},{"shards":"2","objects":1496}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5278","reported_seq":5353,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.277045+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:19.277045+0000","last_peered":"2024-09-16T10:24:19.277045+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:19.277045+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5278,"log_dups_size":0,"ondisk_log_size":5278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21586886,"num_objects":5272,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":21084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":838},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5097","reported_seq":5194,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.286103+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:20.286103+0000","last_peered":"2024-09-16T10:24:20.286103+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:20.286103+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20857798,"num_objects":5094,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5095,"num_write_kb":20371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5028","reported_seq":5110,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.338622+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:20.338622+0000","last_peered":"2024-09-16T10:24:20.338622+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:20.338622+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5028,"log_dups_size":0,"ondisk_log_size":5028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20571078,"num_objects":5024,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5026,"num_write_kb":20091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4957","reported_seq":5033,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.321319+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:19.321319+0000","last_peered":"2024-09-16T10:24:19.321319+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:19.321319+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20296163,"num_objects":4956,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":797},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5198","reported_seq":5327,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.401393+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:19.401393+0000","last_peered":"2024-09-16T10:24:19.401393+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:19.401393+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21259689,"num_objects":5193,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5195,"num_write_kb":20764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":856},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5016","reported_seq":5092,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.404563+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:19.404563+0000","last_peered":"2024-09-16T10:24:19.404563+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:19.404563+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20530118,"num_objects":5014,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5014,"num_write_kb":20050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":847},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5254","reported_seq":5341,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.348017+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:19.348017+0000","last_peered":"2024-09-16T10:24:19.348017+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:19.348017+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5254,"log_dups_size":0,"ondisk_log_size":5254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21473164,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":20975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":851},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":334893558,"num_objects":81777,"num_object_clones":0,"num_object_copies":163554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":81777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59406,"num_write_kb":237512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":644,"num_bytes_recovered":2564579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":668110848,"data_stored":668020523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81841,"ondisk_log_size":81841,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018902,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458472,"kb_used_data":300512,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":93913368,"statfs":{"total":96636764160,"available":96167288832,"internally_reserved":0,"allocated":307724288,"data_stored":307481049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":161738416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":5.6580000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7400000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691846,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117960,"kb_used_data":37952,"kb_used_omap":4,"kb_used_meta":79995,"kb_avail":94253880,"statfs":{"total":96636764160,"available":96515973120,"internally_reserved":0,"allocated":38862848,"data_stored":38540123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":81915562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.226}]}]},{"osd":0,"up_from":24,"seq":103079215130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474372,"kb_used_data":312380,"kb_used_omap":4,"kb_used_meta":161979,"kb_avail":93897468,"statfs":{"total":96636764160,"available":96151007232,"internally_reserved":0,"allocated":319877120,"data_stored":319637495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":165867176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,4,3,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320843776,"data_stored":320840163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39415808,"data_stored":39332709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307851264,"data_stored":307847651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:20.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:21.282 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:21.309 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-16T10:24:20.579153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334893577,"num_objects":81779,"num_object_clones":0,"num_object_copies":163558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":81779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59408,"num_write_kb":237514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":645,"num_bytes_recovered":2564598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81844,"ondisk_log_size":81844,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1050804,"kb_used_data":650844,"kb_used_omap":12,"kb_used_meta":399923,"kb_avail":282064716,"statfs":{"total":289910292480,"available":288834269184,"internally_reserved":0,"allocated":666464256,"data_stored":665658667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":409521154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,7,5,4,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36069376,"num_objects":8806,"num_object_clones":0,"num_object_copies":17612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-409,"num_objects_unfound":0,"num_objects_dirty":8806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8806,"num_write_kb":35224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":1675264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004493"},"pg_stats":[{"pgid":"2.f","version":"42'5034","reported_seq":5099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.345564+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:19.345564+0000","last_peered":"2024-09-16T10:24:19.345564+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:19.345564+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20619264,"num_objects":5034,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":20136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":842},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5159","reported_seq":5313,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:20.307763+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:20.307763+0000","last_peered":"2024-09-16T10:24:20.307763+0000","last_clean":"2024-09-16T10:24:20.307763+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:20.307763+0000","last_undegraded":"2024-09-16T10:24:20.307763+0000","last_fullsized":"2024-09-16T10:24:20.307763+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21106688,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1372,"num_write_kb":5488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5207","reported_seq":5388,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.337057+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:20.337057+0000","last_peered":"2024-09-16T10:24:20.337057+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:20.337057+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21311488,"num_objects":5203,"num_object_clones":0,"num_object_copies":10406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":868}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5155","reported_seq":5208,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:20.309520+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:20.309520+0000","last_peered":"2024-09-16T10:24:20.309520+0000","last_clean":"2024-09-16T10:24:20.309520+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:20.309520+0000","last_undegraded":"2024-09-16T10:24:20.309520+0000","last_fullsized":"2024-09-16T10:24:20.309520+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21114880,"num_objects":5155,"num_object_clones":0,"num_object_copies":10310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":20620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5052","reported_seq":5205,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:19.393782+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:19.393782+0000","last_peered":"2024-09-16T10:24:19.393782+0000","last_clean":"2024-09-16T10:24:19.393782+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:19.393782+0000","last_undegraded":"2024-09-16T10:24:19.393782+0000","last_fullsized":"2024-09-16T10:24:19.393782+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20664320,"num_objects":5045,"num_object_clones":0,"num_object_copies":10090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1694,"num_write_kb":6776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5039","reported_seq":5199,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.393010+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:19.393010+0000","last_peered":"2024-09-16T10:24:19.393010+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:19.393010+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20619264,"num_objects":5034,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1664,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":817},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5243","reported_seq":5413,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.377714+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:19.377714+0000","last_peered":"2024-09-16T10:24:19.377714+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:19.377714+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5243,"log_dups_size":0,"ondisk_log_size":5243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21450752,"num_objects":5237,"num_object_clones":0,"num_object_copies":10474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1485,"num_write_kb":5940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":834},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5051","reported_seq":5223,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:19.359060+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:19.359060+0000","last_peered":"2024-09-16T10:24:19.359060+0000","last_clean":"2024-09-16T10:24:19.359060+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:19.359060+0000","last_undegraded":"2024-09-16T10:24:19.359060+0000","last_fullsized":"2024-09-16T10:24:19.359060+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5051,"log_dups_size":0,"ondisk_log_size":5051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20680704,"num_objects":5049,"num_object_clones":0,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":854,"num_write_kb":3416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5073","reported_seq":6416,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:20.369098+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:20.369098+0000","last_peered":"2024-09-16T10:24:20.369098+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:20.369098+0000","last_undegraded":"2024-09-16T10:24:20.369098+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20751302,"num_objects":5068,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5071,"num_write_kb":20268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":2560000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3572},{"shards":"2","objects":1496}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5278","reported_seq":5353,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.277045+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:19.277045+0000","last_peered":"2024-09-16T10:24:19.277045+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:19.277045+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5278,"log_dups_size":0,"ondisk_log_size":5278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21586886,"num_objects":5272,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":21084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":838},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5097","reported_seq":5194,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.286103+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:20.286103+0000","last_peered":"2024-09-16T10:24:20.286103+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:20.286103+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20857798,"num_objects":5094,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5095,"num_write_kb":20371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5028","reported_seq":5110,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.338622+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:20.338622+0000","last_peered":"2024-09-16T10:24:20.338622+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:20.338622+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5028,"log_dups_size":0,"ondisk_log_size":5028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20571078,"num_objects":5024,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5026,"num_write_kb":20091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4957","reported_seq":5033,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.321319+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:19.321319+0000","last_peered":"2024-09-16T10:24:19.321319+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:19.321319+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20296163,"num_objects":4956,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":797},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5198","reported_seq":5327,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.401393+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:19.401393+0000","last_peered":"2024-09-16T10:24:19.401393+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:19.401393+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21259689,"num_objects":5193,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5195,"num_write_kb":20764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":856},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5016","reported_seq":5092,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.404563+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:19.404563+0000","last_peered":"2024-09-16T10:24:19.404563+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:19.404563+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20530118,"num_objects":5014,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5014,"num_write_kb":20050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":847},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5254","reported_seq":5341,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.348017+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:19.348017+0000","last_peered":"2024-09-16T10:24:19.348017+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:19.348017+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5254,"log_dups_size":0,"ondisk_log_size":5254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21473164,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":20975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":851},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":334893558,"num_objects":81777,"num_object_clones":0,"num_object_copies":163554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":81777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59406,"num_write_kb":237512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":644,"num_bytes_recovered":2564579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":668110848,"data_stored":668020523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81841,"ondisk_log_size":81841,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018902,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458472,"kb_used_data":300512,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":93913368,"statfs":{"total":96636764160,"available":96167288832,"internally_reserved":0,"allocated":307724288,"data_stored":307481049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":161738416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":5.6580000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7400000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691846,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117960,"kb_used_data":37952,"kb_used_omap":4,"kb_used_meta":79995,"kb_avail":94253880,"statfs":{"total":96636764160,"available":96515973120,"internally_reserved":0,"allocated":38862848,"data_stored":38540123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":81915562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.226}]}]},{"osd":0,"up_from":24,"seq":103079215130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474372,"kb_used_data":312380,"kb_used_omap":4,"kb_used_meta":161979,"kb_avail":93897468,"statfs":{"total":96636764160,"available":96151007232,"internally_reserved":0,"allocated":319877120,"data_stored":319637495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":165867176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,4,3,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320843776,"data_stored":320840163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39415808,"data_stored":39332709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307851264,"data_stored":307847651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:21.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:21.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'5034", 'reported_seq': 5099, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:19.345564+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:19.345564+0000', 'last_peered': '2024-09-16T10:24:19.345564+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:19.345564+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5034, 'log_dups_size': 0, 'ondisk_log_size': 5034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20619264, 'num_objects': 5034, 'num_object_clones': 0, 'num_object_copies': 10068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5034, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5034, 'num_write_kb': 20136, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 842}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:21.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:21.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'5207", 'reported_seq': 5388, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:20.337057+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:20.337057+0000', 'last_peered': '2024-09-16T10:24:20.337057+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:20.337057+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 5207, 'log_dups_size': 0, 'ondisk_log_size': 5207, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21311488, 'num_objects': 5203, 'num_object_clones': 0, 'num_object_copies': 10406, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5203, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1264, 'num_write_kb': 5056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 868}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:21.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:21.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'5039", 'reported_seq': 5199, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:19.393010+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:19.393010+0000', 'last_peered': '2024-09-16T10:24:19.393010+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:19.393010+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 5039, 'log_dups_size': 0, 'ondisk_log_size': 5039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20619264, 'num_objects': 5034, 'num_object_clones': 0, 'num_object_copies': 10068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5034, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1664, 'num_write_kb': 6656, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 817}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:21.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:21.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'5243", 'reported_seq': 5413, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:19.377714+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:19.377714+0000', 'last_peered': '2024-09-16T10:24:19.377714+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:19.377714+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 5243, 'log_dups_size': 0, 'ondisk_log_size': 5243, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21450752, 'num_objects': 5237, 'num_object_clones': 0, 'num_object_copies': 10474, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1485, 'num_write_kb': 5940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 834}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:21.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:21.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'5073", 'reported_seq': 6416, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:20.369098+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:20.369098+0000', 'last_peered': '2024-09-16T10:24:20.369098+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:20.369098+0000', 'last_undegraded': '2024-09-16T10:24:20.369098+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5073, 'log_dups_size': 0, 'ondisk_log_size': 5073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20751302, 'num_objects': 5068, 'num_object_clones': 0, 'num_object_copies': 10136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3572, 'num_objects_unfound': 0, 'num_objects_dirty': 5068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5071, 'num_write_kb': 20268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 627, 'num_bytes_recovered': 2560000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3572}, {'shards': '2', 'objects': 1496}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:21.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:21.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'5278", 'reported_seq': 5353, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:19.277045+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:19.277045+0000', 'last_peered': '2024-09-16T10:24:19.277045+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:19.277045+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5278, 'log_dups_size': 0, 'ondisk_log_size': 5278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21586886, 'num_objects': 5272, 'num_object_clones': 0, 'num_object_copies': 10544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5276, 'num_write_kb': 21084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 838}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:21.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:21.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'5097", 'reported_seq': 5194, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:20.286103+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:20.286103+0000', 'last_peered': '2024-09-16T10:24:20.286103+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:20.286103+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 5097, 'log_dups_size': 0, 'ondisk_log_size': 5097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20857798, 'num_objects': 5094, 'num_object_clones': 0, 'num_object_copies': 10188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5095, 'num_write_kb': 20371, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 817}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:21.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:21.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'5028", 'reported_seq': 5110, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:20.338622+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:20.338622+0000', 'last_peered': '2024-09-16T10:24:20.338622+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:20.338622+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5028, 'log_dups_size': 0, 'ondisk_log_size': 5028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20571078, 'num_objects': 5024, 'num_object_clones': 0, 'num_object_copies': 10048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5026, 'num_write_kb': 20091, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 832}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:21.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:21.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'4957", 'reported_seq': 5033, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:19.321319+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:19.321319+0000', 'last_peered': '2024-09-16T10:24:19.321319+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:19.321319+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 4957, 'log_dups_size': 0, 'ondisk_log_size': 4957, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20296163, 'num_objects': 4956, 'num_object_clones': 0, 'num_object_copies': 9912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4956, 'num_write_kb': 19821, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 797}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:21.312 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:21.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'5198", 'reported_seq': 5327, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:19.401393+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:19.401393+0000', 'last_peered': '2024-09-16T10:24:19.401393+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:19.401393+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 5198, 'log_dups_size': 0, 'ondisk_log_size': 5198, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21259689, 'num_objects': 5193, 'num_object_clones': 0, 'num_object_copies': 10386, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5195, 'num_write_kb': 20764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 856}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:21.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:21.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'5016", 'reported_seq': 5092, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:19.404563+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:19.404563+0000', 'last_peered': '2024-09-16T10:24:19.404563+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:19.404563+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5016, 'log_dups_size': 0, 'ondisk_log_size': 5016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20530118, 'num_objects': 5014, 'num_object_clones': 0, 'num_object_copies': 10028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5014, 'num_write_kb': 20050, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 847}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:21.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:21.313 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'5254", 'reported_seq': 5341, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:19.348017+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:19.348017+0000', 'last_peered': '2024-09-16T10:24:19.348017+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:19.348017+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5254, 'log_dups_size': 0, 'ondisk_log_size': 5254, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21473164, 'num_objects': 5246, 'num_object_clones': 0, 'num_object_copies': 10492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5246, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5251, 'num_write_kb': 20975, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 851}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:21.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:21.526 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 69 20 82708 82688 4.68037 2.03906 0.0261558 0.0166822 2024-09-16T10:24:21.975 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:21.975 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/163558 objects degraded (28.814%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":150,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":81779,"data_bytes":334893577,"bytes_used":1076023296,"bytes_avail":288834269184,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":163558,"degraded_ratio":0.2881424326538598,"misplaced_objects":3572,"misplaced_total":163558,"misplaced_ratio":0.021839347509751895,"recovering_objects_per_sec":34,"recovering_bytes_per_sec":139553,"recovering_keys_per_sec":0,"num_objects_recovered":409,"num_bytes_recovered":1675264,"num_keys_recovered":0,"write_bytes_sec":3004656,"read_op_per_sec":0,"write_op_per_sec":733},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T10:23:56.572050+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:24:22.001 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:22.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:22.527 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 70 20 83234 83214 4.64286 2.05469 0.0236661 0.0168192 2024-09-16T10:24:22.567 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:22.569 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:22.594 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-16T10:24:20.579153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334893577,"num_objects":81779,"num_object_clones":0,"num_object_copies":163558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":81779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59408,"num_write_kb":237514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":645,"num_bytes_recovered":2564598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81844,"ondisk_log_size":81844,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1050804,"kb_used_data":650844,"kb_used_omap":12,"kb_used_meta":399923,"kb_avail":282064716,"statfs":{"total":289910292480,"available":288834269184,"internally_reserved":0,"allocated":666464256,"data_stored":665658667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":409521154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,7,5,4,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36069376,"num_objects":8806,"num_object_clones":0,"num_object_copies":17612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-409,"num_objects_unfound":0,"num_objects_dirty":8806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8806,"num_write_kb":35224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":1675264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004493"},"pg_stats":[{"pgid":"2.f","version":"42'5034","reported_seq":5099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.345564+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:19.345564+0000","last_peered":"2024-09-16T10:24:19.345564+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:19.345564+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20619264,"num_objects":5034,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":20136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":842},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5159","reported_seq":5313,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:20.307763+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:20.307763+0000","last_peered":"2024-09-16T10:24:20.307763+0000","last_clean":"2024-09-16T10:24:20.307763+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:20.307763+0000","last_undegraded":"2024-09-16T10:24:20.307763+0000","last_fullsized":"2024-09-16T10:24:20.307763+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21106688,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1372,"num_write_kb":5488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5207","reported_seq":5388,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.337057+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:20.337057+0000","last_peered":"2024-09-16T10:24:20.337057+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:20.337057+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21311488,"num_objects":5203,"num_object_clones":0,"num_object_copies":10406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":868}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5155","reported_seq":5208,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:20.309520+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:20.309520+0000","last_peered":"2024-09-16T10:24:20.309520+0000","last_clean":"2024-09-16T10:24:20.309520+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:20.309520+0000","last_undegraded":"2024-09-16T10:24:20.309520+0000","last_fullsized":"2024-09-16T10:24:20.309520+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21114880,"num_objects":5155,"num_object_clones":0,"num_object_copies":10310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":20620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5052","reported_seq":5205,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:19.393782+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:19.393782+0000","last_peered":"2024-09-16T10:24:19.393782+0000","last_clean":"2024-09-16T10:24:19.393782+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:19.393782+0000","last_undegraded":"2024-09-16T10:24:19.393782+0000","last_fullsized":"2024-09-16T10:24:19.393782+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20664320,"num_objects":5045,"num_object_clones":0,"num_object_copies":10090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1694,"num_write_kb":6776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5039","reported_seq":5199,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.393010+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:19.393010+0000","last_peered":"2024-09-16T10:24:19.393010+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:19.393010+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20619264,"num_objects":5034,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1664,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":817},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5243","reported_seq":5413,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.377714+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:19.377714+0000","last_peered":"2024-09-16T10:24:19.377714+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:19.377714+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5243,"log_dups_size":0,"ondisk_log_size":5243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21450752,"num_objects":5237,"num_object_clones":0,"num_object_copies":10474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1485,"num_write_kb":5940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":834},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5051","reported_seq":5223,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:19.359060+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:19.359060+0000","last_peered":"2024-09-16T10:24:19.359060+0000","last_clean":"2024-09-16T10:24:19.359060+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:19.359060+0000","last_undegraded":"2024-09-16T10:24:19.359060+0000","last_fullsized":"2024-09-16T10:24:19.359060+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5051,"log_dups_size":0,"ondisk_log_size":5051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20680704,"num_objects":5049,"num_object_clones":0,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":854,"num_write_kb":3416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5073","reported_seq":6416,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:20.369098+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:20.369098+0000","last_peered":"2024-09-16T10:24:20.369098+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:20.369098+0000","last_undegraded":"2024-09-16T10:24:20.369098+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20751302,"num_objects":5068,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5071,"num_write_kb":20268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":2560000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3572},{"shards":"2","objects":1496}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5278","reported_seq":5353,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.277045+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:19.277045+0000","last_peered":"2024-09-16T10:24:19.277045+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:19.277045+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5278,"log_dups_size":0,"ondisk_log_size":5278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21586886,"num_objects":5272,"num_object_clones":0,"num_object_copies":10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5276,"num_write_kb":21084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":838},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5097","reported_seq":5194,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.286103+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:20.286103+0000","last_peered":"2024-09-16T10:24:20.286103+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:20.286103+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20857798,"num_objects":5094,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5095,"num_write_kb":20371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5028","reported_seq":5110,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.338622+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:20.338622+0000","last_peered":"2024-09-16T10:24:20.338622+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:20.338622+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5028,"log_dups_size":0,"ondisk_log_size":5028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20571078,"num_objects":5024,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5026,"num_write_kb":20091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'4957","reported_seq":5033,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.321319+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:19.321319+0000","last_peered":"2024-09-16T10:24:19.321319+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:19.321319+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":4957,"log_dups_size":0,"ondisk_log_size":4957,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20296163,"num_objects":4956,"num_object_clones":0,"num_object_copies":9912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":19821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":797},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5198","reported_seq":5327,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.401393+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:19.401393+0000","last_peered":"2024-09-16T10:24:19.401393+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:19.401393+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21259689,"num_objects":5193,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5195,"num_write_kb":20764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":856},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5016","reported_seq":5092,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.404563+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:19.404563+0000","last_peered":"2024-09-16T10:24:19.404563+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:19.404563+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20530118,"num_objects":5014,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5014,"num_write_kb":20050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":847},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5254","reported_seq":5341,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:19.348017+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:19.348017+0000","last_peered":"2024-09-16T10:24:19.348017+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:19.348017+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5254,"log_dups_size":0,"ondisk_log_size":5254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21473164,"num_objects":5246,"num_object_clones":0,"num_object_copies":10492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5251,"num_write_kb":20975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":851},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":334893558,"num_objects":81777,"num_object_clones":0,"num_object_copies":163554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":81777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59406,"num_write_kb":237512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":644,"num_bytes_recovered":2564579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":668110848,"data_stored":668020523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81841,"ondisk_log_size":81841,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018902,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458472,"kb_used_data":300512,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":93913368,"statfs":{"total":96636764160,"available":96167288832,"internally_reserved":0,"allocated":307724288,"data_stored":307481049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":161738416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":5.6580000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7400000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691846,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117960,"kb_used_data":37952,"kb_used_omap":4,"kb_used_meta":79995,"kb_avail":94253880,"statfs":{"total":96636764160,"available":96515973120,"internally_reserved":0,"allocated":38862848,"data_stored":38540123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":81915562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.226}]}]},{"osd":0,"up_from":24,"seq":103079215130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474372,"kb_used_data":312380,"kb_used_omap":4,"kb_used_meta":161979,"kb_avail":93897468,"statfs":{"total":96636764160,"available":96151007232,"internally_reserved":0,"allocated":319877120,"data_stored":319637495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":165867176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,4,3,2,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":320843776,"data_stored":320840163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39415808,"data_stored":39332709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307851264,"data_stored":307847651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:23.526 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 71 20 83839 83819 4.61075 2.36328 0.00946776 0.016936 2024-09-16T10:24:24.526 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 72 20 84427 84407 4.57861 2.29688 0.0193281 0.017055 2024-09-16T10:24:25.526 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 73 20 85019 84999 4.54757 2.3125 0.0195715 0.0171707 2024-09-16T10:24:25.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:26.153 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:26.155 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:26.177 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-16T10:24:24.581658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":342274569,"num_objects":83581,"num_object_clones":0,"num_object_copies":167162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":83581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61210,"num_write_kb":244722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":645,"num_bytes_recovered":2564598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83646,"ondisk_log_size":83646,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1082220,"kb_used_data":668308,"kb_used_omap":12,"kb_used_meta":413875,"kb_avail":282033300,"statfs":{"total":289910292480,"available":288802099200,"internally_reserved":0,"allocated":684347392,"data_stored":683541803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":423808002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,3,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31133696,"num_objects":7601,"num_object_clones":0,"num_object_copies":15202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-273,"num_objects_unfound":0,"num_objects_dirty":7601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7601,"num_write_kb":30404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":1118208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006073"},"pg_stats":[{"pgid":"2.f","version":"42'5230","reported_seq":5295,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.443938+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:24.443938+0000","last_peered":"2024-09-16T10:24:24.443938+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:24.443938+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21422080,"num_objects":5230,"num_object_clones":0,"num_object_copies":10460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5230,"num_write_kb":20920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1038},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5159","reported_seq":5313,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:20.307763+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:20.307763+0000","last_peered":"2024-09-16T10:24:20.307763+0000","last_clean":"2024-09-16T10:24:20.307763+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:20.307763+0000","last_undegraded":"2024-09-16T10:24:20.307763+0000","last_fullsized":"2024-09-16T10:24:20.307763+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21106688,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1372,"num_write_kb":5488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5207","reported_seq":5388,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.337057+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:20.337057+0000","last_peered":"2024-09-16T10:24:20.337057+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:20.337057+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5207,"log_dups_size":0,"ondisk_log_size":5207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21311488,"num_objects":5203,"num_object_clones":0,"num_object_copies":10406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":868}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5155","reported_seq":5208,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:20.309520+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:20.309520+0000","last_peered":"2024-09-16T10:24:20.309520+0000","last_clean":"2024-09-16T10:24:20.309520+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:20.309520+0000","last_undegraded":"2024-09-16T10:24:20.309520+0000","last_fullsized":"2024-09-16T10:24:20.309520+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5155,"log_dups_size":0,"ondisk_log_size":5155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21114880,"num_objects":5155,"num_object_clones":0,"num_object_copies":10310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5155,"num_write_kb":20620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5245","reported_seq":5398,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:24.368274+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:24.368274+0000","last_peered":"2024-09-16T10:24:24.368274+0000","last_clean":"2024-09-16T10:24:24.368274+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:24.368274+0000","last_undegraded":"2024-09-16T10:24:24.368274+0000","last_fullsized":"2024-09-16T10:24:24.368274+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21454848,"num_objects":5238,"num_object_clones":0,"num_object_copies":10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1887,"num_write_kb":7548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5212","reported_seq":5372,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.422542+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:24.422542+0000","last_peered":"2024-09-16T10:24:24.422542+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:24.422542+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5212,"log_dups_size":0,"ondisk_log_size":5212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21327872,"num_objects":5207,"num_object_clones":0,"num_object_copies":10414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1837,"num_write_kb":7348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":990},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5409","reported_seq":5579,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.390784+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:24.390784+0000","last_peered":"2024-09-16T10:24:24.390784+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:24.390784+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5409,"log_dups_size":0,"ondisk_log_size":5409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22130688,"num_objects":5403,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1651,"num_write_kb":6604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1000},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5230","reported_seq":5402,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:24.377483+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:24.377483+0000","last_peered":"2024-09-16T10:24:24.377483+0000","last_clean":"2024-09-16T10:24:24.377483+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:24.377483+0000","last_undegraded":"2024-09-16T10:24:24.377483+0000","last_fullsized":"2024-09-16T10:24:24.377483+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21413888,"num_objects":5228,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1033,"num_write_kb":4132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5073","reported_seq":6416,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:20.369098+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:20.369098+0000","last_peered":"2024-09-16T10:24:20.369098+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:20.369098+0000","last_undegraded":"2024-09-16T10:24:20.369098+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20751302,"num_objects":5068,"num_object_clones":0,"num_object_copies":10136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5071,"num_write_kb":20268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":2560000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3572},{"shards":"2","objects":1496}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5470","reported_seq":5545,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.419634+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:24.419634+0000","last_peered":"2024-09-16T10:24:24.419634+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:24.419634+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22373318,"num_objects":5464,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5468,"num_write_kb":21852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1030},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5097","reported_seq":5194,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.286103+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:20.286103+0000","last_peered":"2024-09-16T10:24:20.286103+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:20.286103+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20857798,"num_objects":5094,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5095,"num_write_kb":20371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5028","reported_seq":5110,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:20.338622+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:20.338622+0000","last_peered":"2024-09-16T10:24:20.338622+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:20.338622+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5028,"log_dups_size":0,"ondisk_log_size":5028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20571078,"num_objects":5024,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5026,"num_write_kb":20091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":832}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5113","reported_seq":5188,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.382139+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:24.382139+0000","last_peered":"2024-09-16T10:24:24.382139+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:24.382139+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20935139,"num_objects":5112,"num_object_clones":0,"num_object_copies":10224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":20445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":953},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5384","reported_seq":5514,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.430127+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:24.430127+0000","last_peered":"2024-09-16T10:24:24.430127+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:24.430127+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22021545,"num_objects":5379,"num_object_clones":0,"num_object_copies":10758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":21508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1042},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5185","reported_seq":5262,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.366346+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:24.366346+0000","last_peered":"2024-09-16T10:24:24.366346+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:24.366346+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21222342,"num_objects":5183,"num_object_clones":0,"num_object_copies":10366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5183,"num_write_kb":20726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1016},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5446","reported_seq":5534,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.432355+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:24.432355+0000","last_peered":"2024-09-16T10:24:24.432355+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:24.432355+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5446,"log_dups_size":0,"ondisk_log_size":5446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22259596,"num_objects":5438,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":21743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1043},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":342274550,"num_objects":83579,"num_object_clones":0,"num_object_copies":167158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3572,"num_objects_unfound":0,"num_objects_dirty":83579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61208,"num_write_kb":244720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":644,"num_bytes_recovered":2564579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":686256128,"data_stored":686165803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83643,"ondisk_log_size":83643,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018902,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458472,"kb_used_data":300512,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":93913368,"statfs":{"total":96636764160,"available":96167288832,"internally_reserved":0,"allocated":307724288,"data_stored":307481049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":161738416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":5.6580000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.7400000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691847,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114924,"kb_used_data":47204,"kb_used_omap":4,"kb_used_meta":67707,"kb_avail":94256916,"statfs":{"total":96636764160,"available":96519081984,"internally_reserved":0,"allocated":48336896,"data_stored":48014171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":69332650},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6760000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215131,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508824,"kb_used_data":320592,"kb_used_omap":4,"kb_used_meta":188219,"kb_avail":93863016,"statfs":{"total":96636764160,"available":96115728384,"internally_reserved":0,"allocated":328286208,"data_stored":328046583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":192736936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5350000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":6.9080000000000004},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":3.5710000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329588736,"data_stored":329585123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":48816128,"data_stored":48733029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307851264,"data_stored":307847651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:26.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:26.526 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 74 20 85546 85526 4.51394 2.05859 0.0654078 0.0172997 2024-09-16T10:24:26.718 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:26.719 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:26.745 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-16T10:24:26.582112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":346382857,"num_objects":84584,"num_object_clones":0,"num_object_copies":169168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3421,"num_objects_unfound":0,"num_objects_dirty":84584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62213,"num_write_kb":248734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":3183094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84649,"ondisk_log_size":84649,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1091876,"kb_used_data":673868,"kb_used_omap":12,"kb_used_meta":417971,"kb_avail":282023644,"statfs":{"total":289910292480,"available":288792211456,"internally_reserved":0,"allocated":690040832,"data_stored":689235243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":428002306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,4,12,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27725824,"num_objects":6769,"num_object_clones":0,"num_object_copies":13538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-424,"num_objects_unfound":0,"num_objects_dirty":6769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6769,"num_write_kb":27076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":424,"num_bytes_recovered":1736704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004625"},"pg_stats":[{"pgid":"2.f","version":"42'5230","reported_seq":5295,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.443938+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:24.443938+0000","last_peered":"2024-09-16T10:24:24.443938+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:24.443938+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21422080,"num_objects":5230,"num_object_clones":0,"num_object_copies":10460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5230,"num_write_kb":20920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1038},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5320","reported_seq":5474,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:25.359642+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:25.359642+0000","last_peered":"2024-09-16T10:24:25.359642+0000","last_clean":"2024-09-16T10:24:25.359642+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:25.359642+0000","last_undegraded":"2024-09-16T10:24:25.359642+0000","last_fullsized":"2024-09-16T10:24:25.359642+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5320,"log_dups_size":0,"ondisk_log_size":5320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21766144,"num_objects":5314,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1533,"num_write_kb":6132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5375","reported_seq":5557,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:25.416374+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:25.416374+0000","last_peered":"2024-09-16T10:24:25.416374+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:25.416374+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5375,"log_dups_size":0,"ondisk_log_size":5375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21999616,"num_objects":5371,"num_object_clones":0,"num_object_copies":10742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1432,"num_write_kb":5728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1036}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5327","reported_seq":5380,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:25.373100+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:25.373100+0000","last_peered":"2024-09-16T10:24:25.373100+0000","last_clean":"2024-09-16T10:24:25.373100+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:25.373100+0000","last_undegraded":"2024-09-16T10:24:25.373100+0000","last_fullsized":"2024-09-16T10:24:25.373100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5327,"log_dups_size":0,"ondisk_log_size":5327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21819392,"num_objects":5327,"num_object_clones":0,"num_object_copies":10654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5327,"num_write_kb":21308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5245","reported_seq":5398,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:24.368274+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:24.368274+0000","last_peered":"2024-09-16T10:24:24.368274+0000","last_clean":"2024-09-16T10:24:24.368274+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:24.368274+0000","last_undegraded":"2024-09-16T10:24:24.368274+0000","last_fullsized":"2024-09-16T10:24:24.368274+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21454848,"num_objects":5238,"num_object_clones":0,"num_object_copies":10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1887,"num_write_kb":7548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5212","reported_seq":5372,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.422542+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:24.422542+0000","last_peered":"2024-09-16T10:24:24.422542+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:24.422542+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5212,"log_dups_size":0,"ondisk_log_size":5212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21327872,"num_objects":5207,"num_object_clones":0,"num_object_copies":10414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1837,"num_write_kb":7348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":990},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5409","reported_seq":5579,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.390784+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:24.390784+0000","last_peered":"2024-09-16T10:24:24.390784+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:24.390784+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5409,"log_dups_size":0,"ondisk_log_size":5409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22130688,"num_objects":5403,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1651,"num_write_kb":6604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1000},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5230","reported_seq":5402,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:24.377483+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:24.377483+0000","last_peered":"2024-09-16T10:24:24.377483+0000","last_clean":"2024-09-16T10:24:24.377483+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:24.377483+0000","last_undegraded":"2024-09-16T10:24:24.377483+0000","last_fullsized":"2024-09-16T10:24:24.377483+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21413888,"num_objects":5228,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1033,"num_write_kb":4132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5221","reported_seq":6865,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:25.405868+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:25.405868+0000","last_peered":"2024-09-16T10:24:25.405868+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:25.405868+0000","last_undegraded":"2024-09-16T10:24:25.405868+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21357510,"num_objects":5216,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3421,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5219,"num_write_kb":20860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":3178496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3421},{"shards":"2","objects":1795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5470","reported_seq":5545,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.419634+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:24.419634+0000","last_peered":"2024-09-16T10:24:24.419634+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:24.419634+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22373318,"num_objects":5464,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5468,"num_write_kb":21852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1030},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5255","reported_seq":5357,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:25.363323+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:25.363323+0000","last_peered":"2024-09-16T10:24:25.363323+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:25.363323+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5255,"log_dups_size":0,"ondisk_log_size":5255,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21504966,"num_objects":5252,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5253,"num_write_kb":21003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":975}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5224","reported_seq":5307,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:25.352021+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:25.352021+0000","last_peered":"2024-09-16T10:24:25.352021+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:25.352021+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21373894,"num_objects":5220,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":20875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1028}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5113","reported_seq":5188,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.382139+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:24.382139+0000","last_peered":"2024-09-16T10:24:24.382139+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:24.382139+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20935139,"num_objects":5112,"num_object_clones":0,"num_object_copies":10224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":20445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":953},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5384","reported_seq":5514,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.430127+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:24.430127+0000","last_peered":"2024-09-16T10:24:24.430127+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:24.430127+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22021545,"num_objects":5379,"num_object_clones":0,"num_object_copies":10758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":21508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1042},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5185","reported_seq":5262,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.366346+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:24.366346+0000","last_peered":"2024-09-16T10:24:24.366346+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:24.366346+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21222342,"num_objects":5183,"num_object_clones":0,"num_object_copies":10366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5183,"num_write_kb":20726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1016},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5446","reported_seq":5534,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.432355+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:24.432355+0000","last_peered":"2024-09-16T10:24:24.432355+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:24.432355+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5446,"log_dups_size":0,"ondisk_log_size":5446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22259596,"num_objects":5438,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":21743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1043},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":346382838,"num_objects":84582,"num_object_clones":0,"num_object_copies":169164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3421,"num_objects_unfound":0,"num_objects_dirty":84582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62211,"num_write_kb":248732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":795,"num_bytes_recovered":3183075,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":691953664,"data_stored":691863339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84646,"ondisk_log_size":84646,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018903,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468128,"kb_used_data":306072,"kb_used_omap":4,"kb_used_meta":162043,"kb_avail":93903712,"statfs":{"total":96636764160,"available":96157401088,"internally_reserved":0,"allocated":313417728,"data_stored":313174489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":165932720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":4.0720000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9569999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691847,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114924,"kb_used_data":47204,"kb_used_omap":4,"kb_used_meta":67707,"kb_avail":94256916,"statfs":{"total":96636764160,"available":96519081984,"internally_reserved":0,"allocated":48336896,"data_stored":48014171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":69332650},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6760000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215131,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508824,"kb_used_data":320592,"kb_used_omap":4,"kb_used_meta":188219,"kb_avail":93863016,"statfs":{"total":96636764160,"available":96115728384,"internally_reserved":0,"allocated":328286208,"data_stored":328046583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":192736936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5350000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":6.9080000000000004},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":3.5710000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329588736,"data_stored":329585123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":48816128,"data_stored":48733029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313548800,"data_stored":313545187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:26.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:26.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'5230", 'reported_seq': 5295, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:24.443938+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:24.443938+0000', 'last_peered': '2024-09-16T10:24:24.443938+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:24.443938+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5230, 'log_dups_size': 0, 'ondisk_log_size': 5230, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21422080, 'num_objects': 5230, 'num_object_clones': 0, 'num_object_copies': 10460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5230, 'num_write_kb': 20920, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1038}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:26.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:26.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'5375", 'reported_seq': 5557, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:25.416374+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:25.416374+0000', 'last_peered': '2024-09-16T10:24:25.416374+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:25.416374+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 5375, 'log_dups_size': 0, 'ondisk_log_size': 5375, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21999616, 'num_objects': 5371, 'num_object_clones': 0, 'num_object_copies': 10742, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1432, 'num_write_kb': 5728, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 1036}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:26.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:26.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'5212", 'reported_seq': 5372, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:24.422542+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:24.422542+0000', 'last_peered': '2024-09-16T10:24:24.422542+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:24.422542+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 5212, 'log_dups_size': 0, 'ondisk_log_size': 5212, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21327872, 'num_objects': 5207, 'num_object_clones': 0, 'num_object_copies': 10414, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1837, 'num_write_kb': 7348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 990}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:26.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:26.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'5409", 'reported_seq': 5579, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:24.390784+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:24.390784+0000', 'last_peered': '2024-09-16T10:24:24.390784+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:24.390784+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 5409, 'log_dups_size': 0, 'ondisk_log_size': 5409, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22130688, 'num_objects': 5403, 'num_object_clones': 0, 'num_object_copies': 10806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5403, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1651, 'num_write_kb': 6604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1000}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:26.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:26.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'5221", 'reported_seq': 6865, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:25.405868+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:25.405868+0000', 'last_peered': '2024-09-16T10:24:25.405868+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:25.405868+0000', 'last_undegraded': '2024-09-16T10:24:25.405868+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5221, 'log_dups_size': 0, 'ondisk_log_size': 5221, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21357510, 'num_objects': 5216, 'num_object_clones': 0, 'num_object_copies': 10432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3421, 'num_objects_unfound': 0, 'num_objects_dirty': 5216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5219, 'num_write_kb': 20860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 778, 'num_bytes_recovered': 3178496, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3421}, {'shards': '2', 'objects': 1795}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:26.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:26.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'5470", 'reported_seq': 5545, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:24.419634+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:24.419634+0000', 'last_peered': '2024-09-16T10:24:24.419634+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:24.419634+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5470, 'log_dups_size': 0, 'ondisk_log_size': 5470, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22373318, 'num_objects': 5464, 'num_object_clones': 0, 'num_object_copies': 10928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5464, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5468, 'num_write_kb': 21852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1030}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:26.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:26.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'5255", 'reported_seq': 5357, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:25.363323+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:25.363323+0000', 'last_peered': '2024-09-16T10:24:25.363323+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:25.363323+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 5255, 'log_dups_size': 0, 'ondisk_log_size': 5255, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21504966, 'num_objects': 5252, 'num_object_clones': 0, 'num_object_copies': 10504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5253, 'num_write_kb': 21003, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 975}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:26.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:26.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'5224", 'reported_seq': 5307, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:25.352021+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:25.352021+0000', 'last_peered': '2024-09-16T10:24:25.352021+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:25.352021+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5224, 'log_dups_size': 0, 'ondisk_log_size': 5224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21373894, 'num_objects': 5220, 'num_object_clones': 0, 'num_object_copies': 10440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5222, 'num_write_kb': 20875, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 1028}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:26.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:26.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'5113", 'reported_seq': 5188, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:24.382139+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:24.382139+0000', 'last_peered': '2024-09-16T10:24:24.382139+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:24.382139+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5113, 'log_dups_size': 0, 'ondisk_log_size': 5113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20935139, 'num_objects': 5112, 'num_object_clones': 0, 'num_object_copies': 10224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5112, 'num_write_kb': 20445, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 953}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:26.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:26.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'5384", 'reported_seq': 5514, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:24.430127+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:24.430127+0000', 'last_peered': '2024-09-16T10:24:24.430127+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:24.430127+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 5384, 'log_dups_size': 0, 'ondisk_log_size': 5384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22021545, 'num_objects': 5379, 'num_object_clones': 0, 'num_object_copies': 10758, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5381, 'num_write_kb': 21508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1042}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:26.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:26.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'5185", 'reported_seq': 5262, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:24.366346+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:24.366346+0000', 'last_peered': '2024-09-16T10:24:24.366346+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:24.366346+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5185, 'log_dups_size': 0, 'ondisk_log_size': 5185, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21222342, 'num_objects': 5183, 'num_object_clones': 0, 'num_object_copies': 10366, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5183, 'num_write_kb': 20726, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1016}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:26.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:26.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'5446", 'reported_seq': 5534, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:24.432355+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:24.432355+0000', 'last_peered': '2024-09-16T10:24:24.432355+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:24.432355+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5446, 'log_dups_size': 0, 'ondisk_log_size': 5446, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22259596, 'num_objects': 5438, 'num_object_clones': 0, 'num_object_copies': 10876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5443, 'num_write_kb': 21743, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1043}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:26.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:27.380 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:27.380 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/169168 objects degraded (27.859%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":156,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":84584,"data_bytes":346382857,"bytes_used":1118081024,"bytes_avail":288792211456,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":169168,"degraded_ratio":0.27858696680223211,"misplaced_objects":3421,"misplaced_total":169168,"misplaced_ratio":0.02022250070935401,"recovering_objects_per_sec":35,"recovering_bytes_per_sec":144669,"recovering_keys_per_sec":0,"num_objects_recovered":424,"num_bytes_recovered":1736704,"num_keys_recovered":0,"write_bytes_sec":2309595,"read_op_per_sec":0,"write_op_per_sec":563},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T10:23:56.572050+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:24:27.403 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:27.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:27.527 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 75 20 86032 86012 4.47906 1.89844 0.0480138 0.0174324 2024-09-16T10:24:27.964 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:27.967 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:27.989 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-16T10:24:26.582112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":346382857,"num_objects":84584,"num_object_clones":0,"num_object_copies":169168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3421,"num_objects_unfound":0,"num_objects_dirty":84584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62213,"num_write_kb":248734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":3183094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84649,"ondisk_log_size":84649,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1091876,"kb_used_data":673868,"kb_used_omap":12,"kb_used_meta":417971,"kb_avail":282023644,"statfs":{"total":289910292480,"available":288792211456,"internally_reserved":0,"allocated":690040832,"data_stored":689235243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":428002306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,4,12,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27725824,"num_objects":6769,"num_object_clones":0,"num_object_copies":13538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-424,"num_objects_unfound":0,"num_objects_dirty":6769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6769,"num_write_kb":27076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":424,"num_bytes_recovered":1736704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004625"},"pg_stats":[{"pgid":"2.f","version":"42'5230","reported_seq":5295,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.443938+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:24.443938+0000","last_peered":"2024-09-16T10:24:24.443938+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:24.443938+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21422080,"num_objects":5230,"num_object_clones":0,"num_object_copies":10460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5230,"num_write_kb":20920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1038},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5320","reported_seq":5474,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:25.359642+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:25.359642+0000","last_peered":"2024-09-16T10:24:25.359642+0000","last_clean":"2024-09-16T10:24:25.359642+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:25.359642+0000","last_undegraded":"2024-09-16T10:24:25.359642+0000","last_fullsized":"2024-09-16T10:24:25.359642+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5320,"log_dups_size":0,"ondisk_log_size":5320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21766144,"num_objects":5314,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1533,"num_write_kb":6132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5375","reported_seq":5557,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:25.416374+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:25.416374+0000","last_peered":"2024-09-16T10:24:25.416374+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:25.416374+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5375,"log_dups_size":0,"ondisk_log_size":5375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21999616,"num_objects":5371,"num_object_clones":0,"num_object_copies":10742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1432,"num_write_kb":5728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1036}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5327","reported_seq":5380,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:25.373100+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:25.373100+0000","last_peered":"2024-09-16T10:24:25.373100+0000","last_clean":"2024-09-16T10:24:25.373100+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:25.373100+0000","last_undegraded":"2024-09-16T10:24:25.373100+0000","last_fullsized":"2024-09-16T10:24:25.373100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5327,"log_dups_size":0,"ondisk_log_size":5327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21819392,"num_objects":5327,"num_object_clones":0,"num_object_copies":10654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5327,"num_write_kb":21308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5245","reported_seq":5398,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:24.368274+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:24.368274+0000","last_peered":"2024-09-16T10:24:24.368274+0000","last_clean":"2024-09-16T10:24:24.368274+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:24.368274+0000","last_undegraded":"2024-09-16T10:24:24.368274+0000","last_fullsized":"2024-09-16T10:24:24.368274+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5245,"log_dups_size":0,"ondisk_log_size":5245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21454848,"num_objects":5238,"num_object_clones":0,"num_object_copies":10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1887,"num_write_kb":7548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5212","reported_seq":5372,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.422542+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:24.422542+0000","last_peered":"2024-09-16T10:24:24.422542+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:24.422542+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5212,"log_dups_size":0,"ondisk_log_size":5212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21327872,"num_objects":5207,"num_object_clones":0,"num_object_copies":10414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1837,"num_write_kb":7348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":990},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5409","reported_seq":5579,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.390784+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:24.390784+0000","last_peered":"2024-09-16T10:24:24.390784+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:24.390784+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5409,"log_dups_size":0,"ondisk_log_size":5409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22130688,"num_objects":5403,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1651,"num_write_kb":6604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1000},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5230","reported_seq":5402,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:24.377483+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:24.377483+0000","last_peered":"2024-09-16T10:24:24.377483+0000","last_clean":"2024-09-16T10:24:24.377483+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:24.377483+0000","last_undegraded":"2024-09-16T10:24:24.377483+0000","last_fullsized":"2024-09-16T10:24:24.377483+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21413888,"num_objects":5228,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1033,"num_write_kb":4132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5221","reported_seq":6865,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:25.405868+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:25.405868+0000","last_peered":"2024-09-16T10:24:25.405868+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:25.405868+0000","last_undegraded":"2024-09-16T10:24:25.405868+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21357510,"num_objects":5216,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3421,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5219,"num_write_kb":20860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":3178496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3421},{"shards":"2","objects":1795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5470","reported_seq":5545,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.419634+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:24.419634+0000","last_peered":"2024-09-16T10:24:24.419634+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:24.419634+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22373318,"num_objects":5464,"num_object_clones":0,"num_object_copies":10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5468,"num_write_kb":21852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1030},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5255","reported_seq":5357,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:25.363323+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:25.363323+0000","last_peered":"2024-09-16T10:24:25.363323+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:25.363323+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5255,"log_dups_size":0,"ondisk_log_size":5255,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21504966,"num_objects":5252,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5253,"num_write_kb":21003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":975}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5224","reported_seq":5307,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:25.352021+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:25.352021+0000","last_peered":"2024-09-16T10:24:25.352021+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:25.352021+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5224,"log_dups_size":0,"ondisk_log_size":5224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21373894,"num_objects":5220,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":20875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1028}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5113","reported_seq":5188,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.382139+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:24.382139+0000","last_peered":"2024-09-16T10:24:24.382139+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:24.382139+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20935139,"num_objects":5112,"num_object_clones":0,"num_object_copies":10224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":20445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":953},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5384","reported_seq":5514,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.430127+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:24.430127+0000","last_peered":"2024-09-16T10:24:24.430127+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:24.430127+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22021545,"num_objects":5379,"num_object_clones":0,"num_object_copies":10758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":21508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1042},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5185","reported_seq":5262,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.366346+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:24.366346+0000","last_peered":"2024-09-16T10:24:24.366346+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:24.366346+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21222342,"num_objects":5183,"num_object_clones":0,"num_object_copies":10366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5183,"num_write_kb":20726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1016},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5446","reported_seq":5534,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:24.432355+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:24.432355+0000","last_peered":"2024-09-16T10:24:24.432355+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:24.432355+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5446,"log_dups_size":0,"ondisk_log_size":5446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22259596,"num_objects":5438,"num_object_clones":0,"num_object_copies":10876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":21743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1043},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":346382838,"num_objects":84582,"num_object_clones":0,"num_object_copies":169164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3421,"num_objects_unfound":0,"num_objects_dirty":84582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62211,"num_write_kb":248732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":795,"num_bytes_recovered":3183075,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":691953664,"data_stored":691863339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84646,"ondisk_log_size":84646,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018903,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468128,"kb_used_data":306072,"kb_used_omap":4,"kb_used_meta":162043,"kb_avail":93903712,"statfs":{"total":96636764160,"available":96157401088,"internally_reserved":0,"allocated":313417728,"data_stored":313174489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":165932720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":4.0720000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.9569999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691847,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114924,"kb_used_data":47204,"kb_used_omap":4,"kb_used_meta":67707,"kb_avail":94256916,"statfs":{"total":96636764160,"available":96519081984,"internally_reserved":0,"allocated":48336896,"data_stored":48014171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":69332650},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.7880000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6760000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215131,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":508824,"kb_used_data":320592,"kb_used_omap":4,"kb_used_meta":188219,"kb_avail":93863016,"statfs":{"total":96636764160,"available":96115728384,"internally_reserved":0,"allocated":328286208,"data_stored":328046583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":192736936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.5350000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":6.9080000000000004},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":3.5710000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":329588736,"data_stored":329585123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":48816128,"data_stored":48733029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313548800,"data_stored":313545187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:28.527 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 76 20 86595 86575 4.44906 2.19922 0.0283094 0.0175518 2024-09-16T10:24:29.527 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 77 20 87168 87148 4.42035 2.23828 0.0395215 0.0176659 2024-09-16T10:24:30.528 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 78 20 87738 87718 4.39218 2.22656 0.0354458 0.0177793 2024-09-16T10:24:30.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:31.525 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:31.527 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:31.528 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 79 20 88282 88262 4.36349 2.125 0.0115719 0.0178947 2024-09-16T10:24:31.549 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-16T10:24:30.583056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":357757449,"num_objects":87361,"num_object_clones":0,"num_object_copies":174722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3281,"num_objects_unfound":0,"num_objects_dirty":87361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64990,"num_write_kb":259842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":936,"num_bytes_recovered":3756534,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87426,"ondisk_log_size":87426,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1114792,"kb_used_data":696784,"kb_used_omap":12,"kb_used_meta":417971,"kb_avail":282000728,"statfs":{"total":289910292480,"available":288768745472,"internally_reserved":0,"allocated":713506816,"data_stored":712701227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":428002306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,5,2,14,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34590720,"num_objects":8445,"num_object_clones":0,"num_object_copies":16890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-425,"num_objects_unfound":0,"num_objects_dirty":8445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8445,"num_write_kb":33780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":1740800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004341"},"pg_stats":[{"pgid":"2.f","version":"42'5410","reported_seq":5475,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.500884+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:29.500884+0000","last_peered":"2024-09-16T10:24:29.500884+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:29.500884+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5410,"log_dups_size":0,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22159360,"num_objects":5410,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":21640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1218},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5482","reported_seq":5636,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:30.375217+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:30.375217+0000","last_peered":"2024-09-16T10:24:30.375217+0000","last_clean":"2024-09-16T10:24:30.375217+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:30.375217+0000","last_undegraded":"2024-09-16T10:24:30.375217+0000","last_fullsized":"2024-09-16T10:24:30.375217+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5482,"log_dups_size":0,"ondisk_log_size":5482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22429696,"num_objects":5476,"num_object_clones":0,"num_object_copies":10952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1695,"num_write_kb":6780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5547","reported_seq":5728,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:30.376661+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:30.376661+0000","last_peered":"2024-09-16T10:24:30.376661+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:30.376661+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5547,"log_dups_size":0,"ondisk_log_size":5547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22704128,"num_objects":5543,"num_object_clones":0,"num_object_copies":11086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1604,"num_write_kb":6416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5481","reported_seq":5534,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:30.346082+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:30.346082+0000","last_peered":"2024-09-16T10:24:30.346082+0000","last_clean":"2024-09-16T10:24:30.346082+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:30.346082+0000","last_undegraded":"2024-09-16T10:24:30.346082+0000","last_fullsized":"2024-09-16T10:24:30.346082+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5481,"log_dups_size":0,"ondisk_log_size":5481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22450176,"num_objects":5481,"num_object_clones":0,"num_object_copies":10962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":21924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5437","reported_seq":5590,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:29.456275+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:29.456275+0000","last_peered":"2024-09-16T10:24:29.456275+0000","last_clean":"2024-09-16T10:24:29.456275+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:29.456275+0000","last_undegraded":"2024-09-16T10:24:29.456275+0000","last_fullsized":"2024-09-16T10:24:29.456275+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5437,"log_dups_size":0,"ondisk_log_size":5437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22241280,"num_objects":5430,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2079,"num_write_kb":8316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5390","reported_seq":5548,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.502769+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:29.502769+0000","last_peered":"2024-09-16T10:24:29.502769+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:29.502769+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5390,"log_dups_size":0,"ondisk_log_size":5390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22056960,"num_objects":5385,"num_object_clones":0,"num_object_copies":10770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":8060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1168},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5592","reported_seq":5762,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.426921+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:29.426921+0000","last_peered":"2024-09-16T10:24:29.426921+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:29.426921+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22880256,"num_objects":5586,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1834,"num_write_kb":7336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1183},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5393","reported_seq":5565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:29.467828+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:29.467828+0000","last_peered":"2024-09-16T10:24:29.467828+0000","last_clean":"2024-09-16T10:24:29.467828+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:29.467828+0000","last_undegraded":"2024-09-16T10:24:29.467828+0000","last_fullsized":"2024-09-16T10:24:29.467828+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22081536,"num_objects":5391,"num_object_clones":0,"num_object_copies":10782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1196,"num_write_kb":4784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5389","reported_seq":7314,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:30.413571+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:30.413571+0000","last_peered":"2024-09-16T10:24:30.413571+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:30.413571+0000","last_undegraded":"2024-09-16T10:24:30.413571+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22045638,"num_objects":5384,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3281,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5387,"num_write_kb":21532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":918,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3281},{"shards":"2","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5639","reported_seq":5714,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.480911+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:29.480911+0000","last_peered":"2024-09-16T10:24:29.480911+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:29.480911+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23065542,"num_objects":5633,"num_object_clones":0,"num_object_copies":11266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5637,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1199},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5421","reported_seq":5523,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:30.338457+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:30.338457+0000","last_peered":"2024-09-16T10:24:30.338457+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:30.338457+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22184902,"num_objects":5418,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5419,"num_write_kb":21667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1141}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5433","reported_seq":5516,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:30.372679+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:30.372679+0000","last_peered":"2024-09-16T10:24:30.372679+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:30.372679+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5433,"log_dups_size":0,"ondisk_log_size":5433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22229958,"num_objects":5429,"num_object_clones":0,"num_object_copies":10858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5431,"num_write_kb":21711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1237}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5282","reported_seq":5358,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.462290+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:29.462290+0000","last_peered":"2024-09-16T10:24:29.462290+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:29.462290+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21627363,"num_objects":5281,"num_object_clones":0,"num_object_copies":10562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5281,"num_write_kb":21121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1122},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5544","reported_seq":5674,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.490814+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:29.490814+0000","last_peered":"2024-09-16T10:24:29.490814+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:29.490814+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22676905,"num_objects":5539,"num_object_clones":0,"num_object_copies":11078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5541,"num_write_kb":22148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1202},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5352","reported_seq":5429,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.516800+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:29.516800+0000","last_peered":"2024-09-16T10:24:29.516800+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:29.516800+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21906374,"num_objects":5350,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":21394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1183},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5631","reported_seq":5720,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.505207+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:29.505207+0000","last_peered":"2024-09-16T10:24:29.505207+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:29.505207+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23017356,"num_objects":5623,"num_object_clones":0,"num_object_copies":11246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":22483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1228},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":357757430,"num_objects":87359,"num_object_clones":0,"num_object_copies":174718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3281,"num_objects_unfound":0,"num_objects_dirty":87359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64988,"num_write_kb":259840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":935,"num_bytes_recovered":3756515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715231232,"data_stored":715140907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87423,"ondisk_log_size":87423,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018904,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477752,"kb_used_data":311600,"kb_used_omap":4,"kb_used_meta":166139,"kb_avail":93894088,"statfs":{"total":96636764160,"available":96147546112,"internally_reserved":0,"allocated":319078400,"data_stored":318835161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":170127024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":0.878}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1789999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691848,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132076,"kb_used_data":56164,"kb_used_omap":4,"kb_used_meta":75899,"kb_avail":94239764,"statfs":{"total":96636764160,"available":96501518336,"internally_reserved":0,"allocated":57511936,"data_stored":57189211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":77721258},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6319999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215132,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504964,"kb_used_data":329020,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93866876,"statfs":{"total":96636764160,"available":96119681024,"internally_reserved":0,"allocated":336916480,"data_stored":336676855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":180154024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":338042880,"data_stored":338039267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58073088,"data_stored":57989989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319115264,"data_stored":319111651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:31.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:32.094 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:32.094 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:32.137 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-16T10:24:30.583056+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":357757449,"num_objects":87361,"num_object_clones":0,"num_object_copies":174722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3281,"num_objects_unfound":0,"num_objects_dirty":87361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64990,"num_write_kb":259842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":936,"num_bytes_recovered":3756534,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87426,"ondisk_log_size":87426,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1114792,"kb_used_data":696784,"kb_used_omap":12,"kb_used_meta":417971,"kb_avail":282000728,"statfs":{"total":289910292480,"available":288768745472,"internally_reserved":0,"allocated":713506816,"data_stored":712701227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":428002306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,5,2,14,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34590720,"num_objects":8445,"num_object_clones":0,"num_object_copies":16890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-425,"num_objects_unfound":0,"num_objects_dirty":8445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8445,"num_write_kb":33780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":425,"num_bytes_recovered":1740800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004341"},"pg_stats":[{"pgid":"2.f","version":"42'5410","reported_seq":5475,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.500884+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:29.500884+0000","last_peered":"2024-09-16T10:24:29.500884+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:29.500884+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5410,"log_dups_size":0,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22159360,"num_objects":5410,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":21640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1218},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5482","reported_seq":5636,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:30.375217+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:30.375217+0000","last_peered":"2024-09-16T10:24:30.375217+0000","last_clean":"2024-09-16T10:24:30.375217+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:30.375217+0000","last_undegraded":"2024-09-16T10:24:30.375217+0000","last_fullsized":"2024-09-16T10:24:30.375217+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5482,"log_dups_size":0,"ondisk_log_size":5482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22429696,"num_objects":5476,"num_object_clones":0,"num_object_copies":10952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1695,"num_write_kb":6780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5547","reported_seq":5728,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:30.376661+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:30.376661+0000","last_peered":"2024-09-16T10:24:30.376661+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:30.376661+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5547,"log_dups_size":0,"ondisk_log_size":5547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22704128,"num_objects":5543,"num_object_clones":0,"num_object_copies":11086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1604,"num_write_kb":6416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5481","reported_seq":5534,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:30.346082+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:30.346082+0000","last_peered":"2024-09-16T10:24:30.346082+0000","last_clean":"2024-09-16T10:24:30.346082+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:30.346082+0000","last_undegraded":"2024-09-16T10:24:30.346082+0000","last_fullsized":"2024-09-16T10:24:30.346082+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5481,"log_dups_size":0,"ondisk_log_size":5481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22450176,"num_objects":5481,"num_object_clones":0,"num_object_copies":10962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":21924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5437","reported_seq":5590,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:29.456275+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:29.456275+0000","last_peered":"2024-09-16T10:24:29.456275+0000","last_clean":"2024-09-16T10:24:29.456275+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:29.456275+0000","last_undegraded":"2024-09-16T10:24:29.456275+0000","last_fullsized":"2024-09-16T10:24:29.456275+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5437,"log_dups_size":0,"ondisk_log_size":5437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22241280,"num_objects":5430,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2079,"num_write_kb":8316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5390","reported_seq":5548,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.502769+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:29.502769+0000","last_peered":"2024-09-16T10:24:29.502769+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:29.502769+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5390,"log_dups_size":0,"ondisk_log_size":5390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22056960,"num_objects":5385,"num_object_clones":0,"num_object_copies":10770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":8060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1168},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5592","reported_seq":5762,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.426921+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:29.426921+0000","last_peered":"2024-09-16T10:24:29.426921+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:29.426921+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22880256,"num_objects":5586,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1834,"num_write_kb":7336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1183},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5393","reported_seq":5565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:29.467828+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:29.467828+0000","last_peered":"2024-09-16T10:24:29.467828+0000","last_clean":"2024-09-16T10:24:29.467828+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:29.467828+0000","last_undegraded":"2024-09-16T10:24:29.467828+0000","last_fullsized":"2024-09-16T10:24:29.467828+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22081536,"num_objects":5391,"num_object_clones":0,"num_object_copies":10782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1196,"num_write_kb":4784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5389","reported_seq":7314,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:30.413571+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:30.413571+0000","last_peered":"2024-09-16T10:24:30.413571+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:30.413571+0000","last_undegraded":"2024-09-16T10:24:30.413571+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22045638,"num_objects":5384,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3281,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5387,"num_write_kb":21532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":918,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3281},{"shards":"2","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5639","reported_seq":5714,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.480911+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:29.480911+0000","last_peered":"2024-09-16T10:24:29.480911+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:29.480911+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23065542,"num_objects":5633,"num_object_clones":0,"num_object_copies":11266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5637,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1199},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5421","reported_seq":5523,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:30.338457+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:30.338457+0000","last_peered":"2024-09-16T10:24:30.338457+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:30.338457+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22184902,"num_objects":5418,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5419,"num_write_kb":21667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1141}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5433","reported_seq":5516,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:30.372679+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:30.372679+0000","last_peered":"2024-09-16T10:24:30.372679+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:30.372679+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5433,"log_dups_size":0,"ondisk_log_size":5433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22229958,"num_objects":5429,"num_object_clones":0,"num_object_copies":10858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5431,"num_write_kb":21711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1237}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5282","reported_seq":5358,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.462290+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:29.462290+0000","last_peered":"2024-09-16T10:24:29.462290+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:29.462290+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21627363,"num_objects":5281,"num_object_clones":0,"num_object_copies":10562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5281,"num_write_kb":21121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1122},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5544","reported_seq":5674,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.490814+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:29.490814+0000","last_peered":"2024-09-16T10:24:29.490814+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:29.490814+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22676905,"num_objects":5539,"num_object_clones":0,"num_object_copies":11078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5541,"num_write_kb":22148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1202},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5352","reported_seq":5429,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.516800+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:29.516800+0000","last_peered":"2024-09-16T10:24:29.516800+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:29.516800+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21906374,"num_objects":5350,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":21394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1183},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5631","reported_seq":5720,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.505207+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:29.505207+0000","last_peered":"2024-09-16T10:24:29.505207+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:29.505207+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23017356,"num_objects":5623,"num_object_clones":0,"num_object_copies":11246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":22483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1228},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":357757430,"num_objects":87359,"num_object_clones":0,"num_object_copies":174718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3281,"num_objects_unfound":0,"num_objects_dirty":87359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64988,"num_write_kb":259840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":935,"num_bytes_recovered":3756515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715231232,"data_stored":715140907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87423,"ondisk_log_size":87423,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018904,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477752,"kb_used_data":311600,"kb_used_omap":4,"kb_used_meta":166139,"kb_avail":93894088,"statfs":{"total":96636764160,"available":96147546112,"internally_reserved":0,"allocated":319078400,"data_stored":318835161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":170127024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":0.878}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1789999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691848,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132076,"kb_used_data":56164,"kb_used_omap":4,"kb_used_meta":75899,"kb_avail":94239764,"statfs":{"total":96636764160,"available":96501518336,"internally_reserved":0,"allocated":57511936,"data_stored":57189211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":77721258},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6319999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215132,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504964,"kb_used_data":329020,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93866876,"statfs":{"total":96636764160,"available":96119681024,"internally_reserved":0,"allocated":336916480,"data_stored":336676855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":180154024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":338042880,"data_stored":338039267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58073088,"data_stored":57989989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319115264,"data_stored":319111651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:32.138 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:32.138 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'5410", 'reported_seq': 5475, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:29.500884+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:29.500884+0000', 'last_peered': '2024-09-16T10:24:29.500884+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:29.500884+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5410, 'log_dups_size': 0, 'ondisk_log_size': 5410, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22159360, 'num_objects': 5410, 'num_object_clones': 0, 'num_object_copies': 10820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5410, 'num_write_kb': 21640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1218}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:32.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:32.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'5547", 'reported_seq': 5728, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:30.376661+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:30.376661+0000', 'last_peered': '2024-09-16T10:24:30.376661+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:30.376661+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 5547, 'log_dups_size': 0, 'ondisk_log_size': 5547, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22704128, 'num_objects': 5543, 'num_object_clones': 0, 'num_object_copies': 11086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1604, 'num_write_kb': 6416, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 1208}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:32.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:32.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'5390", 'reported_seq': 5548, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:29.502769+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:29.502769+0000', 'last_peered': '2024-09-16T10:24:29.502769+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:29.502769+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 5390, 'log_dups_size': 0, 'ondisk_log_size': 5390, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22056960, 'num_objects': 5385, 'num_object_clones': 0, 'num_object_copies': 10770, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5385, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2015, 'num_write_kb': 8060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1168}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:32.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:32.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'5592", 'reported_seq': 5762, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:29.426921+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:29.426921+0000', 'last_peered': '2024-09-16T10:24:29.426921+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:29.426921+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 5592, 'log_dups_size': 0, 'ondisk_log_size': 5592, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22880256, 'num_objects': 5586, 'num_object_clones': 0, 'num_object_copies': 11172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1834, 'num_write_kb': 7336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1183}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:32.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:32.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'5389", 'reported_seq': 7314, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:30.413571+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:30.413571+0000', 'last_peered': '2024-09-16T10:24:30.413571+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:30.413571+0000', 'last_undegraded': '2024-09-16T10:24:30.413571+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5389, 'log_dups_size': 0, 'ondisk_log_size': 5389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22045638, 'num_objects': 5384, 'num_object_clones': 0, 'num_object_copies': 10768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3281, 'num_objects_unfound': 0, 'num_objects_dirty': 5384, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5387, 'num_write_kb': 21532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 918, 'num_bytes_recovered': 3751936, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3281}, {'shards': '2', 'objects': 2103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:32.139 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:32.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'5639", 'reported_seq': 5714, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:29.480911+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:29.480911+0000', 'last_peered': '2024-09-16T10:24:29.480911+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:29.480911+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5639, 'log_dups_size': 0, 'ondisk_log_size': 5639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23065542, 'num_objects': 5633, 'num_object_clones': 0, 'num_object_copies': 11266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5637, 'num_write_kb': 22528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1199}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:32.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:32.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'5421", 'reported_seq': 5523, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:30.338457+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:30.338457+0000', 'last_peered': '2024-09-16T10:24:30.338457+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:30.338457+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 5421, 'log_dups_size': 0, 'ondisk_log_size': 5421, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22184902, 'num_objects': 5418, 'num_object_clones': 0, 'num_object_copies': 10836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5419, 'num_write_kb': 21667, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 1141}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:32.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:32.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'5433", 'reported_seq': 5516, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:30.372679+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:30.372679+0000', 'last_peered': '2024-09-16T10:24:30.372679+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:30.372679+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5433, 'log_dups_size': 0, 'ondisk_log_size': 5433, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22229958, 'num_objects': 5429, 'num_object_clones': 0, 'num_object_copies': 10858, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5431, 'num_write_kb': 21711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 1237}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:32.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:32.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'5282", 'reported_seq': 5358, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:29.462290+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:29.462290+0000', 'last_peered': '2024-09-16T10:24:29.462290+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:29.462290+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5282, 'log_dups_size': 0, 'ondisk_log_size': 5282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21627363, 'num_objects': 5281, 'num_object_clones': 0, 'num_object_copies': 10562, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5281, 'num_write_kb': 21121, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1122}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:32.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:32.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'5544", 'reported_seq': 5674, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:29.490814+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:29.490814+0000', 'last_peered': '2024-09-16T10:24:29.490814+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:29.490814+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 5544, 'log_dups_size': 0, 'ondisk_log_size': 5544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22676905, 'num_objects': 5539, 'num_object_clones': 0, 'num_object_copies': 11078, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5541, 'num_write_kb': 22148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1202}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:32.140 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:32.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'5352", 'reported_seq': 5429, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:29.516800+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:29.516800+0000', 'last_peered': '2024-09-16T10:24:29.516800+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:29.516800+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5352, 'log_dups_size': 0, 'ondisk_log_size': 5352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21906374, 'num_objects': 5350, 'num_object_clones': 0, 'num_object_copies': 10700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5350, 'num_write_kb': 21394, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1183}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:32.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:32.141 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'5631", 'reported_seq': 5720, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:29.505207+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:29.505207+0000', 'last_peered': '2024-09-16T10:24:29.505207+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:29.505207+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5631, 'log_dups_size': 0, 'ondisk_log_size': 5631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23017356, 'num_objects': 5623, 'num_object_clones': 0, 'num_object_copies': 11246, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5623, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5628, 'num_write_kb': 22483, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1228}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:32.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:32.528 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:24:32.527488+0000 min lat: 0.00139595 max lat: 2.61294 avg lat: 0.0180177 2024-09-16T10:24:32.528 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:24:32.528 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 80 20 88798 88778 4.33414 2.01562 0.0174524 0.0180177 2024-09-16T10:24:32.821 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:32.821 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/174722 objects degraded (26.973%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":161,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":87361,"data_bytes":357757449,"bytes_used":1141547008,"bytes_avail":288768745472,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":174722,"degraded_ratio":0.26973134465035886,"misplaced_objects":3281,"misplaced_total":174722,"misplaced_ratio":0.018778402261878869,"recovering_objects_per_sec":35,"recovering_bytes_per_sec":145014,"recovering_keys_per_sec":0,"num_objects_recovered":425,"num_bytes_recovered":1740800,"num_keys_recovered":0,"write_bytes_sec":2881517,"read_op_per_sec":0,"write_op_per_sec":703},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T10:23:56.572050+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:24:32.841 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:32.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:33.403 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:33.406 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:33.433 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-16T10:24:32.583534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":357757449,"num_objects":87361,"num_object_clones":0,"num_object_copies":174722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3281,"num_objects_unfound":0,"num_objects_dirty":87361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64990,"num_write_kb":259842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":936,"num_bytes_recovered":3756534,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87426,"ondisk_log_size":87426,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1114792,"kb_used_data":696784,"kb_used_omap":12,"kb_used_meta":417971,"kb_avail":282000728,"statfs":{"total":289910292480,"available":288768745472,"internally_reserved":0,"allocated":713506816,"data_stored":712701227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":428002306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,5,2,14,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22863872,"num_objects":5582,"num_object_clones":0,"num_object_copies":11164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-291,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5582,"num_write_kb":22328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":291,"num_bytes_recovered":1191936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004380"},"pg_stats":[{"pgid":"2.f","version":"42'5410","reported_seq":5475,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.500884+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:29.500884+0000","last_peered":"2024-09-16T10:24:29.500884+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:29.500884+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5410,"log_dups_size":0,"ondisk_log_size":5410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22159360,"num_objects":5410,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5410,"num_write_kb":21640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1218},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5482","reported_seq":5636,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:30.375217+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:30.375217+0000","last_peered":"2024-09-16T10:24:30.375217+0000","last_clean":"2024-09-16T10:24:30.375217+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:30.375217+0000","last_undegraded":"2024-09-16T10:24:30.375217+0000","last_fullsized":"2024-09-16T10:24:30.375217+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5482,"log_dups_size":0,"ondisk_log_size":5482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22429696,"num_objects":5476,"num_object_clones":0,"num_object_copies":10952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1695,"num_write_kb":6780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5547","reported_seq":5728,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:30.376661+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:30.376661+0000","last_peered":"2024-09-16T10:24:30.376661+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:30.376661+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5547,"log_dups_size":0,"ondisk_log_size":5547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22704128,"num_objects":5543,"num_object_clones":0,"num_object_copies":11086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1604,"num_write_kb":6416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5481","reported_seq":5534,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:30.346082+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:30.346082+0000","last_peered":"2024-09-16T10:24:30.346082+0000","last_clean":"2024-09-16T10:24:30.346082+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:30.346082+0000","last_undegraded":"2024-09-16T10:24:30.346082+0000","last_fullsized":"2024-09-16T10:24:30.346082+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5481,"log_dups_size":0,"ondisk_log_size":5481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22450176,"num_objects":5481,"num_object_clones":0,"num_object_copies":10962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5481,"num_write_kb":21924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5437","reported_seq":5590,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:29.456275+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:29.456275+0000","last_peered":"2024-09-16T10:24:29.456275+0000","last_clean":"2024-09-16T10:24:29.456275+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:29.456275+0000","last_undegraded":"2024-09-16T10:24:29.456275+0000","last_fullsized":"2024-09-16T10:24:29.456275+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5437,"log_dups_size":0,"ondisk_log_size":5437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22241280,"num_objects":5430,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2079,"num_write_kb":8316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5390","reported_seq":5548,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.502769+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:29.502769+0000","last_peered":"2024-09-16T10:24:29.502769+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:29.502769+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5390,"log_dups_size":0,"ondisk_log_size":5390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22056960,"num_objects":5385,"num_object_clones":0,"num_object_copies":10770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2015,"num_write_kb":8060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1168},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5592","reported_seq":5762,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.426921+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:29.426921+0000","last_peered":"2024-09-16T10:24:29.426921+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:29.426921+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22880256,"num_objects":5586,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1834,"num_write_kb":7336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1183},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5393","reported_seq":5565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:29.467828+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:29.467828+0000","last_peered":"2024-09-16T10:24:29.467828+0000","last_clean":"2024-09-16T10:24:29.467828+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:29.467828+0000","last_undegraded":"2024-09-16T10:24:29.467828+0000","last_fullsized":"2024-09-16T10:24:29.467828+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22081536,"num_objects":5391,"num_object_clones":0,"num_object_copies":10782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1196,"num_write_kb":4784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5389","reported_seq":7314,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:30.413571+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:30.413571+0000","last_peered":"2024-09-16T10:24:30.413571+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:30.413571+0000","last_undegraded":"2024-09-16T10:24:30.413571+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22045638,"num_objects":5384,"num_object_clones":0,"num_object_copies":10768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3281,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5387,"num_write_kb":21532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":918,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3281},{"shards":"2","objects":2103}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5639","reported_seq":5714,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.480911+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:29.480911+0000","last_peered":"2024-09-16T10:24:29.480911+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:29.480911+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23065542,"num_objects":5633,"num_object_clones":0,"num_object_copies":11266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5637,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1199},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5421","reported_seq":5523,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:30.338457+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:30.338457+0000","last_peered":"2024-09-16T10:24:30.338457+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:30.338457+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22184902,"num_objects":5418,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5419,"num_write_kb":21667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1141}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5433","reported_seq":5516,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:30.372679+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:30.372679+0000","last_peered":"2024-09-16T10:24:30.372679+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:30.372679+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5433,"log_dups_size":0,"ondisk_log_size":5433,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22229958,"num_objects":5429,"num_object_clones":0,"num_object_copies":10858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5431,"num_write_kb":21711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1237}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5282","reported_seq":5358,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.462290+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:29.462290+0000","last_peered":"2024-09-16T10:24:29.462290+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:29.462290+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5282,"log_dups_size":0,"ondisk_log_size":5282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21627363,"num_objects":5281,"num_object_clones":0,"num_object_copies":10562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5281,"num_write_kb":21121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1122},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5544","reported_seq":5674,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.490814+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:29.490814+0000","last_peered":"2024-09-16T10:24:29.490814+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:29.490814+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22676905,"num_objects":5539,"num_object_clones":0,"num_object_copies":11078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5541,"num_write_kb":22148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1202},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5352","reported_seq":5429,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.516800+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:29.516800+0000","last_peered":"2024-09-16T10:24:29.516800+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:29.516800+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21906374,"num_objects":5350,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":21394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1183},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5631","reported_seq":5720,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:29.505207+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:29.505207+0000","last_peered":"2024-09-16T10:24:29.505207+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:29.505207+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23017356,"num_objects":5623,"num_object_clones":0,"num_object_copies":11246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5628,"num_write_kb":22483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1228},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":357757430,"num_objects":87359,"num_object_clones":0,"num_object_copies":174718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3281,"num_objects_unfound":0,"num_objects_dirty":87359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64988,"num_write_kb":259840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":935,"num_bytes_recovered":3756515,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":715231232,"data_stored":715140907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87423,"ondisk_log_size":87423,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018904,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477752,"kb_used_data":311600,"kb_used_omap":4,"kb_used_meta":166139,"kb_avail":93894088,"statfs":{"total":96636764160,"available":96147546112,"internally_reserved":0,"allocated":319078400,"data_stored":318835161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":170127024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":0.878}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.1789999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691848,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132076,"kb_used_data":56164,"kb_used_omap":4,"kb_used_meta":75899,"kb_avail":94239764,"statfs":{"total":96636764160,"available":96501518336,"internally_reserved":0,"allocated":57511936,"data_stored":57189211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":77721258},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6319999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215132,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":504964,"kb_used_data":329020,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93866876,"statfs":{"total":96636764160,"available":96119681024,"internally_reserved":0,"allocated":336916480,"data_stored":336676855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":180154024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":338042880,"data_stored":338039267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58073088,"data_stored":57989989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319115264,"data_stored":319111651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:33.528 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 81 20 89303 89283 4.30498 1.97266 0.0960991 0.018138 2024-09-16T10:24:34.528 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 82 20 89886 89866 4.28025 2.27734 0.0452516 0.0182439 2024-09-16T10:24:35.530 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 83 20 90437 90417 4.25454 2.15234 0.038876 0.0183558 2024-09-16T10:24:36.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:36.530 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 84 20 91025 91005 4.23124 2.29688 0.0239336 0.0184562 2024-09-16T10:24:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:36.988 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:37.021 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-16T10:24:36.584437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":368845321,"num_objects":90068,"num_object_clones":0,"num_object_copies":180136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3129,"num_objects_unfound":0,"num_objects_dirty":90068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67697,"num_write_kb":270670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":4379126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90133,"ondisk_log_size":90133,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1120888,"kb_used_data":719264,"kb_used_omap":12,"kb_used_meta":401587,"kb_avail":281994632,"statfs":{"total":289910292480,"available":288762503168,"internally_reserved":0,"allocated":736526336,"data_stored":735720747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":411225090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,7,11,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26570752,"num_objects":6487,"num_object_clones":0,"num_object_copies":12974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-443,"num_objects_unfound":0,"num_objects_dirty":6487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":25948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":1814528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002779"},"pg_stats":[{"pgid":"2.f","version":"42'5594","reported_seq":5657,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.531880+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:34.531880+0000","last_peered":"2024-09-16T10:24:34.531880+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:34.531880+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22913024,"num_objects":5594,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5594,"num_write_kb":22376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1402},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5650","reported_seq":5804,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:35.373289+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:35.373289+0000","last_peered":"2024-09-16T10:24:35.373289+0000","last_clean":"2024-09-16T10:24:35.373289+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:35.373289+0000","last_undegraded":"2024-09-16T10:24:35.373289+0000","last_fullsized":"2024-09-16T10:24:35.373289+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5650,"log_dups_size":0,"ondisk_log_size":5650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23117824,"num_objects":5644,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1863,"num_write_kb":7452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5701","reported_seq":5883,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:35.244133+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:35.244133+0000","last_peered":"2024-09-16T10:24:35.244133+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:35.244133+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5701,"log_dups_size":0,"ondisk_log_size":5701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23334912,"num_objects":5697,"num_object_clones":0,"num_object_copies":11394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1758,"num_write_kb":7032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1362}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5655","reported_seq":5708,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:35.434030+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:35.434030+0000","last_peered":"2024-09-16T10:24:35.434030+0000","last_clean":"2024-09-16T10:24:35.434030+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:35.434030+0000","last_undegraded":"2024-09-16T10:24:35.434030+0000","last_fullsized":"2024-09-16T10:24:35.434030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23162880,"num_objects":5655,"num_object_clones":0,"num_object_copies":11310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":22620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5626","reported_seq":5779,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:34.507283+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:34.507283+0000","last_peered":"2024-09-16T10:24:34.507283+0000","last_clean":"2024-09-16T10:24:34.507283+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:34.507283+0000","last_undegraded":"2024-09-16T10:24:34.507283+0000","last_fullsized":"2024-09-16T10:24:34.507283+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015424,"num_objects":5619,"num_object_clones":0,"num_object_copies":11238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2268,"num_write_kb":9072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5539","reported_seq":5697,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.442257+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:34.442257+0000","last_peered":"2024-09-16T10:24:34.442257+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:34.442257+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5539,"log_dups_size":0,"ondisk_log_size":5539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22667264,"num_objects":5534,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2164,"num_write_kb":8656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1317},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5788","reported_seq":5958,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.493526+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:34.493526+0000","last_peered":"2024-09-16T10:24:34.493526+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:34.493526+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5788,"log_dups_size":0,"ondisk_log_size":5788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23683072,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":8120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1379},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5560","reported_seq":5732,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:34.375110+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:34.375110+0000","last_peered":"2024-09-16T10:24:34.375110+0000","last_clean":"2024-09-16T10:24:34.375110+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:34.375110+0000","last_undegraded":"2024-09-16T10:24:34.375110+0000","last_fullsized":"2024-09-16T10:24:34.375110+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5560,"log_dups_size":0,"ondisk_log_size":5560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22765568,"num_objects":5558,"num_object_clones":0,"num_object_copies":11116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1363,"num_write_kb":5452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5555","reported_seq":7781,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:35.421632+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:35.421632+0000","last_peered":"2024-09-16T10:24:35.421632+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:35.421632+0000","last_undegraded":"2024-09-16T10:24:35.421632+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22725574,"num_objects":5550,"num_object_clones":0,"num_object_copies":11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3129,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5553,"num_write_kb":22196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":4374528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3129},{"shards":"2","objects":2421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5806","reported_seq":5881,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.442442+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:34.442442+0000","last_peered":"2024-09-16T10:24:34.442442+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:34.442442+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23749574,"num_objects":5800,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":23196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1366},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5601","reported_seq":5703,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:35.433597+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:35.433597+0000","last_peered":"2024-09-16T10:24:35.433597+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:35.433597+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22922182,"num_objects":5598,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5599,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5594","reported_seq":5677,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:35.386986+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:35.386986+0000","last_peered":"2024-09-16T10:24:35.386986+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:35.386986+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22889414,"num_objects":5590,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5592,"num_write_kb":22355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1398}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5442","reported_seq":5517,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.487359+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:34.487359+0000","last_peered":"2024-09-16T10:24:34.487359+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:34.487359+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22282723,"num_objects":5441,"num_object_clones":0,"num_object_copies":10882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":21761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1282},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5698","reported_seq":5828,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.506007+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:34.506007+0000","last_peered":"2024-09-16T10:24:34.506007+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:34.506007+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5698,"log_dups_size":0,"ondisk_log_size":5698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23307689,"num_objects":5693,"num_object_clones":0,"num_object_copies":11386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1356},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5521","reported_seq":5598,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.521933+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:34.521933+0000","last_peered":"2024-09-16T10:24:34.521933+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:34.521933+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5521,"log_dups_size":0,"ondisk_log_size":5521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22598598,"num_objects":5519,"num_object_clones":0,"num_object_copies":11038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":22070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1352},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5800","reported_seq":5889,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.487288+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:34.487288+0000","last_peered":"2024-09-16T10:24:34.487288+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:34.487288+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23709580,"num_objects":5792,"num_object_clones":0,"num_object_copies":11584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5797,"num_write_kb":23159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1397},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":368845302,"num_objects":90066,"num_object_clones":0,"num_object_copies":180132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3129,"num_objects_unfound":0,"num_objects_dirty":90066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67695,"num_write_kb":270668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":4379107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":738340864,"data_stored":738250539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90130,"ondisk_log_size":90130,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018905,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470964,"kb_used_data":317100,"kb_used_omap":4,"kb_used_meta":153851,"kb_avail":93900876,"statfs":{"total":96636764160,"available":96154497024,"internally_reserved":0,"allocated":324710400,"data_stored":324467161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":157544112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":1.458},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":7.976}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691849,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132880,"kb_used_data":65160,"kb_used_omap":4,"kb_used_meta":67707,"kb_avail":94238960,"statfs":{"total":96636764160,"available":96500695040,"internally_reserved":0,"allocated":66723840,"data_stored":66401115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":69332650},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.3159999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.714}]}]},{"osd":0,"up_from":24,"seq":103079215133,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517044,"kb_used_data":337004,"kb_used_omap":4,"kb_used_meta":180027,"kb_avail":93854796,"statfs":{"total":96636764160,"available":96107311104,"internally_reserved":0,"allocated":345092096,"data_stored":344852471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":184348328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7509999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":1.385}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346402816,"data_stored":346399203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67244032,"data_stored":67160933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":324694016,"data_stored":324690403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:37.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:37.530 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 85 20 91531 91511 4.20471 1.97656 0.0411124 0.0185727 2024-09-16T10:24:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:37.563 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:37.592 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-16T10:24:36.584437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":368845321,"num_objects":90068,"num_object_clones":0,"num_object_copies":180136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3129,"num_objects_unfound":0,"num_objects_dirty":90068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67697,"num_write_kb":270670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":4379126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90133,"ondisk_log_size":90133,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1120888,"kb_used_data":719264,"kb_used_omap":12,"kb_used_meta":401587,"kb_avail":281994632,"statfs":{"total":289910292480,"available":288762503168,"internally_reserved":0,"allocated":736526336,"data_stored":735720747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":411225090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,7,11,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26570752,"num_objects":6487,"num_object_clones":0,"num_object_copies":12974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-443,"num_objects_unfound":0,"num_objects_dirty":6487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":25948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":1814528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002779"},"pg_stats":[{"pgid":"2.f","version":"42'5594","reported_seq":5657,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.531880+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:34.531880+0000","last_peered":"2024-09-16T10:24:34.531880+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:34.531880+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22913024,"num_objects":5594,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5594,"num_write_kb":22376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1402},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5650","reported_seq":5804,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:35.373289+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:35.373289+0000","last_peered":"2024-09-16T10:24:35.373289+0000","last_clean":"2024-09-16T10:24:35.373289+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:35.373289+0000","last_undegraded":"2024-09-16T10:24:35.373289+0000","last_fullsized":"2024-09-16T10:24:35.373289+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5650,"log_dups_size":0,"ondisk_log_size":5650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23117824,"num_objects":5644,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1863,"num_write_kb":7452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5701","reported_seq":5883,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:35.244133+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:35.244133+0000","last_peered":"2024-09-16T10:24:35.244133+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:35.244133+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5701,"log_dups_size":0,"ondisk_log_size":5701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23334912,"num_objects":5697,"num_object_clones":0,"num_object_copies":11394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1758,"num_write_kb":7032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1362}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5655","reported_seq":5708,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:35.434030+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:35.434030+0000","last_peered":"2024-09-16T10:24:35.434030+0000","last_clean":"2024-09-16T10:24:35.434030+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:35.434030+0000","last_undegraded":"2024-09-16T10:24:35.434030+0000","last_fullsized":"2024-09-16T10:24:35.434030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23162880,"num_objects":5655,"num_object_clones":0,"num_object_copies":11310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":22620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5626","reported_seq":5779,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:34.507283+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:34.507283+0000","last_peered":"2024-09-16T10:24:34.507283+0000","last_clean":"2024-09-16T10:24:34.507283+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:34.507283+0000","last_undegraded":"2024-09-16T10:24:34.507283+0000","last_fullsized":"2024-09-16T10:24:34.507283+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015424,"num_objects":5619,"num_object_clones":0,"num_object_copies":11238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2268,"num_write_kb":9072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5539","reported_seq":5697,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.442257+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:34.442257+0000","last_peered":"2024-09-16T10:24:34.442257+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:34.442257+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5539,"log_dups_size":0,"ondisk_log_size":5539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22667264,"num_objects":5534,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2164,"num_write_kb":8656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1317},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5788","reported_seq":5958,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.493526+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:34.493526+0000","last_peered":"2024-09-16T10:24:34.493526+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:34.493526+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5788,"log_dups_size":0,"ondisk_log_size":5788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23683072,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":8120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1379},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5560","reported_seq":5732,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:34.375110+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:34.375110+0000","last_peered":"2024-09-16T10:24:34.375110+0000","last_clean":"2024-09-16T10:24:34.375110+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:34.375110+0000","last_undegraded":"2024-09-16T10:24:34.375110+0000","last_fullsized":"2024-09-16T10:24:34.375110+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5560,"log_dups_size":0,"ondisk_log_size":5560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22765568,"num_objects":5558,"num_object_clones":0,"num_object_copies":11116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1363,"num_write_kb":5452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5555","reported_seq":7781,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:35.421632+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:35.421632+0000","last_peered":"2024-09-16T10:24:35.421632+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:35.421632+0000","last_undegraded":"2024-09-16T10:24:35.421632+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22725574,"num_objects":5550,"num_object_clones":0,"num_object_copies":11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3129,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5553,"num_write_kb":22196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":4374528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3129},{"shards":"2","objects":2421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5806","reported_seq":5881,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.442442+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:34.442442+0000","last_peered":"2024-09-16T10:24:34.442442+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:34.442442+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23749574,"num_objects":5800,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":23196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1366},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5601","reported_seq":5703,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:35.433597+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:35.433597+0000","last_peered":"2024-09-16T10:24:35.433597+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:35.433597+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22922182,"num_objects":5598,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5599,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5594","reported_seq":5677,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:35.386986+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:35.386986+0000","last_peered":"2024-09-16T10:24:35.386986+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:35.386986+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22889414,"num_objects":5590,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5592,"num_write_kb":22355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1398}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5442","reported_seq":5517,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.487359+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:34.487359+0000","last_peered":"2024-09-16T10:24:34.487359+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:34.487359+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22282723,"num_objects":5441,"num_object_clones":0,"num_object_copies":10882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":21761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1282},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5698","reported_seq":5828,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.506007+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:34.506007+0000","last_peered":"2024-09-16T10:24:34.506007+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:34.506007+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5698,"log_dups_size":0,"ondisk_log_size":5698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23307689,"num_objects":5693,"num_object_clones":0,"num_object_copies":11386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1356},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5521","reported_seq":5598,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.521933+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:34.521933+0000","last_peered":"2024-09-16T10:24:34.521933+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:34.521933+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5521,"log_dups_size":0,"ondisk_log_size":5521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22598598,"num_objects":5519,"num_object_clones":0,"num_object_copies":11038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":22070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1352},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5800","reported_seq":5889,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.487288+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:34.487288+0000","last_peered":"2024-09-16T10:24:34.487288+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:34.487288+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23709580,"num_objects":5792,"num_object_clones":0,"num_object_copies":11584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5797,"num_write_kb":23159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1397},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":368845302,"num_objects":90066,"num_object_clones":0,"num_object_copies":180132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3129,"num_objects_unfound":0,"num_objects_dirty":90066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67695,"num_write_kb":270668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":4379107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":738340864,"data_stored":738250539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90130,"ondisk_log_size":90130,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018905,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470964,"kb_used_data":317100,"kb_used_omap":4,"kb_used_meta":153851,"kb_avail":93900876,"statfs":{"total":96636764160,"available":96154497024,"internally_reserved":0,"allocated":324710400,"data_stored":324467161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":157544112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":1.458},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":7.976}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691849,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132880,"kb_used_data":65160,"kb_used_omap":4,"kb_used_meta":67707,"kb_avail":94238960,"statfs":{"total":96636764160,"available":96500695040,"internally_reserved":0,"allocated":66723840,"data_stored":66401115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":69332650},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.3159999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.714}]}]},{"osd":0,"up_from":24,"seq":103079215133,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517044,"kb_used_data":337004,"kb_used_omap":4,"kb_used_meta":180027,"kb_avail":93854796,"statfs":{"total":96636764160,"available":96107311104,"internally_reserved":0,"allocated":345092096,"data_stored":344852471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":184348328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7509999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":1.385}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346402816,"data_stored":346399203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67244032,"data_stored":67160933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":324694016,"data_stored":324690403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:37.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:37.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'5594", 'reported_seq': 5657, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:34.531880+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:34.531880+0000', 'last_peered': '2024-09-16T10:24:34.531880+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:34.531880+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5594, 'log_dups_size': 0, 'ondisk_log_size': 5594, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22913024, 'num_objects': 5594, 'num_object_clones': 0, 'num_object_copies': 11188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5594, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5594, 'num_write_kb': 22376, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1402}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:37.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:37.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'5701", 'reported_seq': 5883, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:35.244133+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:35.244133+0000', 'last_peered': '2024-09-16T10:24:35.244133+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:35.244133+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 5701, 'log_dups_size': 0, 'ondisk_log_size': 5701, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23334912, 'num_objects': 5697, 'num_object_clones': 0, 'num_object_copies': 11394, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1758, 'num_write_kb': 7032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 1362}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:37.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:37.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'5539", 'reported_seq': 5697, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:34.442257+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:34.442257+0000', 'last_peered': '2024-09-16T10:24:34.442257+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:34.442257+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 5539, 'log_dups_size': 0, 'ondisk_log_size': 5539, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22667264, 'num_objects': 5534, 'num_object_clones': 0, 'num_object_copies': 11068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2164, 'num_write_kb': 8656, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1317}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:37.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:37.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'5788", 'reported_seq': 5958, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:34.493526+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:34.493526+0000', 'last_peered': '2024-09-16T10:24:34.493526+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:34.493526+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 5788, 'log_dups_size': 0, 'ondisk_log_size': 5788, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23683072, 'num_objects': 5782, 'num_object_clones': 0, 'num_object_copies': 11564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2030, 'num_write_kb': 8120, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1379}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:37.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:37.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'5555", 'reported_seq': 7781, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:35.421632+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:35.421632+0000', 'last_peered': '2024-09-16T10:24:35.421632+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:35.421632+0000', 'last_undegraded': '2024-09-16T10:24:35.421632+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5555, 'log_dups_size': 0, 'ondisk_log_size': 5555, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22725574, 'num_objects': 5550, 'num_object_clones': 0, 'num_object_copies': 11100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3129, 'num_objects_unfound': 0, 'num_objects_dirty': 5550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5553, 'num_write_kb': 22196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1070, 'num_bytes_recovered': 4374528, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3129}, {'shards': '2', 'objects': 2421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:37.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:37.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'5806", 'reported_seq': 5881, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:34.442442+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:34.442442+0000', 'last_peered': '2024-09-16T10:24:34.442442+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:34.442442+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5806, 'log_dups_size': 0, 'ondisk_log_size': 5806, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23749574, 'num_objects': 5800, 'num_object_clones': 0, 'num_object_copies': 11600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5804, 'num_write_kb': 23196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1366}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:37.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:37.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'5601", 'reported_seq': 5703, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:35.433597+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:35.433597+0000', 'last_peered': '2024-09-16T10:24:35.433597+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:35.433597+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 5601, 'log_dups_size': 0, 'ondisk_log_size': 5601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22922182, 'num_objects': 5598, 'num_object_clones': 0, 'num_object_copies': 11196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5599, 'num_write_kb': 22387, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 1321}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:37.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:37.594 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'5594", 'reported_seq': 5677, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:35.386986+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:35.386986+0000', 'last_peered': '2024-09-16T10:24:35.386986+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:35.386986+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5594, 'log_dups_size': 0, 'ondisk_log_size': 5594, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22889414, 'num_objects': 5590, 'num_object_clones': 0, 'num_object_copies': 11180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5592, 'num_write_kb': 22355, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 1398}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:37.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:37.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'5442", 'reported_seq': 5517, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:34.487359+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:34.487359+0000', 'last_peered': '2024-09-16T10:24:34.487359+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:34.487359+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5442, 'log_dups_size': 0, 'ondisk_log_size': 5442, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22282723, 'num_objects': 5441, 'num_object_clones': 0, 'num_object_copies': 10882, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5441, 'num_write_kb': 21761, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1282}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:37.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:37.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'5698", 'reported_seq': 5828, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:34.506007+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:34.506007+0000', 'last_peered': '2024-09-16T10:24:34.506007+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:34.506007+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 5698, 'log_dups_size': 0, 'ondisk_log_size': 5698, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23307689, 'num_objects': 5693, 'num_object_clones': 0, 'num_object_copies': 11386, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5695, 'num_write_kb': 22764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1356}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:37.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:37.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'5521", 'reported_seq': 5598, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:34.521933+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:34.521933+0000', 'last_peered': '2024-09-16T10:24:34.521933+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:34.521933+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5521, 'log_dups_size': 0, 'ondisk_log_size': 5521, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22598598, 'num_objects': 5519, 'num_object_clones': 0, 'num_object_copies': 11038, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5519, 'num_write_kb': 22070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1352}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:37.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:37.595 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'5800", 'reported_seq': 5889, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:34.487288+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:34.487288+0000', 'last_peered': '2024-09-16T10:24:34.487288+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:34.487288+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5800, 'log_dups_size': 0, 'ondisk_log_size': 5800, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23709580, 'num_objects': 5792, 'num_object_clones': 0, 'num_object_copies': 11584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5792, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5797, 'num_write_kb': 23159, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1397}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:37.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:38.273 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:38.274 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/180136 objects degraded (26.162%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":167,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":90068,"data_bytes":368845321,"bytes_used":1147789312,"bytes_avail":288762503168,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":180136,"degraded_ratio":0.2616245503397433,"misplaced_objects":3129,"misplaced_total":180136,"misplaced_ratio":0.01737020917528978,"recovering_objects_per_sec":36,"recovering_bytes_per_sec":151175,"recovering_keys_per_sec":0,"num_objects_recovered":443,"num_bytes_recovered":1814528,"num_keys_recovered":0,"write_bytes_sec":2213716,"read_op_per_sec":0,"write_op_per_sec":540},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T10:23:56.572050+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:24:38.300 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:38.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:38.530 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 86 20 92045 92025 4.17917 2.00781 0.0425084 0.0186861 2024-09-16T10:24:38.843 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:38.843 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:38.871 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-16T10:24:38.584925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":368845321,"num_objects":90068,"num_object_clones":0,"num_object_copies":180136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3129,"num_objects_unfound":0,"num_objects_dirty":90068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67697,"num_write_kb":270670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1088,"num_bytes_recovered":4379126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90133,"ondisk_log_size":90133,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1155880,"kb_used_data":728016,"kb_used_omap":12,"kb_used_meta":427827,"kb_avail":281959640,"statfs":{"total":289910292480,"available":288726671360,"internally_reserved":0,"allocated":745488384,"data_stored":744682795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":438094850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,5,6,10,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22462464,"num_objects":5484,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-292,"num_objects_unfound":0,"num_objects_dirty":5484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5484,"num_write_kb":21936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002812"},"pg_stats":[{"pgid":"2.f","version":"42'5594","reported_seq":5657,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.531880+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:34.531880+0000","last_peered":"2024-09-16T10:24:34.531880+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:34.531880+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22913024,"num_objects":5594,"num_object_clones":0,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5594,"num_write_kb":22376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1402},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5650","reported_seq":5804,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:35.373289+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:35.373289+0000","last_peered":"2024-09-16T10:24:35.373289+0000","last_clean":"2024-09-16T10:24:35.373289+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:35.373289+0000","last_undegraded":"2024-09-16T10:24:35.373289+0000","last_fullsized":"2024-09-16T10:24:35.373289+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5650,"log_dups_size":0,"ondisk_log_size":5650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23117824,"num_objects":5644,"num_object_clones":0,"num_object_copies":11288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1863,"num_write_kb":7452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5701","reported_seq":5883,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:35.244133+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:35.244133+0000","last_peered":"2024-09-16T10:24:35.244133+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:35.244133+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5701,"log_dups_size":0,"ondisk_log_size":5701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23334912,"num_objects":5697,"num_object_clones":0,"num_object_copies":11394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1758,"num_write_kb":7032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1362}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5655","reported_seq":5708,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:35.434030+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:35.434030+0000","last_peered":"2024-09-16T10:24:35.434030+0000","last_clean":"2024-09-16T10:24:35.434030+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:35.434030+0000","last_undegraded":"2024-09-16T10:24:35.434030+0000","last_fullsized":"2024-09-16T10:24:35.434030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23162880,"num_objects":5655,"num_object_clones":0,"num_object_copies":11310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5655,"num_write_kb":22620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5626","reported_seq":5779,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:34.507283+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:34.507283+0000","last_peered":"2024-09-16T10:24:34.507283+0000","last_clean":"2024-09-16T10:24:34.507283+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:34.507283+0000","last_undegraded":"2024-09-16T10:24:34.507283+0000","last_fullsized":"2024-09-16T10:24:34.507283+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015424,"num_objects":5619,"num_object_clones":0,"num_object_copies":11238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2268,"num_write_kb":9072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5539","reported_seq":5697,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.442257+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:34.442257+0000","last_peered":"2024-09-16T10:24:34.442257+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:34.442257+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5539,"log_dups_size":0,"ondisk_log_size":5539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22667264,"num_objects":5534,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2164,"num_write_kb":8656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1317},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5788","reported_seq":5958,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.493526+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:34.493526+0000","last_peered":"2024-09-16T10:24:34.493526+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:34.493526+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5788,"log_dups_size":0,"ondisk_log_size":5788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23683072,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2030,"num_write_kb":8120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1379},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5560","reported_seq":5732,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:34.375110+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:34.375110+0000","last_peered":"2024-09-16T10:24:34.375110+0000","last_clean":"2024-09-16T10:24:34.375110+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:34.375110+0000","last_undegraded":"2024-09-16T10:24:34.375110+0000","last_fullsized":"2024-09-16T10:24:34.375110+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5560,"log_dups_size":0,"ondisk_log_size":5560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22765568,"num_objects":5558,"num_object_clones":0,"num_object_copies":11116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1363,"num_write_kb":5452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5555","reported_seq":7781,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:35.421632+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:35.421632+0000","last_peered":"2024-09-16T10:24:35.421632+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:35.421632+0000","last_undegraded":"2024-09-16T10:24:35.421632+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22725574,"num_objects":5550,"num_object_clones":0,"num_object_copies":11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3129,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5553,"num_write_kb":22196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":4374528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3129},{"shards":"2","objects":2421}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5806","reported_seq":5881,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.442442+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:34.442442+0000","last_peered":"2024-09-16T10:24:34.442442+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:34.442442+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23749574,"num_objects":5800,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5804,"num_write_kb":23196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1366},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":84,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896838+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.896838+0000","last_peered":"2024-09-16T10:23:56.896838+0000","last_clean":"2024-09-16T10:23:56.896838+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.896838+0000","last_undegraded":"2024-09-16T10:23:56.896838+0000","last_fullsized":"2024-09-16T10:23:56.896838+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5601","reported_seq":5703,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:35.433597+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:35.433597+0000","last_peered":"2024-09-16T10:24:35.433597+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:35.433597+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22922182,"num_objects":5598,"num_object_clones":0,"num_object_copies":11196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5599,"num_write_kb":22387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5594","reported_seq":5677,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:35.386986+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:35.386986+0000","last_peered":"2024-09-16T10:24:35.386986+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:35.386986+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22889414,"num_objects":5590,"num_object_clones":0,"num_object_copies":11180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5592,"num_write_kb":22355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1398}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5442","reported_seq":5517,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.487359+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:34.487359+0000","last_peered":"2024-09-16T10:24:34.487359+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:34.487359+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22282723,"num_objects":5441,"num_object_clones":0,"num_object_copies":10882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5441,"num_write_kb":21761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1282},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5698","reported_seq":5828,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.506007+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:34.506007+0000","last_peered":"2024-09-16T10:24:34.506007+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:34.506007+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5698,"log_dups_size":0,"ondisk_log_size":5698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23307689,"num_objects":5693,"num_object_clones":0,"num_object_copies":11386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":22764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1356},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5521","reported_seq":5598,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.521933+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:34.521933+0000","last_peered":"2024-09-16T10:24:34.521933+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:34.521933+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5521,"log_dups_size":0,"ondisk_log_size":5521,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22598598,"num_objects":5519,"num_object_clones":0,"num_object_copies":11038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":22070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1352},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5800","reported_seq":5889,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:34.487288+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:34.487288+0000","last_peered":"2024-09-16T10:24:34.487288+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:34.487288+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23709580,"num_objects":5792,"num_object_clones":0,"num_object_copies":11584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5797,"num_write_kb":23159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1397},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":368845302,"num_objects":90066,"num_object_clones":0,"num_object_copies":180132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":3129,"num_objects_unfound":0,"num_objects_dirty":90066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67695,"num_write_kb":270668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":4379107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":747335680,"data_stored":747245355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90130,"ondisk_log_size":90130,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018905,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470964,"kb_used_data":317100,"kb_used_omap":4,"kb_used_meta":153851,"kb_avail":93900876,"statfs":{"total":96636764160,"available":96154497024,"internally_reserved":0,"allocated":324710400,"data_stored":324467161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":157544112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":1.458},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":7.976}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691850,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":167872,"kb_used_data":73912,"kb_used_omap":4,"kb_used_meta":93947,"kb_avail":94203968,"statfs":{"total":96636764160,"available":96464863232,"internally_reserved":0,"allocated":75685888,"data_stored":75363163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":96202410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215133,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517044,"kb_used_data":337004,"kb_used_omap":4,"kb_used_meta":180027,"kb_avail":93854796,"statfs":{"total":96636764160,"available":96107311104,"internally_reserved":0,"allocated":345092096,"data_stored":344852471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":184348328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7509999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":1.385}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":346402816,"data_stored":346399203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":76238848,"data_stored":76155749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":324694016,"data_stored":324690403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:39.530 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 87 20 92602 92582 4.15614 2.17578 0.046665 0.018791 2024-09-16T10:24:40.530 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 88 20 93169 93149 4.13408 2.21484 0.0185248 0.0188896 2024-09-16T10:24:41.530 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 89 20 93728 93708 4.11216 2.18359 0.0340606 0.0189909 2024-09-16T10:24:41.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:42.443 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:42.443 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:42.472 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-16T10:24:40.585360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":380068361,"num_objects":92808,"num_object_clones":0,"num_object_copies":185616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":92808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70437,"num_write_kb":281630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":4944374,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92873,"ondisk_log_size":92873,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1181928,"kb_used_data":741776,"kb_used_omap":12,"kb_used_meta":440115,"kb_avail":281933592,"statfs":{"total":289910292480,"available":288699998208,"internally_reserved":0,"allocated":759578624,"data_stored":758773035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":450677762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,6,11,17,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33685504,"num_objects":8224,"num_object_clones":0,"num_object_copies":16448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-430,"num_objects_unfound":0,"num_objects_dirty":8224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8224,"num_write_kb":32896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":1761280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002787"},"pg_stats":[{"pgid":"2.f","version":"42'5754","reported_seq":5819,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.557088+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:39.557088+0000","last_peered":"2024-09-16T10:24:39.557088+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:39.557088+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5754,"log_dups_size":0,"ondisk_log_size":5754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23568384,"num_objects":5754,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1562},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5813","reported_seq":5967,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:40.432710+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:40.432710+0000","last_peered":"2024-09-16T10:24:40.432710+0000","last_clean":"2024-09-16T10:24:40.432710+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:40.432710+0000","last_undegraded":"2024-09-16T10:24:40.432710+0000","last_fullsized":"2024-09-16T10:24:40.432710+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23785472,"num_objects":5807,"num_object_clones":0,"num_object_copies":11614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":8104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5883","reported_seq":6063,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:40.476320+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:40.476320+0000","last_peered":"2024-09-16T10:24:40.476320+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:40.476320+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24080384,"num_objects":5879,"num_object_clones":0,"num_object_copies":11758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1940,"num_write_kb":7760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1544}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5828","reported_seq":5881,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:40.472381+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:40.472381+0000","last_peered":"2024-09-16T10:24:40.472381+0000","last_clean":"2024-09-16T10:24:40.472381+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:40.472381+0000","last_undegraded":"2024-09-16T10:24:40.472381+0000","last_fullsized":"2024-09-16T10:24:40.472381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5828,"log_dups_size":0,"ondisk_log_size":5828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23871488,"num_objects":5828,"num_object_clones":0,"num_object_copies":11656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5828,"num_write_kb":23312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5794","reported_seq":5947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:39.511721+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:39.511721+0000","last_peered":"2024-09-16T10:24:39.511721+0000","last_clean":"2024-09-16T10:24:39.511721+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:39.511721+0000","last_undegraded":"2024-09-16T10:24:39.511721+0000","last_fullsized":"2024-09-16T10:24:39.511721+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23703552,"num_objects":5787,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":9744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5712","reported_seq":5871,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.555742+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:39.555742+0000","last_peered":"2024-09-16T10:24:39.555742+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:39.555742+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23375872,"num_objects":5707,"num_object_clones":0,"num_object_copies":11414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":9348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1490},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5959","reported_seq":6128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.547929+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:39.547929+0000","last_peered":"2024-09-16T10:24:39.547929+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:39.547929+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5959,"log_dups_size":0,"ondisk_log_size":5959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24383488,"num_objects":5953,"num_object_clones":0,"num_object_copies":11906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":8804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1550},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5742","reported_seq":5914,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:39.517688+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:39.517688+0000","last_peered":"2024-09-16T10:24:39.517688+0000","last_clean":"2024-09-16T10:24:39.517688+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:39.517688+0000","last_undegraded":"2024-09-16T10:24:39.517688+0000","last_fullsized":"2024-09-16T10:24:39.517688+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23511040,"num_objects":5740,"num_object_clones":0,"num_object_copies":11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1545,"num_write_kb":6180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5724","reported_seq":8228,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:40.437592+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:40.437592+0000","last_peered":"2024-09-16T10:24:40.437592+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:40.437592+0000","last_undegraded":"2024-09-16T10:24:40.437592+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23417798,"num_objects":5719,"num_object_clones":0,"num_object_copies":11438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":5719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5722,"num_write_kb":22872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":4939776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2991},{"shards":"2","objects":2728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5981","reported_seq":6057,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.501172+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:39.501172+0000","last_peered":"2024-09-16T10:24:39.501172+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:39.501172+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5981,"log_dups_size":0,"ondisk_log_size":5981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466374,"num_objects":5975,"num_object_clones":0,"num_object_copies":11950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":23896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1541},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5766","reported_seq":5868,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:40.407512+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:40.407512+0000","last_peered":"2024-09-16T10:24:40.407512+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:40.407512+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23598022,"num_objects":5763,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":23047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1486}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5761","reported_seq":5844,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:40.351416+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:40.351416+0000","last_peered":"2024-09-16T10:24:40.351416+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:40.351416+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5761,"log_dups_size":0,"ondisk_log_size":5761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23573446,"num_objects":5757,"num_object_clones":0,"num_object_copies":11514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":23023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1565}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5615","reported_seq":5690,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.544903+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:39.544903+0000","last_peered":"2024-09-16T10:24:39.544903+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:39.544903+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5615,"log_dups_size":0,"ondisk_log_size":5615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22991331,"num_objects":5614,"num_object_clones":0,"num_object_copies":11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5614,"num_write_kb":22453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1455},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5880","reported_seq":6010,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.547648+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:39.547648+0000","last_peered":"2024-09-16T10:24:39.547648+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:39.547648+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5880,"log_dups_size":0,"ondisk_log_size":5880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24053161,"num_objects":5875,"num_object_clones":0,"num_object_copies":11750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5877,"num_write_kb":23492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1538},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5697","reported_seq":5774,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.544024+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:39.544024+0000","last_peered":"2024-09-16T10:24:39.544024+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:39.544024+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5697,"log_dups_size":0,"ondisk_log_size":5697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23319494,"num_objects":5695,"num_object_clones":0,"num_object_copies":11390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":22774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1528},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5961","reported_seq":6050,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.520960+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:39.520960+0000","last_peered":"2024-09-16T10:24:39.520960+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:39.520960+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24369036,"num_objects":5953,"num_object_clones":0,"num_object_copies":11906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":23803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1558},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":380068342,"num_objects":92806,"num_object_clones":0,"num_object_copies":185612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":92806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70435,"num_write_kb":281628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1225,"num_bytes_recovered":4944355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":761573376,"data_stored":761483051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92870,"ondisk_log_size":92870,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018906,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480432,"kb_used_data":322472,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":93891408,"statfs":{"total":96636764160,"available":96144801792,"internally_reserved":0,"allocated":330211328,"data_stored":329968089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":161738416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,6,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.246},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":6.2809999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.242}]}]},{"osd":1,"up_from":40,"seq":171798691850,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":167872,"kb_used_data":73912,"kb_used_omap":4,"kb_used_meta":93947,"kb_avail":94203968,"statfs":{"total":96636764160,"available":96464863232,"internally_reserved":0,"allocated":75685888,"data_stored":75363163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":96202410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215134,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533624,"kb_used_data":345392,"kb_used_omap":4,"kb_used_meta":188219,"kb_avail":93838216,"statfs":{"total":96636764160,"available":96090333184,"internally_reserved":0,"allocated":353681408,"data_stored":353441783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":192736936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.423},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354910208,"data_stored":354906595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":76238848,"data_stored":76155749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330424320,"data_stored":330420707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:42.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:42.530 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 90 20 94248 94228 4.08904 2.03125 0.0148878 0.0190958 2024-09-16T10:24:43.045 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:43.046 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:43.085 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-16T10:24:42.585826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":380068361,"num_objects":92808,"num_object_clones":0,"num_object_copies":185616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":92808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70437,"num_write_kb":281630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":4944374,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92873,"ondisk_log_size":92873,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1181928,"kb_used_data":741776,"kb_used_omap":12,"kb_used_meta":440115,"kb_avail":281933592,"statfs":{"total":289910292480,"available":288699998208,"internally_reserved":0,"allocated":759578624,"data_stored":758773035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":450677762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,6,11,17,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22310912,"num_objects":5447,"num_object_clones":0,"num_object_copies":10894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-290,"num_objects_unfound":0,"num_objects_dirty":5447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5447,"num_write_kb":21788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002770"},"pg_stats":[{"pgid":"2.f","version":"42'5754","reported_seq":5819,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.557088+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:39.557088+0000","last_peered":"2024-09-16T10:24:39.557088+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:39.557088+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5754,"log_dups_size":0,"ondisk_log_size":5754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23568384,"num_objects":5754,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1562},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5813","reported_seq":5967,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:40.432710+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:40.432710+0000","last_peered":"2024-09-16T10:24:40.432710+0000","last_clean":"2024-09-16T10:24:40.432710+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:40.432710+0000","last_undegraded":"2024-09-16T10:24:40.432710+0000","last_fullsized":"2024-09-16T10:24:40.432710+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23785472,"num_objects":5807,"num_object_clones":0,"num_object_copies":11614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":8104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5883","reported_seq":6063,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:40.476320+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:40.476320+0000","last_peered":"2024-09-16T10:24:40.476320+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:40.476320+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24080384,"num_objects":5879,"num_object_clones":0,"num_object_copies":11758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1940,"num_write_kb":7760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1544}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5828","reported_seq":5881,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:40.472381+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:40.472381+0000","last_peered":"2024-09-16T10:24:40.472381+0000","last_clean":"2024-09-16T10:24:40.472381+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:40.472381+0000","last_undegraded":"2024-09-16T10:24:40.472381+0000","last_fullsized":"2024-09-16T10:24:40.472381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5828,"log_dups_size":0,"ondisk_log_size":5828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23871488,"num_objects":5828,"num_object_clones":0,"num_object_copies":11656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5828,"num_write_kb":23312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5794","reported_seq":5947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:39.511721+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:39.511721+0000","last_peered":"2024-09-16T10:24:39.511721+0000","last_clean":"2024-09-16T10:24:39.511721+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:39.511721+0000","last_undegraded":"2024-09-16T10:24:39.511721+0000","last_fullsized":"2024-09-16T10:24:39.511721+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23703552,"num_objects":5787,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":9744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5712","reported_seq":5871,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.555742+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:39.555742+0000","last_peered":"2024-09-16T10:24:39.555742+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:39.555742+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23375872,"num_objects":5707,"num_object_clones":0,"num_object_copies":11414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":9348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1490},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5959","reported_seq":6128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.547929+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:39.547929+0000","last_peered":"2024-09-16T10:24:39.547929+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:39.547929+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5959,"log_dups_size":0,"ondisk_log_size":5959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24383488,"num_objects":5953,"num_object_clones":0,"num_object_copies":11906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":8804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1550},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5742","reported_seq":5914,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:39.517688+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:39.517688+0000","last_peered":"2024-09-16T10:24:39.517688+0000","last_clean":"2024-09-16T10:24:39.517688+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:39.517688+0000","last_undegraded":"2024-09-16T10:24:39.517688+0000","last_fullsized":"2024-09-16T10:24:39.517688+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23511040,"num_objects":5740,"num_object_clones":0,"num_object_copies":11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1545,"num_write_kb":6180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5724","reported_seq":8228,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:40.437592+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:40.437592+0000","last_peered":"2024-09-16T10:24:40.437592+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:40.437592+0000","last_undegraded":"2024-09-16T10:24:40.437592+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23417798,"num_objects":5719,"num_object_clones":0,"num_object_copies":11438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":5719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5722,"num_write_kb":22872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":4939776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2991},{"shards":"2","objects":2728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5981","reported_seq":6057,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.501172+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:39.501172+0000","last_peered":"2024-09-16T10:24:39.501172+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:39.501172+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5981,"log_dups_size":0,"ondisk_log_size":5981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466374,"num_objects":5975,"num_object_clones":0,"num_object_copies":11950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":23896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1541},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5766","reported_seq":5868,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:40.407512+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:40.407512+0000","last_peered":"2024-09-16T10:24:40.407512+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:40.407512+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23598022,"num_objects":5763,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":23047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1486}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5761","reported_seq":5844,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:40.351416+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:40.351416+0000","last_peered":"2024-09-16T10:24:40.351416+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:40.351416+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5761,"log_dups_size":0,"ondisk_log_size":5761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23573446,"num_objects":5757,"num_object_clones":0,"num_object_copies":11514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":23023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1565}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5615","reported_seq":5690,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.544903+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:39.544903+0000","last_peered":"2024-09-16T10:24:39.544903+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:39.544903+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5615,"log_dups_size":0,"ondisk_log_size":5615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22991331,"num_objects":5614,"num_object_clones":0,"num_object_copies":11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5614,"num_write_kb":22453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1455},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5880","reported_seq":6010,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.547648+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:39.547648+0000","last_peered":"2024-09-16T10:24:39.547648+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:39.547648+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5880,"log_dups_size":0,"ondisk_log_size":5880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24053161,"num_objects":5875,"num_object_clones":0,"num_object_copies":11750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5877,"num_write_kb":23492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1538},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5697","reported_seq":5774,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.544024+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:39.544024+0000","last_peered":"2024-09-16T10:24:39.544024+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:39.544024+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5697,"log_dups_size":0,"ondisk_log_size":5697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23319494,"num_objects":5695,"num_object_clones":0,"num_object_copies":11390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":22774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1528},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5961","reported_seq":6050,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.520960+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:39.520960+0000","last_peered":"2024-09-16T10:24:39.520960+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:39.520960+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24369036,"num_objects":5953,"num_object_clones":0,"num_object_copies":11906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":23803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1558},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":380068342,"num_objects":92806,"num_object_clones":0,"num_object_copies":185612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":92806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70435,"num_write_kb":281628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1225,"num_bytes_recovered":4944355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":761573376,"data_stored":761483051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92870,"ondisk_log_size":92870,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018906,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480432,"kb_used_data":322472,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":93891408,"statfs":{"total":96636764160,"available":96144801792,"internally_reserved":0,"allocated":330211328,"data_stored":329968089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":161738416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,6,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.246},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":6.2809999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.242}]}]},{"osd":1,"up_from":40,"seq":171798691850,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":167872,"kb_used_data":73912,"kb_used_omap":4,"kb_used_meta":93947,"kb_avail":94203968,"statfs":{"total":96636764160,"available":96464863232,"internally_reserved":0,"allocated":75685888,"data_stored":75363163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":96202410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215134,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533624,"kb_used_data":345392,"kb_used_omap":4,"kb_used_meta":188219,"kb_avail":93838216,"statfs":{"total":96636764160,"available":96090333184,"internally_reserved":0,"allocated":353681408,"data_stored":353441783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":192736936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.423},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354910208,"data_stored":354906595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":76238848,"data_stored":76155749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330424320,"data_stored":330420707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:43.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:43.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'5754", 'reported_seq': 5819, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:39.557088+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:39.557088+0000', 'last_peered': '2024-09-16T10:24:39.557088+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:39.557088+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5754, 'log_dups_size': 0, 'ondisk_log_size': 5754, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23568384, 'num_objects': 5754, 'num_object_clones': 0, 'num_object_copies': 11508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5754, 'num_write_kb': 23016, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1562}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:43.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:43.086 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'5883", 'reported_seq': 6063, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:40.476320+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:40.476320+0000', 'last_peered': '2024-09-16T10:24:40.476320+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:40.476320+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 5883, 'log_dups_size': 0, 'ondisk_log_size': 5883, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24080384, 'num_objects': 5879, 'num_object_clones': 0, 'num_object_copies': 11758, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1940, 'num_write_kb': 7760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 1544}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:43.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:43.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'5712", 'reported_seq': 5871, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:39.555742+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:39.555742+0000', 'last_peered': '2024-09-16T10:24:39.555742+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:39.555742+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 5712, 'log_dups_size': 0, 'ondisk_log_size': 5712, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23375872, 'num_objects': 5707, 'num_object_clones': 0, 'num_object_copies': 11414, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2337, 'num_write_kb': 9348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1490}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:43.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:43.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'5959", 'reported_seq': 6128, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:39.547929+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:39.547929+0000', 'last_peered': '2024-09-16T10:24:39.547929+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:39.547929+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 5959, 'log_dups_size': 0, 'ondisk_log_size': 5959, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24383488, 'num_objects': 5953, 'num_object_clones': 0, 'num_object_copies': 11906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2201, 'num_write_kb': 8804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1550}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:43.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:43.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'5724", 'reported_seq': 8228, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:40.437592+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:40.437592+0000', 'last_peered': '2024-09-16T10:24:40.437592+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:40.437592+0000', 'last_undegraded': '2024-09-16T10:24:40.437592+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5724, 'log_dups_size': 0, 'ondisk_log_size': 5724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23417798, 'num_objects': 5719, 'num_object_clones': 0, 'num_object_copies': 11438, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2991, 'num_objects_unfound': 0, 'num_objects_dirty': 5719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5722, 'num_write_kb': 22872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1208, 'num_bytes_recovered': 4939776, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2991}, {'shards': '2', 'objects': 2728}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:43.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:43.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'5981", 'reported_seq': 6057, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:39.501172+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:39.501172+0000', 'last_peered': '2024-09-16T10:24:39.501172+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:39.501172+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5981, 'log_dups_size': 0, 'ondisk_log_size': 5981, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24466374, 'num_objects': 5975, 'num_object_clones': 0, 'num_object_copies': 11950, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5979, 'num_write_kb': 23896, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1541}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'5766", 'reported_seq': 5868, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:40.407512+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:40.407512+0000', 'last_peered': '2024-09-16T10:24:40.407512+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:40.407512+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 5766, 'log_dups_size': 0, 'ondisk_log_size': 5766, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23598022, 'num_objects': 5763, 'num_object_clones': 0, 'num_object_copies': 11526, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5764, 'num_write_kb': 23047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 1486}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'5761", 'reported_seq': 5844, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:40.351416+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:40.351416+0000', 'last_peered': '2024-09-16T10:24:40.351416+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:40.351416+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5761, 'log_dups_size': 0, 'ondisk_log_size': 5761, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23573446, 'num_objects': 5757, 'num_object_clones': 0, 'num_object_copies': 11514, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5759, 'num_write_kb': 23023, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 1565}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'5615", 'reported_seq': 5690, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:39.544903+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:39.544903+0000', 'last_peered': '2024-09-16T10:24:39.544903+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:39.544903+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5615, 'log_dups_size': 0, 'ondisk_log_size': 5615, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22991331, 'num_objects': 5614, 'num_object_clones': 0, 'num_object_copies': 11228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5614, 'num_write_kb': 22453, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1455}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'5880", 'reported_seq': 6010, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:39.547648+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:39.547648+0000', 'last_peered': '2024-09-16T10:24:39.547648+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:39.547648+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 5880, 'log_dups_size': 0, 'ondisk_log_size': 5880, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24053161, 'num_objects': 5875, 'num_object_clones': 0, 'num_object_copies': 11750, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5877, 'num_write_kb': 23492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1538}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:43.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:43.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'5697", 'reported_seq': 5774, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:39.544024+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:39.544024+0000', 'last_peered': '2024-09-16T10:24:39.544024+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:39.544024+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5697, 'log_dups_size': 0, 'ondisk_log_size': 5697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23319494, 'num_objects': 5695, 'num_object_clones': 0, 'num_object_copies': 11390, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5695, 'num_write_kb': 22774, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1528}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:43.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:43.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'5961", 'reported_seq': 6050, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:39.520960+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:39.520960+0000', 'last_peered': '2024-09-16T10:24:39.520960+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:39.520960+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5961, 'log_dups_size': 0, 'ondisk_log_size': 5961, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24369036, 'num_objects': 5953, 'num_object_clones': 0, 'num_object_copies': 11906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5958, 'num_write_kb': 23803, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1558}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:43.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:43.530 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 91 20 94743 94723 4.06535 1.93359 0.0472378 0.0192083 2024-09-16T10:24:43.764 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:43.764 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/185616 objects degraded (25.390%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":172,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":92808,"data_bytes":380068361,"bytes_used":1210294272,"bytes_avail":288699998208,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":185616,"degraded_ratio":0.25390052581673994,"misplaced_objects":2991,"misplaced_total":185616,"misplaced_ratio":0.016113912593741918,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":98963,"recovering_keys_per_sec":0,"num_objects_recovered":290,"num_bytes_recovered":1187840,"num_keys_recovered":0,"write_bytes_sec":1858813,"read_op_per_sec":0,"write_op_per_sec":453},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T10:23:56.572050+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:24:43.787 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:43.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:44.330 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:44.333 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:44.358 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-16T10:24:42.585826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":380068361,"num_objects":92808,"num_object_clones":0,"num_object_copies":185616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":92808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70437,"num_write_kb":281630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":4944374,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92873,"ondisk_log_size":92873,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1181928,"kb_used_data":741776,"kb_used_omap":12,"kb_used_meta":440115,"kb_avail":281933592,"statfs":{"total":289910292480,"available":288699998208,"internally_reserved":0,"allocated":759578624,"data_stored":758773035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":450677762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,6,11,17,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22310912,"num_objects":5447,"num_object_clones":0,"num_object_copies":10894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-290,"num_objects_unfound":0,"num_objects_dirty":5447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5447,"num_write_kb":21788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002770"},"pg_stats":[{"pgid":"2.f","version":"42'5754","reported_seq":5819,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.557088+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:39.557088+0000","last_peered":"2024-09-16T10:24:39.557088+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:39.557088+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5754,"log_dups_size":0,"ondisk_log_size":5754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23568384,"num_objects":5754,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":23016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1562},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5813","reported_seq":5967,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:40.432710+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:40.432710+0000","last_peered":"2024-09-16T10:24:40.432710+0000","last_clean":"2024-09-16T10:24:40.432710+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:40.432710+0000","last_undegraded":"2024-09-16T10:24:40.432710+0000","last_fullsized":"2024-09-16T10:24:40.432710+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23785472,"num_objects":5807,"num_object_clones":0,"num_object_copies":11614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":8104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'5883","reported_seq":6063,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:40.476320+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:40.476320+0000","last_peered":"2024-09-16T10:24:40.476320+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:40.476320+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":5883,"log_dups_size":0,"ondisk_log_size":5883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24080384,"num_objects":5879,"num_object_clones":0,"num_object_copies":11758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1940,"num_write_kb":7760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1544}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5828","reported_seq":5881,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:40.472381+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:40.472381+0000","last_peered":"2024-09-16T10:24:40.472381+0000","last_clean":"2024-09-16T10:24:40.472381+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:40.472381+0000","last_undegraded":"2024-09-16T10:24:40.472381+0000","last_fullsized":"2024-09-16T10:24:40.472381+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5828,"log_dups_size":0,"ondisk_log_size":5828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23871488,"num_objects":5828,"num_object_clones":0,"num_object_copies":11656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5828,"num_write_kb":23312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5794","reported_seq":5947,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:39.511721+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:39.511721+0000","last_peered":"2024-09-16T10:24:39.511721+0000","last_clean":"2024-09-16T10:24:39.511721+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:39.511721+0000","last_undegraded":"2024-09-16T10:24:39.511721+0000","last_fullsized":"2024-09-16T10:24:39.511721+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23703552,"num_objects":5787,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2436,"num_write_kb":9744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5712","reported_seq":5871,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.555742+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:39.555742+0000","last_peered":"2024-09-16T10:24:39.555742+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:39.555742+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5712,"log_dups_size":0,"ondisk_log_size":5712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23375872,"num_objects":5707,"num_object_clones":0,"num_object_copies":11414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2337,"num_write_kb":9348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1490},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'5959","reported_seq":6128,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.547929+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:39.547929+0000","last_peered":"2024-09-16T10:24:39.547929+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:39.547929+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":5959,"log_dups_size":0,"ondisk_log_size":5959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24383488,"num_objects":5953,"num_object_clones":0,"num_object_copies":11906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":8804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1550},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5742","reported_seq":5914,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:39.517688+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:39.517688+0000","last_peered":"2024-09-16T10:24:39.517688+0000","last_clean":"2024-09-16T10:24:39.517688+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:39.517688+0000","last_undegraded":"2024-09-16T10:24:39.517688+0000","last_fullsized":"2024-09-16T10:24:39.517688+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5742,"log_dups_size":0,"ondisk_log_size":5742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23511040,"num_objects":5740,"num_object_clones":0,"num_object_copies":11480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1545,"num_write_kb":6180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5724","reported_seq":8228,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:40.437592+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:40.437592+0000","last_peered":"2024-09-16T10:24:40.437592+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:40.437592+0000","last_undegraded":"2024-09-16T10:24:40.437592+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5724,"log_dups_size":0,"ondisk_log_size":5724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23417798,"num_objects":5719,"num_object_clones":0,"num_object_copies":11438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":5719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5722,"num_write_kb":22872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1208,"num_bytes_recovered":4939776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2991},{"shards":"2","objects":2728}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'5981","reported_seq":6057,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.501172+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:39.501172+0000","last_peered":"2024-09-16T10:24:39.501172+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:39.501172+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5981,"log_dups_size":0,"ondisk_log_size":5981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466374,"num_objects":5975,"num_object_clones":0,"num_object_copies":11950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5979,"num_write_kb":23896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1541},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5766","reported_seq":5868,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:40.407512+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:40.407512+0000","last_peered":"2024-09-16T10:24:40.407512+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:40.407512+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23598022,"num_objects":5763,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":23047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1486}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5761","reported_seq":5844,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:40.351416+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:40.351416+0000","last_peered":"2024-09-16T10:24:40.351416+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:40.351416+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5761,"log_dups_size":0,"ondisk_log_size":5761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23573446,"num_objects":5757,"num_object_clones":0,"num_object_copies":11514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":23023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1565}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5615","reported_seq":5690,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.544903+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:39.544903+0000","last_peered":"2024-09-16T10:24:39.544903+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:39.544903+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5615,"log_dups_size":0,"ondisk_log_size":5615,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22991331,"num_objects":5614,"num_object_clones":0,"num_object_copies":11228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5614,"num_write_kb":22453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1455},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'5880","reported_seq":6010,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.547648+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:39.547648+0000","last_peered":"2024-09-16T10:24:39.547648+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:39.547648+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":5880,"log_dups_size":0,"ondisk_log_size":5880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24053161,"num_objects":5875,"num_object_clones":0,"num_object_copies":11750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5877,"num_write_kb":23492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1538},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5697","reported_seq":5774,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.544024+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:39.544024+0000","last_peered":"2024-09-16T10:24:39.544024+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:39.544024+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5697,"log_dups_size":0,"ondisk_log_size":5697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23319494,"num_objects":5695,"num_object_clones":0,"num_object_copies":11390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5695,"num_write_kb":22774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1528},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'5961","reported_seq":6050,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:39.520960+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:39.520960+0000","last_peered":"2024-09-16T10:24:39.520960+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:39.520960+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5961,"log_dups_size":0,"ondisk_log_size":5961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24369036,"num_objects":5953,"num_object_clones":0,"num_object_copies":11906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5958,"num_write_kb":23803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1558},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":380068342,"num_objects":92806,"num_object_clones":0,"num_object_copies":185612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2991,"num_objects_unfound":0,"num_objects_dirty":92806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70435,"num_write_kb":281628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1225,"num_bytes_recovered":4944355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":761573376,"data_stored":761483051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92870,"ondisk_log_size":92870,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018906,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480432,"kb_used_data":322472,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":93891408,"statfs":{"total":96636764160,"available":96144801792,"internally_reserved":0,"allocated":330211328,"data_stored":329968089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":161738416},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,6,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.246},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":6.2809999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.242}]}]},{"osd":1,"up_from":40,"seq":171798691850,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":167872,"kb_used_data":73912,"kb_used_omap":4,"kb_used_meta":93947,"kb_avail":94203968,"statfs":{"total":96636764160,"available":96464863232,"internally_reserved":0,"allocated":75685888,"data_stored":75363163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":96202410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215134,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533624,"kb_used_data":345392,"kb_used_omap":4,"kb_used_meta":188219,"kb_avail":93838216,"statfs":{"total":96636764160,"available":96090333184,"internally_reserved":0,"allocated":353681408,"data_stored":353441783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":192736936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,7,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.423},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.58099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354910208,"data_stored":354906595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":76238848,"data_stored":76155749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330424320,"data_stored":330420707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:44.530 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 92 20 95256 95236 4.04295 2.00391 0.0169619 0.0193165 2024-09-16T10:24:45.531 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 93 20 95835 95815 4.02379 2.26172 0.0324189 0.0194093 2024-09-16T10:24:46.531 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 94 20 96395 96375 4.00425 2.1875 0.0280848 0.0195025 2024-09-16T10:24:47.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:47.531 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 95 20 96942 96922 3.9846 2.13672 0.0303248 0.0195981 2024-09-16T10:24:47.913 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:47.915 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:47.942 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-16T10:24:46.586697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":391139849,"num_objects":95511,"num_object_clones":0,"num_object_copies":191022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2839,"num_objects_unfound":0,"num_objects_dirty":95511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73140,"num_write_kb":292442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1378,"num_bytes_recovered":5566966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95576,"ondisk_log_size":95576,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1219284,"kb_used_data":763516,"kb_used_omap":12,"kb_used_meta":455731,"kb_avail":281896236,"statfs":{"total":289910292480,"available":288661745664,"internally_reserved":0,"allocated":781840384,"data_stored":781034795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":466668546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,7,9,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26402816,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-442,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6446,"num_write_kb":25784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":442,"num_bytes_recovered":1810432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002742"},"pg_stats":[{"pgid":"2.f","version":"42'5946","reported_seq":6011,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.491935+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:44.491935+0000","last_peered":"2024-09-16T10:24:44.491935+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:44.491935+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24354816,"num_objects":5946,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":23784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1754},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5978","reported_seq":6131,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:45.520151+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:45.520151+0000","last_peered":"2024-09-16T10:24:45.520151+0000","last_clean":"2024-09-16T10:24:45.520151+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:45.520151+0000","last_undegraded":"2024-09-16T10:24:45.520151+0000","last_fullsized":"2024-09-16T10:24:45.520151+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5978,"log_dups_size":0,"ondisk_log_size":5978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24461312,"num_objects":5972,"num_object_clones":0,"num_object_copies":11944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":8764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6038","reported_seq":6220,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:45.461998+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:45.461998+0000","last_peered":"2024-09-16T10:24:45.461998+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:45.461998+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24715264,"num_objects":6034,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":8380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5988","reported_seq":6040,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:45.524729+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:45.524729+0000","last_peered":"2024-09-16T10:24:45.524729+0000","last_clean":"2024-09-16T10:24:45.524729+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:45.524729+0000","last_undegraded":"2024-09-16T10:24:45.524729+0000","last_fullsized":"2024-09-16T10:24:45.524729+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5988,"log_dups_size":0,"ondisk_log_size":5988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24526848,"num_objects":5988,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":23952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5948","reported_seq":6101,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:44.585389+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:44.585389+0000","last_peered":"2024-09-16T10:24:44.585389+0000","last_clean":"2024-09-16T10:24:44.585389+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:44.585389+0000","last_undegraded":"2024-09-16T10:24:44.585389+0000","last_fullsized":"2024-09-16T10:24:44.585389+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24334336,"num_objects":5941,"num_object_clones":0,"num_object_copies":11882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":10360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5875","reported_seq":6035,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.582887+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:44.582887+0000","last_peered":"2024-09-16T10:24:44.582887+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:44.582887+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5875,"log_dups_size":0,"ondisk_log_size":5875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24043520,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2500,"num_write_kb":10000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1653},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6137","reported_seq":6307,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.574510+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:44.574510+0000","last_peered":"2024-09-16T10:24:44.574510+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:44.574510+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6137,"log_dups_size":0,"ondisk_log_size":6137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25112576,"num_objects":6131,"num_object_clones":0,"num_object_copies":12262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":9516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1728},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5910","reported_seq":6079,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:44.614123+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:44.614123+0000","last_peered":"2024-09-16T10:24:44.614123+0000","last_clean":"2024-09-16T10:24:44.614123+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:44.614123+0000","last_undegraded":"2024-09-16T10:24:44.614123+0000","last_fullsized":"2024-09-16T10:24:44.614123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5910,"log_dups_size":0,"ondisk_log_size":5910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24199168,"num_objects":5908,"num_object_clones":0,"num_object_copies":11816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1713,"num_write_kb":6852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5899","reported_seq":8708,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:45.526184+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:45.526184+0000","last_peered":"2024-09-16T10:24:45.526184+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:45.526184+0000","last_undegraded":"2024-09-16T10:24:45.526184+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24134598,"num_objects":5894,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2839,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5897,"num_write_kb":23572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":5562368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2839},{"shards":"2","objects":3055}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6140","reported_seq":6216,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.620215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:44.620215+0000","last_peered":"2024-09-16T10:24:44.620215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:44.620215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25117638,"num_objects":6134,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":24532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1700},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5934","reported_seq":6036,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:45.472484+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:45.472484+0000","last_peered":"2024-09-16T10:24:45.472484+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:45.472484+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286150,"num_objects":5931,"num_object_clones":0,"num_object_copies":11862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5932,"num_write_kb":23719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1654}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5952","reported_seq":6035,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:45.489826+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:45.489826+0000","last_peered":"2024-09-16T10:24:45.489826+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:45.489826+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24355782,"num_objects":5948,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":23787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1756}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5791","reported_seq":5867,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.536150+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:44.536150+0000","last_peered":"2024-09-16T10:24:44.536150+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:44.536150+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5791,"log_dups_size":0,"ondisk_log_size":5791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23712227,"num_objects":5790,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5790,"num_write_kb":23157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1631},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6049","reported_seq":6179,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.570741+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:44.570741+0000","last_peered":"2024-09-16T10:24:44.570741+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:44.570741+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6049,"log_dups_size":0,"ondisk_log_size":6049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24745385,"num_objects":6044,"num_object_clones":0,"num_object_copies":12088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6046,"num_write_kb":24168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1707},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5844","reported_seq":5921,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.573208+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:44.573208+0000","last_peered":"2024-09-16T10:24:44.573208+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:44.573208+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5844,"log_dups_size":0,"ondisk_log_size":5844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23921606,"num_objects":5842,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5842,"num_write_kb":23362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1675},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6144","reported_seq":6233,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.528709+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:44.528709+0000","last_peered":"2024-09-16T10:24:44.528709+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:44.528709+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6144,"log_dups_size":0,"ondisk_log_size":6144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25118604,"num_objects":6136,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":24535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1741},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":391139830,"num_objects":95509,"num_object_clones":0,"num_object_copies":191018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2839,"num_objects_unfound":0,"num_objects_dirty":95509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73138,"num_write_kb":292440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1377,"num_bytes_recovered":5566947,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":784297984,"data_stored":784207659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95573,"ondisk_log_size":95573,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018907,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507764,"kb_used_data":327660,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93864076,"statfs":{"total":96636764160,"available":96116813824,"internally_reserved":0,"allocated":335523840,"data_stored":335280601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":184413872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":3.9239999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":3.1059999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]}]},{"osd":1,"up_from":40,"seq":171798691851,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182272,"kb_used_data":82552,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94189568,"statfs":{"total":96636764160,"available":96450117632,"internally_reserved":0,"allocated":84533248,"data_stored":84210523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":102100650},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1989999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215135,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529248,"kb_used_data":353304,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93842592,"statfs":{"total":96636764160,"available":96094814208,"internally_reserved":0,"allocated":361783296,"data_stored":361543671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":180154024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.089}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.0489999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":363028480,"data_stored":363024867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":85422080,"data_stored":85338981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335847424,"data_stored":335843811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:47.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:48.500 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:48.500 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:48.526 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-09-16T10:24:46.586697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":391139849,"num_objects":95511,"num_object_clones":0,"num_object_copies":191022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2839,"num_objects_unfound":0,"num_objects_dirty":95511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73140,"num_write_kb":292442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1378,"num_bytes_recovered":5566966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95576,"ondisk_log_size":95576,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1219284,"kb_used_data":763516,"kb_used_omap":12,"kb_used_meta":455731,"kb_avail":281896236,"statfs":{"total":289910292480,"available":288661745664,"internally_reserved":0,"allocated":781840384,"data_stored":781034795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":466668546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,4,7,9,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26402816,"num_objects":6446,"num_object_clones":0,"num_object_copies":12892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-442,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6446,"num_write_kb":25784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":442,"num_bytes_recovered":1810432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002742"},"pg_stats":[{"pgid":"2.f","version":"42'5946","reported_seq":6011,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.491935+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:44.491935+0000","last_peered":"2024-09-16T10:24:44.491935+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:44.491935+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24354816,"num_objects":5946,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":23784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1754},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5978","reported_seq":6131,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:45.520151+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:45.520151+0000","last_peered":"2024-09-16T10:24:45.520151+0000","last_clean":"2024-09-16T10:24:45.520151+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:45.520151+0000","last_undegraded":"2024-09-16T10:24:45.520151+0000","last_fullsized":"2024-09-16T10:24:45.520151+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5978,"log_dups_size":0,"ondisk_log_size":5978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24461312,"num_objects":5972,"num_object_clones":0,"num_object_copies":11944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":8764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6038","reported_seq":6220,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:45.461998+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:45.461998+0000","last_peered":"2024-09-16T10:24:45.461998+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:45.461998+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24715264,"num_objects":6034,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":8380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5988","reported_seq":6040,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:45.524729+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:45.524729+0000","last_peered":"2024-09-16T10:24:45.524729+0000","last_clean":"2024-09-16T10:24:45.524729+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:45.524729+0000","last_undegraded":"2024-09-16T10:24:45.524729+0000","last_fullsized":"2024-09-16T10:24:45.524729+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5988,"log_dups_size":0,"ondisk_log_size":5988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24526848,"num_objects":5988,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":23952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5948","reported_seq":6101,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:44.585389+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:44.585389+0000","last_peered":"2024-09-16T10:24:44.585389+0000","last_clean":"2024-09-16T10:24:44.585389+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:44.585389+0000","last_undegraded":"2024-09-16T10:24:44.585389+0000","last_fullsized":"2024-09-16T10:24:44.585389+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24334336,"num_objects":5941,"num_object_clones":0,"num_object_copies":11882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":10360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5875","reported_seq":6035,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.582887+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:44.582887+0000","last_peered":"2024-09-16T10:24:44.582887+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:44.582887+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5875,"log_dups_size":0,"ondisk_log_size":5875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24043520,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2500,"num_write_kb":10000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1653},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6137","reported_seq":6307,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.574510+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:44.574510+0000","last_peered":"2024-09-16T10:24:44.574510+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:44.574510+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6137,"log_dups_size":0,"ondisk_log_size":6137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25112576,"num_objects":6131,"num_object_clones":0,"num_object_copies":12262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":9516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1728},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5910","reported_seq":6079,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:44.614123+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:44.614123+0000","last_peered":"2024-09-16T10:24:44.614123+0000","last_clean":"2024-09-16T10:24:44.614123+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:44.614123+0000","last_undegraded":"2024-09-16T10:24:44.614123+0000","last_fullsized":"2024-09-16T10:24:44.614123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5910,"log_dups_size":0,"ondisk_log_size":5910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24199168,"num_objects":5908,"num_object_clones":0,"num_object_copies":11816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1713,"num_write_kb":6852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5899","reported_seq":8708,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:45.526184+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:45.526184+0000","last_peered":"2024-09-16T10:24:45.526184+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:45.526184+0000","last_undegraded":"2024-09-16T10:24:45.526184+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24134598,"num_objects":5894,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2839,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5897,"num_write_kb":23572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":5562368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2839},{"shards":"2","objects":3055}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6140","reported_seq":6216,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.620215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:44.620215+0000","last_peered":"2024-09-16T10:24:44.620215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:44.620215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25117638,"num_objects":6134,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":24532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1700},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5934","reported_seq":6036,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:45.472484+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:45.472484+0000","last_peered":"2024-09-16T10:24:45.472484+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:45.472484+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286150,"num_objects":5931,"num_object_clones":0,"num_object_copies":11862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5932,"num_write_kb":23719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1654}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5952","reported_seq":6035,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:45.489826+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:45.489826+0000","last_peered":"2024-09-16T10:24:45.489826+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:45.489826+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24355782,"num_objects":5948,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":23787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1756}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5791","reported_seq":5867,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.536150+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:44.536150+0000","last_peered":"2024-09-16T10:24:44.536150+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:44.536150+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5791,"log_dups_size":0,"ondisk_log_size":5791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23712227,"num_objects":5790,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5790,"num_write_kb":23157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1631},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6049","reported_seq":6179,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.570741+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:44.570741+0000","last_peered":"2024-09-16T10:24:44.570741+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:44.570741+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6049,"log_dups_size":0,"ondisk_log_size":6049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24745385,"num_objects":6044,"num_object_clones":0,"num_object_copies":12088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6046,"num_write_kb":24168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1707},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5844","reported_seq":5921,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.573208+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:44.573208+0000","last_peered":"2024-09-16T10:24:44.573208+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:44.573208+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5844,"log_dups_size":0,"ondisk_log_size":5844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23921606,"num_objects":5842,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5842,"num_write_kb":23362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1675},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6144","reported_seq":6233,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.528709+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:44.528709+0000","last_peered":"2024-09-16T10:24:44.528709+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:44.528709+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6144,"log_dups_size":0,"ondisk_log_size":6144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25118604,"num_objects":6136,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":24535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1741},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":391139830,"num_objects":95509,"num_object_clones":0,"num_object_copies":191018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2839,"num_objects_unfound":0,"num_objects_dirty":95509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73138,"num_write_kb":292440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1377,"num_bytes_recovered":5566947,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":784297984,"data_stored":784207659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95573,"ondisk_log_size":95573,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018907,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507764,"kb_used_data":327660,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93864076,"statfs":{"total":96636764160,"available":96116813824,"internally_reserved":0,"allocated":335523840,"data_stored":335280601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":184413872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":3.9239999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":3.1059999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]}]},{"osd":1,"up_from":40,"seq":171798691851,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182272,"kb_used_data":82552,"kb_used_omap":4,"kb_used_meta":99707,"kb_avail":94189568,"statfs":{"total":96636764160,"available":96450117632,"internally_reserved":0,"allocated":84533248,"data_stored":84210523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":102100650},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.1989999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215135,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529248,"kb_used_data":353304,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93842592,"statfs":{"total":96636764160,"available":96094814208,"internally_reserved":0,"allocated":361783296,"data_stored":361543671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":180154024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.089}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.0489999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":363028480,"data_stored":363024867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":85422080,"data_stored":85338981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335847424,"data_stored":335843811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:48.527 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:48.527 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'5946", 'reported_seq': 6011, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:44.491935+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:44.491935+0000', 'last_peered': '2024-09-16T10:24:44.491935+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:44.491935+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5946, 'log_dups_size': 0, 'ondisk_log_size': 5946, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24354816, 'num_objects': 5946, 'num_object_clones': 0, 'num_object_copies': 11892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5946, 'num_write_kb': 23784, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1754}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:48.527 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:48.527 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'6038", 'reported_seq': 6220, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:45.461998+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:45.461998+0000', 'last_peered': '2024-09-16T10:24:45.461998+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:45.461998+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 6038, 'log_dups_size': 0, 'ondisk_log_size': 6038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24715264, 'num_objects': 6034, 'num_object_clones': 0, 'num_object_copies': 12068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6034, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2095, 'num_write_kb': 8380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 1699}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:48.527 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'5875", 'reported_seq': 6035, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:44.582887+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:44.582887+0000', 'last_peered': '2024-09-16T10:24:44.582887+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:44.582887+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 5875, 'log_dups_size': 0, 'ondisk_log_size': 5875, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24043520, 'num_objects': 5870, 'num_object_clones': 0, 'num_object_copies': 11740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2500, 'num_write_kb': 10000, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1653}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'6137", 'reported_seq': 6307, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:44.574510+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:44.574510+0000', 'last_peered': '2024-09-16T10:24:44.574510+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:44.574510+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 6137, 'log_dups_size': 0, 'ondisk_log_size': 6137, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25112576, 'num_objects': 6131, 'num_object_clones': 0, 'num_object_copies': 12262, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2379, 'num_write_kb': 9516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1728}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'5899", 'reported_seq': 8708, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:45.526184+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:45.526184+0000', 'last_peered': '2024-09-16T10:24:45.526184+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:45.526184+0000', 'last_undegraded': '2024-09-16T10:24:45.526184+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5899, 'log_dups_size': 0, 'ondisk_log_size': 5899, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24134598, 'num_objects': 5894, 'num_object_clones': 0, 'num_object_copies': 11788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2839, 'num_objects_unfound': 0, 'num_objects_dirty': 5894, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5897, 'num_write_kb': 23572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1360, 'num_bytes_recovered': 5562368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2839}, {'shards': '2', 'objects': 3055}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'6140", 'reported_seq': 6216, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:44.620215+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:44.620215+0000', 'last_peered': '2024-09-16T10:24:44.620215+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:44.620215+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6140, 'log_dups_size': 0, 'ondisk_log_size': 6140, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25117638, 'num_objects': 6134, 'num_object_clones': 0, 'num_object_copies': 12268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6138, 'num_write_kb': 24532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1700}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'5934", 'reported_seq': 6036, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:45.472484+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:45.472484+0000', 'last_peered': '2024-09-16T10:24:45.472484+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:45.472484+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 5934, 'log_dups_size': 0, 'ondisk_log_size': 5934, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24286150, 'num_objects': 5931, 'num_object_clones': 0, 'num_object_copies': 11862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5932, 'num_write_kb': 23719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 1654}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:48.528 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'5952", 'reported_seq': 6035, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:45.489826+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:45.489826+0000', 'last_peered': '2024-09-16T10:24:45.489826+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:45.489826+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5952, 'log_dups_size': 0, 'ondisk_log_size': 5952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24355782, 'num_objects': 5948, 'num_object_clones': 0, 'num_object_copies': 11896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5950, 'num_write_kb': 23787, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 1756}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:48.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:48.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'5791", 'reported_seq': 5867, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:44.536150+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:44.536150+0000', 'last_peered': '2024-09-16T10:24:44.536150+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:44.536150+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5791, 'log_dups_size': 0, 'ondisk_log_size': 5791, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23712227, 'num_objects': 5790, 'num_object_clones': 0, 'num_object_copies': 11580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5790, 'num_write_kb': 23157, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1631}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:48.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:48.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'6049", 'reported_seq': 6179, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:44.570741+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:44.570741+0000', 'last_peered': '2024-09-16T10:24:44.570741+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:44.570741+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 6049, 'log_dups_size': 0, 'ondisk_log_size': 6049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24745385, 'num_objects': 6044, 'num_object_clones': 0, 'num_object_copies': 12088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6046, 'num_write_kb': 24168, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1707}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:48.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:48.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'5844", 'reported_seq': 5921, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:44.573208+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:44.573208+0000', 'last_peered': '2024-09-16T10:24:44.573208+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:44.573208+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5844, 'log_dups_size': 0, 'ondisk_log_size': 5844, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23921606, 'num_objects': 5842, 'num_object_clones': 0, 'num_object_copies': 11684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5842, 'num_write_kb': 23362, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1675}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:48.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:48.529 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'6144", 'reported_seq': 6233, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:44.528709+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:44.528709+0000', 'last_peered': '2024-09-16T10:24:44.528709+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:44.528709+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6144, 'log_dups_size': 0, 'ondisk_log_size': 6144, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25118604, 'num_objects': 6136, 'num_object_clones': 0, 'num_object_copies': 12272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6141, 'num_write_kb': 24535, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1741}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:48.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:48.534 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 96 20 97441 97421 3.96326 1.94922 0.0348833 0.0197034 2024-09-16T10:24:49.210 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:49.210 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/191022 objects degraded (24.672%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":177,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":95511,"data_bytes":391139849,"bytes_used":1265860608,"bytes_avail":288644431872,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":191022,"degraded_ratio":0.24671503805844353,"misplaced_objects":2839,"misplaced_total":191022,"misplaced_ratio":0.014862162473432382,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":98963,"recovering_keys_per_sec":0,"num_objects_recovered":290,"num_bytes_recovered":1187840,"num_keys_recovered":0,"write_bytes_sec":1857450,"read_op_per_sec":0,"write_op_per_sec":453},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-16T10:23:56.572050+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:24:49.242 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:49.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:49.534 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 97 20 97924 97904 3.94185 1.88672 0.00871595 0.0198075 2024-09-16T10:24:49.806 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:49.808 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:49.833 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-16T10:24:48.587195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":391139849,"num_objects":95511,"num_object_clones":0,"num_object_copies":191022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2839,"num_objects_unfound":0,"num_objects_dirty":95511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73140,"num_write_kb":292442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1378,"num_bytes_recovered":5566966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95576,"ondisk_log_size":95576,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1236192,"kb_used_data":772232,"kb_used_omap":12,"kb_used_meta":463923,"kb_avail":281879328,"statfs":{"total":289910292480,"available":288644431872,"internally_reserved":0,"allocated":790765568,"data_stored":789959979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":475057154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,7,11,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22294528,"num_objects":5443,"num_object_clones":0,"num_object_copies":10886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-290,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5443,"num_write_kb":21772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002758"},"pg_stats":[{"pgid":"2.f","version":"42'5946","reported_seq":6011,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.491935+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:44.491935+0000","last_peered":"2024-09-16T10:24:44.491935+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:44.491935+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5946,"log_dups_size":0,"ondisk_log_size":5946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24354816,"num_objects":5946,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5946,"num_write_kb":23784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1754},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'5978","reported_seq":6131,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:45.520151+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:45.520151+0000","last_peered":"2024-09-16T10:24:45.520151+0000","last_clean":"2024-09-16T10:24:45.520151+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:45.520151+0000","last_undegraded":"2024-09-16T10:24:45.520151+0000","last_fullsized":"2024-09-16T10:24:45.520151+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":5978,"log_dups_size":0,"ondisk_log_size":5978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24461312,"num_objects":5972,"num_object_clones":0,"num_object_copies":11944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":8764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6038","reported_seq":6220,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:45.461998+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:45.461998+0000","last_peered":"2024-09-16T10:24:45.461998+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:45.461998+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6038,"log_dups_size":0,"ondisk_log_size":6038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24715264,"num_objects":6034,"num_object_clones":0,"num_object_copies":12068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":8380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1699}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'5988","reported_seq":6040,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:45.524729+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:45.524729+0000","last_peered":"2024-09-16T10:24:45.524729+0000","last_clean":"2024-09-16T10:24:45.524729+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:45.524729+0000","last_undegraded":"2024-09-16T10:24:45.524729+0000","last_fullsized":"2024-09-16T10:24:45.524729+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5988,"log_dups_size":0,"ondisk_log_size":5988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24526848,"num_objects":5988,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":23952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'5948","reported_seq":6101,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:44.585389+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:44.585389+0000","last_peered":"2024-09-16T10:24:44.585389+0000","last_clean":"2024-09-16T10:24:44.585389+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:44.585389+0000","last_undegraded":"2024-09-16T10:24:44.585389+0000","last_fullsized":"2024-09-16T10:24:44.585389+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":5948,"log_dups_size":0,"ondisk_log_size":5948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24334336,"num_objects":5941,"num_object_clones":0,"num_object_copies":11882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2590,"num_write_kb":10360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'5875","reported_seq":6035,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.582887+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:44.582887+0000","last_peered":"2024-09-16T10:24:44.582887+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:44.582887+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":5875,"log_dups_size":0,"ondisk_log_size":5875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24043520,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2500,"num_write_kb":10000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1653},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6137","reported_seq":6307,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.574510+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:44.574510+0000","last_peered":"2024-09-16T10:24:44.574510+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:44.574510+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6137,"log_dups_size":0,"ondisk_log_size":6137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25112576,"num_objects":6131,"num_object_clones":0,"num_object_copies":12262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2379,"num_write_kb":9516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1728},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'5910","reported_seq":6079,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:44.614123+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:44.614123+0000","last_peered":"2024-09-16T10:24:44.614123+0000","last_clean":"2024-09-16T10:24:44.614123+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:44.614123+0000","last_undegraded":"2024-09-16T10:24:44.614123+0000","last_fullsized":"2024-09-16T10:24:44.614123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":5910,"log_dups_size":0,"ondisk_log_size":5910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24199168,"num_objects":5908,"num_object_clones":0,"num_object_copies":11816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1713,"num_write_kb":6852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'5899","reported_seq":8708,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:45.526184+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:45.526184+0000","last_peered":"2024-09-16T10:24:45.526184+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:45.526184+0000","last_undegraded":"2024-09-16T10:24:45.526184+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5899,"log_dups_size":0,"ondisk_log_size":5899,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24134598,"num_objects":5894,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2839,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5897,"num_write_kb":23572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1360,"num_bytes_recovered":5562368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2839},{"shards":"2","objects":3055}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6140","reported_seq":6216,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.620215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:44.620215+0000","last_peered":"2024-09-16T10:24:44.620215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:44.620215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25117638,"num_objects":6134,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":24532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1700},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'5934","reported_seq":6036,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:45.472484+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:45.472484+0000","last_peered":"2024-09-16T10:24:45.472484+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:45.472484+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":5934,"log_dups_size":0,"ondisk_log_size":5934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286150,"num_objects":5931,"num_object_clones":0,"num_object_copies":11862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5932,"num_write_kb":23719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1654}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'5952","reported_seq":6035,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:45.489826+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:45.489826+0000","last_peered":"2024-09-16T10:24:45.489826+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:45.489826+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24355782,"num_objects":5948,"num_object_clones":0,"num_object_copies":11896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":23787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1756}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5791","reported_seq":5867,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.536150+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:44.536150+0000","last_peered":"2024-09-16T10:24:44.536150+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:44.536150+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5791,"log_dups_size":0,"ondisk_log_size":5791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23712227,"num_objects":5790,"num_object_clones":0,"num_object_copies":11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5790,"num_write_kb":23157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1631},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6049","reported_seq":6179,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.570741+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:44.570741+0000","last_peered":"2024-09-16T10:24:44.570741+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:44.570741+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6049,"log_dups_size":0,"ondisk_log_size":6049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24745385,"num_objects":6044,"num_object_clones":0,"num_object_copies":12088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6046,"num_write_kb":24168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1707},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5844","reported_seq":5921,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.573208+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:44.573208+0000","last_peered":"2024-09-16T10:24:44.573208+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:44.573208+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5844,"log_dups_size":0,"ondisk_log_size":5844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23921606,"num_objects":5842,"num_object_clones":0,"num_object_copies":11684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5842,"num_write_kb":23362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1675},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6144","reported_seq":6233,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:44.528709+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:44.528709+0000","last_peered":"2024-09-16T10:24:44.528709+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:44.528709+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6144,"log_dups_size":0,"ondisk_log_size":6144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25118604,"num_objects":6136,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6141,"num_write_kb":24535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1741},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":391139830,"num_objects":95509,"num_object_clones":0,"num_object_copies":191018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2839,"num_objects_unfound":0,"num_objects_dirty":95509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73138,"num_write_kb":292440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1377,"num_bytes_recovered":5566947,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":793165824,"data_stored":793075499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95573,"ondisk_log_size":95573,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018907,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507764,"kb_used_data":327660,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93864076,"statfs":{"total":96636764160,"available":96116813824,"internally_reserved":0,"allocated":335523840,"data_stored":335280601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":184413872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":3.9239999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":3.1059999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]}]},{"osd":1,"up_from":40,"seq":171798691852,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":199180,"kb_used_data":91268,"kb_used_omap":4,"kb_used_meta":107899,"kb_avail":94172660,"statfs":{"total":96636764160,"available":96432803840,"internally_reserved":0,"allocated":93458432,"data_stored":93135707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":110489258},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.244}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215135,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529248,"kb_used_data":353304,"kb_used_omap":4,"kb_used_meta":175931,"kb_avail":93842592,"statfs":{"total":96636764160,"available":96094814208,"internally_reserved":0,"allocated":361783296,"data_stored":361543671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":180154024},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.089}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.0489999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":363028480,"data_stored":363024867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94289920,"data_stored":94206821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335847424,"data_stored":335843811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:50.534 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 98 20 98463 98443 3.92311 2.10547 0.0581695 0.0199058 2024-09-16T10:24:51.534 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 99 20 99016 98996 3.9053 2.16016 0.0219394 0.0199966 2024-09-16T10:24:52.535 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:24:52.534130+0000 min lat: 0.00139595 max lat: 2.61294 avg lat: 0.0200843 2024-09-16T10:24:52.535 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:24:52.535 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 100 20 99576 99556 3.88813 2.1875 0.0468157 0.0200843 2024-09-16T10:24:52.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:53.432 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:53.432 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:53.454 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-16T10:24:52.588278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":402043401,"num_objects":98173,"num_object_clones":0,"num_object_copies":196346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2679,"num_objects_unfound":0,"num_objects_dirty":98173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75802,"num_write_kb":303090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":6222326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98238,"ondisk_log_size":98238,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1257988,"kb_used_data":785836,"kb_used_omap":12,"kb_used_meta":472115,"kb_avail":281857532,"statfs":{"total":289910292480,"available":288622112768,"internally_reserved":0,"allocated":804696064,"data_stored":803890475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":483445762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,7,8,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21975040,"num_objects":5365,"num_object_clones":0,"num_object_copies":10730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-312,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5365,"num_write_kb":21460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":1277952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002918"},"pg_stats":[{"pgid":"2.f","version":"42'6105","reported_seq":6168,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.602182+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:49.602182+0000","last_peered":"2024-09-16T10:24:49.602182+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:49.602182+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25006080,"num_objects":6105,"num_object_clones":0,"num_object_copies":12210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6105,"num_write_kb":24420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1913},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6128","reported_seq":6281,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:50.531162+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:50.531162+0000","last_peered":"2024-09-16T10:24:50.531162+0000","last_clean":"2024-09-16T10:24:50.531162+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:50.531162+0000","last_undegraded":"2024-09-16T10:24:50.531162+0000","last_fullsized":"2024-09-16T10:24:50.531162+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25075712,"num_objects":6122,"num_object_clones":0,"num_object_copies":12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2341,"num_write_kb":9364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6211","reported_seq":6393,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:50.539227+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:50.539227+0000","last_peered":"2024-09-16T10:24:50.539227+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:50.539227+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6211,"log_dups_size":0,"ondisk_log_size":6211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25423872,"num_objects":6207,"num_object_clones":0,"num_object_copies":12414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2268,"num_write_kb":9072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1872}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6157","reported_seq":6209,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:50.500855+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:50.500855+0000","last_peered":"2024-09-16T10:24:50.500855+0000","last_clean":"2024-09-16T10:24:50.500855+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:50.500855+0000","last_undegraded":"2024-09-16T10:24:50.500855+0000","last_fullsized":"2024-09-16T10:24:50.500855+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6157,"log_dups_size":0,"ondisk_log_size":6157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25219072,"num_objects":6157,"num_object_clones":0,"num_object_copies":12314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":24628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6117","reported_seq":6270,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:49.616200+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:49.616200+0000","last_peered":"2024-09-16T10:24:49.616200+0000","last_clean":"2024-09-16T10:24:49.616200+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:49.616200+0000","last_undegraded":"2024-09-16T10:24:49.616200+0000","last_fullsized":"2024-09-16T10:24:49.616200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6117,"log_dups_size":0,"ondisk_log_size":6117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25026560,"num_objects":6110,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2759,"num_write_kb":11036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6042","reported_seq":6202,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.614797+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:49.614797+0000","last_peered":"2024-09-16T10:24:49.614797+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:49.614797+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6042,"log_dups_size":0,"ondisk_log_size":6042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24727552,"num_objects":6037,"num_object_clones":0,"num_object_copies":12074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2667,"num_write_kb":10668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1820},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6338","reported_seq":6505,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.628397+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:49.628397+0000","last_peered":"2024-09-16T10:24:49.628397+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:49.628397+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6338,"log_dups_size":0,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25935872,"num_objects":6332,"num_object_clones":0,"num_object_copies":12664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2580,"num_write_kb":10320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1929},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6075","reported_seq":6247,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:49.530677+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:49.530677+0000","last_peered":"2024-09-16T10:24:49.530677+0000","last_clean":"2024-09-16T10:24:49.530677+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:49.530677+0000","last_undegraded":"2024-09-16T10:24:49.530677+0000","last_fullsized":"2024-09-16T10:24:49.530677+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24875008,"num_objects":6073,"num_object_clones":0,"num_object_copies":12146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1878,"num_write_kb":7512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6054","reported_seq":9183,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:50.539163+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:50.539163+0000","last_peered":"2024-09-16T10:24:50.539163+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:50.539163+0000","last_undegraded":"2024-09-16T10:24:50.539163+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6054,"log_dups_size":0,"ondisk_log_size":6054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24769478,"num_objects":6049,"num_object_clones":0,"num_object_copies":12098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2679,"num_objects_unfound":0,"num_objects_dirty":6049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":24192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1520,"num_bytes_recovered":6217728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2679},{"shards":"2","objects":3370}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6319","reported_seq":6395,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.621234+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:49.621234+0000","last_peered":"2024-09-16T10:24:49.621234+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:49.621234+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25850822,"num_objects":6313,"num_object_clones":0,"num_object_copies":12626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6317,"num_write_kb":25248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1879},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6081","reported_seq":6183,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:50.474207+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:50.474207+0000","last_peered":"2024-09-16T10:24:50.474207+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:50.474207+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24888262,"num_objects":6078,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":24307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1801}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6120","reported_seq":6203,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:50.434160+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:50.434160+0000","last_peered":"2024-09-16T10:24:50.434160+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:50.434160+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6120,"log_dups_size":0,"ondisk_log_size":6120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25043910,"num_objects":6116,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6118,"num_write_kb":24459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1924}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5970","reported_seq":6046,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.547966+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:49.547966+0000","last_peered":"2024-09-16T10:24:49.547966+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:49.547966+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5970,"log_dups_size":0,"ondisk_log_size":5970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24445411,"num_objects":5969,"num_object_clones":0,"num_object_copies":11938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":23873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1810},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6209","reported_seq":6339,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.563375+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:49.563375+0000","last_peered":"2024-09-16T10:24:49.563375+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:49.563375+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400745,"num_objects":6204,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":24808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1867},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5993","reported_seq":6070,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.596445+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:49.596445+0000","last_peered":"2024-09-16T10:24:49.596445+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:49.596445+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5993,"log_dups_size":0,"ondisk_log_size":5993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24531910,"num_objects":5991,"num_object_clones":0,"num_object_copies":11982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5991,"num_write_kb":23958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1824},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6316","reported_seq":6405,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.609435+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:49.609435+0000","last_peered":"2024-09-16T10:24:49.609435+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:49.609435+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25823116,"num_objects":6308,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":25223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1913},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":402043382,"num_objects":98171,"num_object_clones":0,"num_object_copies":196342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2679,"num_objects_unfound":0,"num_objects_dirty":98171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75800,"num_write_kb":303088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1537,"num_bytes_recovered":6222307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":806764544,"data_stored":806674219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98235,"ondisk_log_size":98235,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018908,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517268,"kb_used_data":333068,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93854572,"statfs":{"total":96636764160,"available":96107081728,"internally_reserved":0,"allocated":341061632,"data_stored":340818393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":188608176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":2.4119999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":2.7869999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6739999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691852,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":199180,"kb_used_data":91268,"kb_used_omap":4,"kb_used_meta":107899,"kb_avail":94172660,"statfs":{"total":96636764160,"available":96432803840,"internally_reserved":0,"allocated":93458432,"data_stored":93135707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":110489258},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.244}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541540,"kb_used_data":361500,"kb_used_omap":4,"kb_used_meta":180027,"kb_avail":93830300,"statfs":{"total":96636764160,"available":96082227200,"internally_reserved":0,"allocated":370176000,"data_stored":369936375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":184348328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.089}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.0489999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":371355648,"data_stored":371352035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94289920,"data_stored":94206821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341118976,"data_stored":341115363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:53.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:53.535 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 101 20 100082 100062 3.8692 1.97656 0.0250348 0.0201824 2024-09-16T10:24:54.031 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:54.032 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:54.058 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-16T10:24:52.588278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":402043401,"num_objects":98173,"num_object_clones":0,"num_object_copies":196346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2679,"num_objects_unfound":0,"num_objects_dirty":98173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75802,"num_write_kb":303090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":6222326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98238,"ondisk_log_size":98238,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1257988,"kb_used_data":785836,"kb_used_omap":12,"kb_used_meta":472115,"kb_avail":281857532,"statfs":{"total":289910292480,"available":288622112768,"internally_reserved":0,"allocated":804696064,"data_stored":803890475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":483445762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,7,8,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21975040,"num_objects":5365,"num_object_clones":0,"num_object_copies":10730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-312,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5365,"num_write_kb":21460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":1277952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002918"},"pg_stats":[{"pgid":"2.f","version":"42'6105","reported_seq":6168,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.602182+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:49.602182+0000","last_peered":"2024-09-16T10:24:49.602182+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:49.602182+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25006080,"num_objects":6105,"num_object_clones":0,"num_object_copies":12210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6105,"num_write_kb":24420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1913},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6128","reported_seq":6281,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:50.531162+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:50.531162+0000","last_peered":"2024-09-16T10:24:50.531162+0000","last_clean":"2024-09-16T10:24:50.531162+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:50.531162+0000","last_undegraded":"2024-09-16T10:24:50.531162+0000","last_fullsized":"2024-09-16T10:24:50.531162+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25075712,"num_objects":6122,"num_object_clones":0,"num_object_copies":12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2341,"num_write_kb":9364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6211","reported_seq":6393,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:50.539227+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:50.539227+0000","last_peered":"2024-09-16T10:24:50.539227+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:50.539227+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6211,"log_dups_size":0,"ondisk_log_size":6211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25423872,"num_objects":6207,"num_object_clones":0,"num_object_copies":12414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2268,"num_write_kb":9072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1872}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6157","reported_seq":6209,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:50.500855+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:50.500855+0000","last_peered":"2024-09-16T10:24:50.500855+0000","last_clean":"2024-09-16T10:24:50.500855+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:50.500855+0000","last_undegraded":"2024-09-16T10:24:50.500855+0000","last_fullsized":"2024-09-16T10:24:50.500855+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6157,"log_dups_size":0,"ondisk_log_size":6157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25219072,"num_objects":6157,"num_object_clones":0,"num_object_copies":12314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":24628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6117","reported_seq":6270,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:49.616200+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:49.616200+0000","last_peered":"2024-09-16T10:24:49.616200+0000","last_clean":"2024-09-16T10:24:49.616200+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:49.616200+0000","last_undegraded":"2024-09-16T10:24:49.616200+0000","last_fullsized":"2024-09-16T10:24:49.616200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6117,"log_dups_size":0,"ondisk_log_size":6117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25026560,"num_objects":6110,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2759,"num_write_kb":11036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6042","reported_seq":6202,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.614797+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:49.614797+0000","last_peered":"2024-09-16T10:24:49.614797+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:49.614797+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6042,"log_dups_size":0,"ondisk_log_size":6042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24727552,"num_objects":6037,"num_object_clones":0,"num_object_copies":12074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2667,"num_write_kb":10668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1820},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6338","reported_seq":6505,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.628397+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:49.628397+0000","last_peered":"2024-09-16T10:24:49.628397+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:49.628397+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6338,"log_dups_size":0,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25935872,"num_objects":6332,"num_object_clones":0,"num_object_copies":12664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2580,"num_write_kb":10320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1929},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6075","reported_seq":6247,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:49.530677+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:49.530677+0000","last_peered":"2024-09-16T10:24:49.530677+0000","last_clean":"2024-09-16T10:24:49.530677+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:49.530677+0000","last_undegraded":"2024-09-16T10:24:49.530677+0000","last_fullsized":"2024-09-16T10:24:49.530677+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24875008,"num_objects":6073,"num_object_clones":0,"num_object_copies":12146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1878,"num_write_kb":7512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6054","reported_seq":9183,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:50.539163+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:50.539163+0000","last_peered":"2024-09-16T10:24:50.539163+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:50.539163+0000","last_undegraded":"2024-09-16T10:24:50.539163+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6054,"log_dups_size":0,"ondisk_log_size":6054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24769478,"num_objects":6049,"num_object_clones":0,"num_object_copies":12098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2679,"num_objects_unfound":0,"num_objects_dirty":6049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":24192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1520,"num_bytes_recovered":6217728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2679},{"shards":"2","objects":3370}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6319","reported_seq":6395,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.621234+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:49.621234+0000","last_peered":"2024-09-16T10:24:49.621234+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:49.621234+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25850822,"num_objects":6313,"num_object_clones":0,"num_object_copies":12626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6317,"num_write_kb":25248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1879},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6081","reported_seq":6183,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:50.474207+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:50.474207+0000","last_peered":"2024-09-16T10:24:50.474207+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:50.474207+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24888262,"num_objects":6078,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":24307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1801}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6120","reported_seq":6203,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:50.434160+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:50.434160+0000","last_peered":"2024-09-16T10:24:50.434160+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:50.434160+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6120,"log_dups_size":0,"ondisk_log_size":6120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25043910,"num_objects":6116,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6118,"num_write_kb":24459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1924}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5970","reported_seq":6046,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.547966+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:49.547966+0000","last_peered":"2024-09-16T10:24:49.547966+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:49.547966+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5970,"log_dups_size":0,"ondisk_log_size":5970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24445411,"num_objects":5969,"num_object_clones":0,"num_object_copies":11938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":23873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1810},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6209","reported_seq":6339,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.563375+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:49.563375+0000","last_peered":"2024-09-16T10:24:49.563375+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:49.563375+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400745,"num_objects":6204,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":24808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1867},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5993","reported_seq":6070,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.596445+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:49.596445+0000","last_peered":"2024-09-16T10:24:49.596445+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:49.596445+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5993,"log_dups_size":0,"ondisk_log_size":5993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24531910,"num_objects":5991,"num_object_clones":0,"num_object_copies":11982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5991,"num_write_kb":23958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1824},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6316","reported_seq":6405,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.609435+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:49.609435+0000","last_peered":"2024-09-16T10:24:49.609435+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:49.609435+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25823116,"num_objects":6308,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":25223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1913},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":402043382,"num_objects":98171,"num_object_clones":0,"num_object_copies":196342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2679,"num_objects_unfound":0,"num_objects_dirty":98171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75800,"num_write_kb":303088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1537,"num_bytes_recovered":6222307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":806764544,"data_stored":806674219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98235,"ondisk_log_size":98235,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018908,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517268,"kb_used_data":333068,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93854572,"statfs":{"total":96636764160,"available":96107081728,"internally_reserved":0,"allocated":341061632,"data_stored":340818393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":188608176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":2.4119999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":2.7869999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6739999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691852,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":199180,"kb_used_data":91268,"kb_used_omap":4,"kb_used_meta":107899,"kb_avail":94172660,"statfs":{"total":96636764160,"available":96432803840,"internally_reserved":0,"allocated":93458432,"data_stored":93135707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":110489258},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.244}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541540,"kb_used_data":361500,"kb_used_omap":4,"kb_used_meta":180027,"kb_avail":93830300,"statfs":{"total":96636764160,"available":96082227200,"internally_reserved":0,"allocated":370176000,"data_stored":369936375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":184348328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.089}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.0489999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":371355648,"data_stored":371352035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94289920,"data_stored":94206821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341118976,"data_stored":341115363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:54.059 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:54.059 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'6105", 'reported_seq': 6168, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:49.602182+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:49.602182+0000', 'last_peered': '2024-09-16T10:24:49.602182+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:49.602182+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6105, 'log_dups_size': 0, 'ondisk_log_size': 6105, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25006080, 'num_objects': 6105, 'num_object_clones': 0, 'num_object_copies': 12210, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6105, 'num_write_kb': 24420, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1913}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:54.059 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:54.059 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'6211", 'reported_seq': 6393, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:50.539227+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:50.539227+0000', 'last_peered': '2024-09-16T10:24:50.539227+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:50.539227+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 6211, 'log_dups_size': 0, 'ondisk_log_size': 6211, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25423872, 'num_objects': 6207, 'num_object_clones': 0, 'num_object_copies': 12414, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2268, 'num_write_kb': 9072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 1872}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:54.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:54.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'6042", 'reported_seq': 6202, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:49.614797+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:49.614797+0000', 'last_peered': '2024-09-16T10:24:49.614797+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:49.614797+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 6042, 'log_dups_size': 0, 'ondisk_log_size': 6042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24727552, 'num_objects': 6037, 'num_object_clones': 0, 'num_object_copies': 12074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2667, 'num_write_kb': 10668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1820}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:54.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:54.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'6338", 'reported_seq': 6505, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:49.628397+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:49.628397+0000', 'last_peered': '2024-09-16T10:24:49.628397+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:49.628397+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 6338, 'log_dups_size': 0, 'ondisk_log_size': 6338, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25935872, 'num_objects': 6332, 'num_object_clones': 0, 'num_object_copies': 12664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2580, 'num_write_kb': 10320, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1929}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:54.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:54.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'6054", 'reported_seq': 9183, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:50.539163+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:50.539163+0000', 'last_peered': '2024-09-16T10:24:50.539163+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:50.539163+0000', 'last_undegraded': '2024-09-16T10:24:50.539163+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6054, 'log_dups_size': 0, 'ondisk_log_size': 6054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24769478, 'num_objects': 6049, 'num_object_clones': 0, 'num_object_copies': 12098, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2679, 'num_objects_unfound': 0, 'num_objects_dirty': 6049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6052, 'num_write_kb': 24192, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1520, 'num_bytes_recovered': 6217728, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2679}, {'shards': '2', 'objects': 3370}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:54.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:54.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'6319", 'reported_seq': 6395, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:49.621234+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:49.621234+0000', 'last_peered': '2024-09-16T10:24:49.621234+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:49.621234+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6319, 'log_dups_size': 0, 'ondisk_log_size': 6319, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25850822, 'num_objects': 6313, 'num_object_clones': 0, 'num_object_copies': 12626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6317, 'num_write_kb': 25248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1879}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:54.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:54.060 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'6081", 'reported_seq': 6183, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:50.474207+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:50.474207+0000', 'last_peered': '2024-09-16T10:24:50.474207+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:50.474207+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 6081, 'log_dups_size': 0, 'ondisk_log_size': 6081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24888262, 'num_objects': 6078, 'num_object_clones': 0, 'num_object_copies': 12156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6079, 'num_write_kb': 24307, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 1801}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:54.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:54.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'6120", 'reported_seq': 6203, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:50.434160+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:50.434160+0000', 'last_peered': '2024-09-16T10:24:50.434160+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:50.434160+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6120, 'log_dups_size': 0, 'ondisk_log_size': 6120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25043910, 'num_objects': 6116, 'num_object_clones': 0, 'num_object_copies': 12232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6118, 'num_write_kb': 24459, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 1924}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:54.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:54.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'5970", 'reported_seq': 6046, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:49.547966+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:49.547966+0000', 'last_peered': '2024-09-16T10:24:49.547966+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:49.547966+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5970, 'log_dups_size': 0, 'ondisk_log_size': 5970, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24445411, 'num_objects': 5969, 'num_object_clones': 0, 'num_object_copies': 11938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5969, 'num_write_kb': 23873, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1810}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:54.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:54.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'6209", 'reported_seq': 6339, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:49.563375+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:49.563375+0000', 'last_peered': '2024-09-16T10:24:49.563375+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:49.563375+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 6209, 'log_dups_size': 0, 'ondisk_log_size': 6209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25400745, 'num_objects': 6204, 'num_object_clones': 0, 'num_object_copies': 12408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6206, 'num_write_kb': 24808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1867}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:54.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:54.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'5993", 'reported_seq': 6070, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:49.596445+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:49.596445+0000', 'last_peered': '2024-09-16T10:24:49.596445+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:49.596445+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 5993, 'log_dups_size': 0, 'ondisk_log_size': 5993, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24531910, 'num_objects': 5991, 'num_object_clones': 0, 'num_object_copies': 11982, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5991, 'num_write_kb': 23958, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1824}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:54.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:54.061 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'6316", 'reported_seq': 6405, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:49.609435+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:49.609435+0000', 'last_peered': '2024-09-16T10:24:49.609435+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:49.609435+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6316, 'log_dups_size': 0, 'ondisk_log_size': 6316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25823116, 'num_objects': 6308, 'num_object_clones': 0, 'num_object_copies': 12616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6308, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6313, 'num_write_kb': 25223, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1913}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:54.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:24:54.537 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 102 20 100570 100550 3.84986 1.90625 0.0256883 0.0202856 2024-09-16T10:24:54.752 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:54.753 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/196346 objects degraded (24.003%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":183,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":98173,"data_bytes":402043401,"bytes_used":1288179712,"bytes_avail":288622112768,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":196346,"degraded_ratio":0.24002526152811876,"misplaced_objects":2679,"misplaced_total":196346,"misplaced_ratio":0.013644281014128118,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":106470,"recovering_keys_per_sec":0,"num_objects_recovered":312,"num_bytes_recovered":1277952,"num_keys_recovered":0,"write_bytes_sec":1830808,"read_op_per_sec":0,"write_op_per_sec":446},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:24:54.782 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:24:54.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:55.352 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:55.352 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:55.375 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-16T10:24:54.588743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":402043401,"num_objects":98173,"num_object_clones":0,"num_object_copies":196346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2679,"num_objects_unfound":0,"num_objects_dirty":98173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75802,"num_write_kb":303090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":6222326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98238,"ondisk_log_size":98238,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1274692,"kb_used_data":794348,"kb_used_omap":12,"kb_used_meta":480307,"kb_avail":281840828,"statfs":{"total":289910292480,"available":288605007872,"internally_reserved":0,"allocated":813412352,"data_stored":812606763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":491834370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,10,6,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21975040,"num_objects":5365,"num_object_clones":0,"num_object_copies":10730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-312,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5365,"num_write_kb":21460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":1277952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"2.f","version":"42'6105","reported_seq":6168,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.602182+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:49.602182+0000","last_peered":"2024-09-16T10:24:49.602182+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:49.602182+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6105,"log_dups_size":0,"ondisk_log_size":6105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25006080,"num_objects":6105,"num_object_clones":0,"num_object_copies":12210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6105,"num_write_kb":24420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1913},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6128","reported_seq":6281,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:50.531162+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:50.531162+0000","last_peered":"2024-09-16T10:24:50.531162+0000","last_clean":"2024-09-16T10:24:50.531162+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:50.531162+0000","last_undegraded":"2024-09-16T10:24:50.531162+0000","last_fullsized":"2024-09-16T10:24:50.531162+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25075712,"num_objects":6122,"num_object_clones":0,"num_object_copies":12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2341,"num_write_kb":9364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6211","reported_seq":6393,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:50.539227+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:50.539227+0000","last_peered":"2024-09-16T10:24:50.539227+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:50.539227+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6211,"log_dups_size":0,"ondisk_log_size":6211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25423872,"num_objects":6207,"num_object_clones":0,"num_object_copies":12414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2268,"num_write_kb":9072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":1872}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6157","reported_seq":6209,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:50.500855+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:50.500855+0000","last_peered":"2024-09-16T10:24:50.500855+0000","last_clean":"2024-09-16T10:24:50.500855+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:50.500855+0000","last_undegraded":"2024-09-16T10:24:50.500855+0000","last_fullsized":"2024-09-16T10:24:50.500855+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6157,"log_dups_size":0,"ondisk_log_size":6157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25219072,"num_objects":6157,"num_object_clones":0,"num_object_copies":12314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":24628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6117","reported_seq":6270,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:49.616200+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:49.616200+0000","last_peered":"2024-09-16T10:24:49.616200+0000","last_clean":"2024-09-16T10:24:49.616200+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:49.616200+0000","last_undegraded":"2024-09-16T10:24:49.616200+0000","last_fullsized":"2024-09-16T10:24:49.616200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6117,"log_dups_size":0,"ondisk_log_size":6117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25026560,"num_objects":6110,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2759,"num_write_kb":11036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6042","reported_seq":6202,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.614797+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:49.614797+0000","last_peered":"2024-09-16T10:24:49.614797+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:49.614797+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6042,"log_dups_size":0,"ondisk_log_size":6042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24727552,"num_objects":6037,"num_object_clones":0,"num_object_copies":12074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2667,"num_write_kb":10668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1820},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6338","reported_seq":6505,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.628397+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:49.628397+0000","last_peered":"2024-09-16T10:24:49.628397+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:49.628397+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6338,"log_dups_size":0,"ondisk_log_size":6338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25935872,"num_objects":6332,"num_object_clones":0,"num_object_copies":12664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2580,"num_write_kb":10320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1929},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6075","reported_seq":6247,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:49.530677+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:49.530677+0000","last_peered":"2024-09-16T10:24:49.530677+0000","last_clean":"2024-09-16T10:24:49.530677+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:49.530677+0000","last_undegraded":"2024-09-16T10:24:49.530677+0000","last_fullsized":"2024-09-16T10:24:49.530677+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24875008,"num_objects":6073,"num_object_clones":0,"num_object_copies":12146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1878,"num_write_kb":7512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6054","reported_seq":9183,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:50.539163+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:50.539163+0000","last_peered":"2024-09-16T10:24:50.539163+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:50.539163+0000","last_undegraded":"2024-09-16T10:24:50.539163+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6054,"log_dups_size":0,"ondisk_log_size":6054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24769478,"num_objects":6049,"num_object_clones":0,"num_object_copies":12098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2679,"num_objects_unfound":0,"num_objects_dirty":6049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":24192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1520,"num_bytes_recovered":6217728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2679},{"shards":"2","objects":3370}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.366075+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:55.366075+0000","last_peered":"2024-09-16T10:23:55.366075+0000","last_clean":"2024-09-16T10:23:55.366075+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:55.366075+0000","last_undegraded":"2024-09-16T10:23:55.366075+0000","last_fullsized":"2024-09-16T10:23:55.366075+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6319","reported_seq":6395,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.621234+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:49.621234+0000","last_peered":"2024-09-16T10:24:49.621234+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:49.621234+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25850822,"num_objects":6313,"num_object_clones":0,"num_object_copies":12626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6317,"num_write_kb":25248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1879},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6081","reported_seq":6183,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:50.474207+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:50.474207+0000","last_peered":"2024-09-16T10:24:50.474207+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:50.474207+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6081,"log_dups_size":0,"ondisk_log_size":6081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24888262,"num_objects":6078,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":24307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1801}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6120","reported_seq":6203,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:50.434160+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:50.434160+0000","last_peered":"2024-09-16T10:24:50.434160+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:50.434160+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6120,"log_dups_size":0,"ondisk_log_size":6120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25043910,"num_objects":6116,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6118,"num_write_kb":24459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":1924}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'5970","reported_seq":6046,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.547966+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:49.547966+0000","last_peered":"2024-09-16T10:24:49.547966+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:49.547966+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5970,"log_dups_size":0,"ondisk_log_size":5970,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24445411,"num_objects":5969,"num_object_clones":0,"num_object_copies":11938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":23873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1810},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6209","reported_seq":6339,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.563375+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:49.563375+0000","last_peered":"2024-09-16T10:24:49.563375+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:49.563375+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25400745,"num_objects":6204,"num_object_clones":0,"num_object_copies":12408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6206,"num_write_kb":24808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1867},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'5993","reported_seq":6070,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.596445+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:49.596445+0000","last_peered":"2024-09-16T10:24:49.596445+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:49.596445+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":5993,"log_dups_size":0,"ondisk_log_size":5993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24531910,"num_objects":5991,"num_object_clones":0,"num_object_copies":11982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5991,"num_write_kb":23958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1824},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6316","reported_seq":6405,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:49.609435+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:49.609435+0000","last_peered":"2024-09-16T10:24:49.609435+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:49.609435+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25823116,"num_objects":6308,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":25223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1913},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":402043382,"num_objects":98171,"num_object_clones":0,"num_object_copies":196342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2679,"num_objects_unfound":0,"num_objects_dirty":98171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75800,"num_write_kb":303088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1537,"num_bytes_recovered":6222307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":815607808,"data_stored":815517483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98235,"ondisk_log_size":98235,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018908,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517268,"kb_used_data":333068,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93854572,"statfs":{"total":96636764160,"available":96107081728,"internally_reserved":0,"allocated":341061632,"data_stored":340818393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":188608176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":2.4119999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":2.7869999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6739999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":215884,"kb_used_data":99780,"kb_used_omap":4,"kb_used_meta":116091,"kb_avail":94155956,"statfs":{"total":96636764160,"available":96415698944,"internally_reserved":0,"allocated":102174720,"data_stored":101851995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":118877866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5740000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3380000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541540,"kb_used_data":361500,"kb_used_omap":4,"kb_used_meta":180027,"kb_avail":93830300,"statfs":{"total":96636764160,"available":96082227200,"internally_reserved":0,"allocated":370176000,"data_stored":369936375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":184348328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.089}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.0489999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":371355648,"data_stored":371352035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103133184,"data_stored":103050085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":341118976,"data_stored":341115363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:55.537 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 103 20 101067 101047 3.83134 1.94141 0.0194467 0.0203833 2024-09-16T10:24:56.537 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 104 20 101628 101608 3.81557 2.19141 0.0495674 0.020467 2024-09-16T10:24:57.537 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 105 20 102175 102155 3.79958 2.13672 0.0194443 0.020554 2024-09-16T10:24:58.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:58.537 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 106 20 102715 102695 3.78363 2.10938 0.020232 0.0206403 2024-09-16T10:24:58.916 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:58.917 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:58.939 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-16T10:24:58.589783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":412901897,"num_objects":100824,"num_object_clones":0,"num_object_copies":201648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2520,"num_objects_unfound":0,"num_objects_dirty":100824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78453,"num_write_kb":313694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1697,"num_bytes_recovered":6873590,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100889,"ondisk_log_size":100889,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1280032,"kb_used_data":816072,"kb_used_omap":12,"kb_used_meta":463923,"kb_avail":281835488,"statfs":{"total":289910292480,"available":288599539712,"internally_reserved":0,"allocated":835657728,"data_stored":834852139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":475057154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,5,9,7,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21762048,"num_objects":5313,"num_object_clones":0,"num_object_copies":10626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-319,"num_objects_unfound":0,"num_objects_dirty":5313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5313,"num_write_kb":21252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":1306624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003085"},"pg_stats":[{"pgid":"2.f","version":"42'6272","reported_seq":6338,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.582540+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:54.582540+0000","last_peered":"2024-09-16T10:24:54.582540+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:54.582540+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6272,"log_dups_size":0,"ondisk_log_size":6272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":6272,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":25088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2080},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6303","reported_seq":6457,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:55.521437+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:55.521437+0000","last_peered":"2024-09-16T10:24:55.521437+0000","last_clean":"2024-09-16T10:24:55.521437+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:55.521437+0000","last_undegraded":"2024-09-16T10:24:55.521437+0000","last_fullsized":"2024-09-16T10:24:55.521437+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25792512,"num_objects":6297,"num_object_clones":0,"num_object_copies":12594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":10064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6372","reported_seq":6554,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:55.583548+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:55.583548+0000","last_peered":"2024-09-16T10:24:55.583548+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:55.583548+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6372,"log_dups_size":0,"ondisk_log_size":6372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":6368,"num_object_clones":0,"num_object_copies":12736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":9716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6322","reported_seq":6375,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:55.551437+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:55.551437+0000","last_peered":"2024-09-16T10:24:55.551437+0000","last_clean":"2024-09-16T10:24:55.551437+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:55.551437+0000","last_undegraded":"2024-09-16T10:24:55.551437+0000","last_fullsized":"2024-09-16T10:24:55.551437+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25894912,"num_objects":6322,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":25288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6282","reported_seq":6436,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:54.521140+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:54.521140+0000","last_peered":"2024-09-16T10:24:54.521140+0000","last_clean":"2024-09-16T10:24:54.521140+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:54.521140+0000","last_undegraded":"2024-09-16T10:24:54.521140+0000","last_fullsized":"2024-09-16T10:24:54.521140+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25702400,"num_objects":6275,"num_object_clones":0,"num_object_copies":12550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":11696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6200","reported_seq":6360,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.620463+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:54.620463+0000","last_peered":"2024-09-16T10:24:54.620463+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:54.620463+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6200,"log_dups_size":0,"ondisk_log_size":6200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25374720,"num_objects":6195,"num_object_clones":0,"num_object_copies":12390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2825,"num_write_kb":11300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1978},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6486","reported_seq":6656,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.387751+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:54.387751+0000","last_peered":"2024-09-16T10:24:54.387751+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:54.387751+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":6480,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2728,"num_write_kb":10912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2077},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6232","reported_seq":6404,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:54.614736+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:54.614736+0000","last_peered":"2024-09-16T10:24:54.614736+0000","last_clean":"2024-09-16T10:24:54.614736+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:54.614736+0000","last_undegraded":"2024-09-16T10:24:54.614736+0000","last_fullsized":"2024-09-16T10:24:54.614736+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6232,"log_dups_size":0,"ondisk_log_size":6232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25518080,"num_objects":6230,"num_object_clones":0,"num_object_copies":12460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2035,"num_write_kb":8140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6214","reported_seq":9661,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:55.576488+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:55.576488+0000","last_peered":"2024-09-16T10:24:55.576488+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:55.576488+0000","last_undegraded":"2024-09-16T10:24:55.576488+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6214,"log_dups_size":0,"ondisk_log_size":6214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25424838,"num_objects":6209,"num_object_clones":0,"num_object_copies":12418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2520,"num_objects_unfound":0,"num_objects_dirty":6209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6212,"num_write_kb":24832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1679,"num_bytes_recovered":6868992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2520},{"shards":"2","objects":3689}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6475","reported_seq":6550,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.414520+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:54.414520+0000","last_peered":"2024-09-16T10:24:54.414520+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:54.414520+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6475,"log_dups_size":0,"ondisk_log_size":6475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26489798,"num_objects":6469,"num_object_clones":0,"num_object_copies":12938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6473,"num_write_kb":25872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2035},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6259","reported_seq":6361,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:55.562238+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:55.562238+0000","last_peered":"2024-09-16T10:24:55.562238+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:55.562238+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25617350,"num_objects":6256,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1979}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6304","reported_seq":6385,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:55.509002+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:55.509002+0000","last_peered":"2024-09-16T10:24:55.509002+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:55.509002+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6304,"log_dups_size":0,"ondisk_log_size":6304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25797574,"num_objects":6300,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6302,"num_write_kb":25195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6121","reported_seq":6197,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.580326+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:54.580326+0000","last_peered":"2024-09-16T10:24:54.580326+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:54.580326+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6121,"log_dups_size":0,"ondisk_log_size":6121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25063907,"num_objects":6120,"num_object_clones":0,"num_object_copies":12240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":24477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1961},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6396","reported_seq":6526,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.574781+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:54.574781+0000","last_peered":"2024-09-16T10:24:54.574781+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:54.574781+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26166697,"num_objects":6391,"num_object_clones":0,"num_object_copies":12782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":25556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2054},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6161","reported_seq":6238,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.594088+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:54.594088+0000","last_peered":"2024-09-16T10:24:54.594088+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:54.594088+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25220038,"num_objects":6159,"num_object_clones":0,"num_object_copies":12318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6159,"num_write_kb":24630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1992},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6487","reported_seq":6576,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.652098+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:54.652098+0000","last_peered":"2024-09-16T10:24:54.652098+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:54.652098+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6487,"log_dups_size":0,"ondisk_log_size":6487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26523532,"num_objects":6479,"num_object_clones":0,"num_object_copies":12958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6484,"num_write_kb":25907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2084},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":412901878,"num_objects":100822,"num_object_clones":0,"num_object_copies":201644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2520,"num_objects_unfound":0,"num_objects_dirty":100822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78451,"num_write_kb":313692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":6873571,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":837836800,"data_stored":837746475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100886,"ondisk_log_size":100886,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018909,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510268,"kb_used_data":338356,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93861572,"statfs":{"total":96636764160,"available":96114249728,"internally_reserved":0,"allocated":346476544,"data_stored":346233305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":176025264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,3,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":212020,"kb_used_data":108204,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94159820,"statfs":{"total":96636764160,"available":96419655680,"internally_reserved":0,"allocated":110800896,"data_stored":110478171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":106294954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0140000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8149999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215137,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557744,"kb_used_data":369512,"kb_used_omap":4,"kb_used_meta":188219,"kb_avail":93814096,"statfs":{"total":96636764160,"available":96065634304,"internally_reserved":0,"allocated":378380288,"data_stored":378140663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":192736936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,5,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9249999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":2.8620000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.8839999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379416576,"data_stored":379412963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111841280,"data_stored":111758181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346578944,"data_stored":346575331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:58.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:24:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:24:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:24:59.538 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 107 20 103199 103179 3.76594 1.89062 0.0631412 0.0207359 2024-09-16T10:24:59.565 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-16T10:24:58.589783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":412901897,"num_objects":100824,"num_object_clones":0,"num_object_copies":201648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2520,"num_objects_unfound":0,"num_objects_dirty":100824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78453,"num_write_kb":313694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1697,"num_bytes_recovered":6873590,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100889,"ondisk_log_size":100889,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1280032,"kb_used_data":816072,"kb_used_omap":12,"kb_used_meta":463923,"kb_avail":281835488,"statfs":{"total":289910292480,"available":288599539712,"internally_reserved":0,"allocated":835657728,"data_stored":834852139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":475057154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,5,9,7,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21762048,"num_objects":5313,"num_object_clones":0,"num_object_copies":10626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-319,"num_objects_unfound":0,"num_objects_dirty":5313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5313,"num_write_kb":21252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":1306624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003085"},"pg_stats":[{"pgid":"2.f","version":"42'6272","reported_seq":6338,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.582540+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:54.582540+0000","last_peered":"2024-09-16T10:24:54.582540+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:54.582540+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6272,"log_dups_size":0,"ondisk_log_size":6272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":6272,"num_object_clones":0,"num_object_copies":12544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6272,"num_write_kb":25088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2080},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6303","reported_seq":6457,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:55.521437+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:55.521437+0000","last_peered":"2024-09-16T10:24:55.521437+0000","last_clean":"2024-09-16T10:24:55.521437+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:55.521437+0000","last_undegraded":"2024-09-16T10:24:55.521437+0000","last_fullsized":"2024-09-16T10:24:55.521437+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25792512,"num_objects":6297,"num_object_clones":0,"num_object_copies":12594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":10064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6372","reported_seq":6554,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:55.583548+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:55.583548+0000","last_peered":"2024-09-16T10:24:55.583548+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:55.583548+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6372,"log_dups_size":0,"ondisk_log_size":6372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":6368,"num_object_clones":0,"num_object_copies":12736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":9716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6322","reported_seq":6375,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:55.551437+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:55.551437+0000","last_peered":"2024-09-16T10:24:55.551437+0000","last_clean":"2024-09-16T10:24:55.551437+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:55.551437+0000","last_undegraded":"2024-09-16T10:24:55.551437+0000","last_fullsized":"2024-09-16T10:24:55.551437+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25894912,"num_objects":6322,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":25288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6282","reported_seq":6436,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:54.521140+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:54.521140+0000","last_peered":"2024-09-16T10:24:54.521140+0000","last_clean":"2024-09-16T10:24:54.521140+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:54.521140+0000","last_undegraded":"2024-09-16T10:24:54.521140+0000","last_fullsized":"2024-09-16T10:24:54.521140+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25702400,"num_objects":6275,"num_object_clones":0,"num_object_copies":12550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":11696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6200","reported_seq":6360,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.620463+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:54.620463+0000","last_peered":"2024-09-16T10:24:54.620463+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:54.620463+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6200,"log_dups_size":0,"ondisk_log_size":6200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25374720,"num_objects":6195,"num_object_clones":0,"num_object_copies":12390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2825,"num_write_kb":11300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1978},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6486","reported_seq":6656,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.387751+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:54.387751+0000","last_peered":"2024-09-16T10:24:54.387751+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:54.387751+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6486,"log_dups_size":0,"ondisk_log_size":6486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":6480,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2728,"num_write_kb":10912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2077},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6232","reported_seq":6404,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:54.614736+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:54.614736+0000","last_peered":"2024-09-16T10:24:54.614736+0000","last_clean":"2024-09-16T10:24:54.614736+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:54.614736+0000","last_undegraded":"2024-09-16T10:24:54.614736+0000","last_fullsized":"2024-09-16T10:24:54.614736+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6232,"log_dups_size":0,"ondisk_log_size":6232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25518080,"num_objects":6230,"num_object_clones":0,"num_object_copies":12460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2035,"num_write_kb":8140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6214","reported_seq":9661,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:55.576488+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:55.576488+0000","last_peered":"2024-09-16T10:24:55.576488+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:55.576488+0000","last_undegraded":"2024-09-16T10:24:55.576488+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6214,"log_dups_size":0,"ondisk_log_size":6214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25424838,"num_objects":6209,"num_object_clones":0,"num_object_copies":12418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2520,"num_objects_unfound":0,"num_objects_dirty":6209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6212,"num_write_kb":24832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1679,"num_bytes_recovered":6868992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2520},{"shards":"2","objects":3689}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6475","reported_seq":6550,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.414520+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:54.414520+0000","last_peered":"2024-09-16T10:24:54.414520+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:54.414520+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6475,"log_dups_size":0,"ondisk_log_size":6475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26489798,"num_objects":6469,"num_object_clones":0,"num_object_copies":12938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6473,"num_write_kb":25872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2035},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6259","reported_seq":6361,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:55.562238+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:55.562238+0000","last_peered":"2024-09-16T10:24:55.562238+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:55.562238+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25617350,"num_objects":6256,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1979}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6304","reported_seq":6385,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:55.509002+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:55.509002+0000","last_peered":"2024-09-16T10:24:55.509002+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:55.509002+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6304,"log_dups_size":0,"ondisk_log_size":6304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25797574,"num_objects":6300,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6302,"num_write_kb":25195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6121","reported_seq":6197,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.580326+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:54.580326+0000","last_peered":"2024-09-16T10:24:54.580326+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:54.580326+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6121,"log_dups_size":0,"ondisk_log_size":6121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25063907,"num_objects":6120,"num_object_clones":0,"num_object_copies":12240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6120,"num_write_kb":24477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1961},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6396","reported_seq":6526,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.574781+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:54.574781+0000","last_peered":"2024-09-16T10:24:54.574781+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:54.574781+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6396,"log_dups_size":0,"ondisk_log_size":6396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26166697,"num_objects":6391,"num_object_clones":0,"num_object_copies":12782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6393,"num_write_kb":25556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2054},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6161","reported_seq":6238,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.594088+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:54.594088+0000","last_peered":"2024-09-16T10:24:54.594088+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:54.594088+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6161,"log_dups_size":0,"ondisk_log_size":6161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25220038,"num_objects":6159,"num_object_clones":0,"num_object_copies":12318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6159,"num_write_kb":24630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1992},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6487","reported_seq":6576,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:54.652098+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:54.652098+0000","last_peered":"2024-09-16T10:24:54.652098+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:54.652098+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6487,"log_dups_size":0,"ondisk_log_size":6487,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26523532,"num_objects":6479,"num_object_clones":0,"num_object_copies":12958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6484,"num_write_kb":25907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2084},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":412901878,"num_objects":100822,"num_object_clones":0,"num_object_copies":201644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2520,"num_objects_unfound":0,"num_objects_dirty":100822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78451,"num_write_kb":313692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":6873571,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":837836800,"data_stored":837746475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100886,"ondisk_log_size":100886,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018909,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510268,"kb_used_data":338356,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93861572,"statfs":{"total":96636764160,"available":96114249728,"internally_reserved":0,"allocated":346476544,"data_stored":346233305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":176025264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,3,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":212020,"kb_used_data":108204,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94159820,"statfs":{"total":96636764160,"available":96419655680,"internally_reserved":0,"allocated":110800896,"data_stored":110478171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":106294954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0140000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8149999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215137,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557744,"kb_used_data":369512,"kb_used_omap":4,"kb_used_meta":188219,"kb_avail":93814096,"statfs":{"total":96636764160,"available":96065634304,"internally_reserved":0,"allocated":378380288,"data_stored":378140663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":192736936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,5,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.9249999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:23:58 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":2.8620000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.163,"15min":1.163},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.5030000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.8839999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":379416576,"data_stored":379412963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111841280,"data_stored":111758181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346578944,"data_stored":346575331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:24:59.567 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:24:59.567 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'6272", 'reported_seq': 6338, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:54.582540+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:54.582540+0000', 'last_peered': '2024-09-16T10:24:54.582540+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:54.582540+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6272, 'log_dups_size': 0, 'ondisk_log_size': 6272, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25690112, 'num_objects': 6272, 'num_object_clones': 0, 'num_object_copies': 12544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6272, 'num_write_kb': 25088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2080}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:59.567 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:24:59.568 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'6372", 'reported_seq': 6554, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:55.583548+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:24:55.583548+0000', 'last_peered': '2024-09-16T10:24:55.583548+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:24:55.583548+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 6372, 'log_dups_size': 0, 'ondisk_log_size': 6372, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26083328, 'num_objects': 6368, 'num_object_clones': 0, 'num_object_copies': 12736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2429, 'num_write_kb': 9716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 2033}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:59.568 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:24:59.568 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'6200", 'reported_seq': 6360, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:54.620463+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:54.620463+0000', 'last_peered': '2024-09-16T10:24:54.620463+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:54.620463+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 6200, 'log_dups_size': 0, 'ondisk_log_size': 6200, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25374720, 'num_objects': 6195, 'num_object_clones': 0, 'num_object_copies': 12390, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2825, 'num_write_kb': 11300, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1978}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:59.568 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:24:59.569 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'6486", 'reported_seq': 6656, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:54.387751+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:54.387751+0000', 'last_peered': '2024-09-16T10:24:54.387751+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:54.387751+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 6486, 'log_dups_size': 0, 'ondisk_log_size': 6486, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26542080, 'num_objects': 6480, 'num_object_clones': 0, 'num_object_copies': 12960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2728, 'num_write_kb': 10912, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2077}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:59.569 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:24:59.569 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'6214", 'reported_seq': 9661, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:24:55.576488+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:24:55.576488+0000', 'last_peered': '2024-09-16T10:24:55.576488+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:24:55.576488+0000', 'last_undegraded': '2024-09-16T10:24:55.576488+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6214, 'log_dups_size': 0, 'ondisk_log_size': 6214, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25424838, 'num_objects': 6209, 'num_object_clones': 0, 'num_object_copies': 12418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2520, 'num_objects_unfound': 0, 'num_objects_dirty': 6209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6212, 'num_write_kb': 24832, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1679, 'num_bytes_recovered': 6868992, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2520}, {'shards': '2', 'objects': 3689}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:59.569 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:24:59.569 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'6475", 'reported_seq': 6550, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:54.414520+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:54.414520+0000', 'last_peered': '2024-09-16T10:24:54.414520+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:54.414520+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6475, 'log_dups_size': 0, 'ondisk_log_size': 6475, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26489798, 'num_objects': 6469, 'num_object_clones': 0, 'num_object_copies': 12938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6473, 'num_write_kb': 25872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2035}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:59.570 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:24:59.570 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'6259", 'reported_seq': 6361, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:55.562238+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:24:55.562238+0000', 'last_peered': '2024-09-16T10:24:55.562238+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:24:55.562238+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 6259, 'log_dups_size': 0, 'ondisk_log_size': 6259, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25617350, 'num_objects': 6256, 'num_object_clones': 0, 'num_object_copies': 12512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6257, 'num_write_kb': 25019, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 1979}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:59.570 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:24:59.570 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'6304", 'reported_seq': 6385, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:55.509002+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:24:55.509002+0000', 'last_peered': '2024-09-16T10:24:55.509002+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:24:55.509002+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6304, 'log_dups_size': 0, 'ondisk_log_size': 6304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25797574, 'num_objects': 6300, 'num_object_clones': 0, 'num_object_copies': 12600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6302, 'num_write_kb': 25195, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 2108}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:24:59.570 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:24:59.570 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'6121", 'reported_seq': 6197, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:54.580326+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:54.580326+0000', 'last_peered': '2024-09-16T10:24:54.580326+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:54.580326+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6121, 'log_dups_size': 0, 'ondisk_log_size': 6121, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25063907, 'num_objects': 6120, 'num_object_clones': 0, 'num_object_copies': 12240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6120, 'num_write_kb': 24477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1961}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:59.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:24:59.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'6396", 'reported_seq': 6526, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:54.574781+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:54.574781+0000', 'last_peered': '2024-09-16T10:24:54.574781+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:54.574781+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 6396, 'log_dups_size': 0, 'ondisk_log_size': 6396, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26166697, 'num_objects': 6391, 'num_object_clones': 0, 'num_object_copies': 12782, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6393, 'num_write_kb': 25556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2054}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:59.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:24:59.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'6161", 'reported_seq': 6238, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:54.594088+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:54.594088+0000', 'last_peered': '2024-09-16T10:24:54.594088+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:54.594088+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6161, 'log_dups_size': 0, 'ondisk_log_size': 6161, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25220038, 'num_objects': 6159, 'num_object_clones': 0, 'num_object_copies': 12318, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6159, 'num_write_kb': 24630, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1992}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:59.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:24:59.571 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'6487", 'reported_seq': 6576, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:54.652098+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:54.652098+0000', 'last_peered': '2024-09-16T10:24:54.652098+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:54.652098+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6487, 'log_dups_size': 0, 'ondisk_log_size': 6487, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26523532, 'num_objects': 6479, 'num_object_clones': 0, 'num_object_copies': 12958, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6484, 'num_write_kb': 25907, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2084}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:24:59.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:00.249 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:00.250 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/201648 objects degraded (23.371%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":189,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":100824,"data_bytes":412901897,"bytes_used":1310752768,"bytes_avail":288599539712,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":201648,"degraded_ratio":0.23371419503292867,"misplaced_objects":2520,"misplaced_total":201648,"misplaced_ratio":0.012497024517971911,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":108857,"recovering_keys_per_sec":0,"num_objects_recovered":319,"num_bytes_recovered":1306624,"num_keys_recovered":0,"write_bytes_sec":1813037,"read_op_per_sec":0,"write_op_per_sec":442},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:00.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:00.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:00.538 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 108 19 103681 103662 3.74854 1.88672 0.0228047 0.0208339 2024-09-16T10:25:00.865 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:00.867 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:00.888 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-09-16T10:25:00.590252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":419774985,"num_objects":102502,"num_object_clones":0,"num_object_copies":205004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2520,"num_objects_unfound":0,"num_objects_dirty":102502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80131,"num_write_kb":320406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1697,"num_bytes_recovered":6873590,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102567,"ondisk_log_size":102567,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1293852,"kb_used_data":824132,"kb_used_omap":12,"kb_used_meta":469683,"kb_avail":281821668,"statfs":{"total":289910292480,"available":288585388032,"internally_reserved":0,"allocated":843911168,"data_stored":843105579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":480955394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,4,5,6,12,14],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28635136,"num_objects":6991,"num_object_clones":0,"num_object_copies":13982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-319,"num_objects_unfound":0,"num_objects_dirty":6991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6991,"num_write_kb":27964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":1306624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003056"},"pg_stats":[{"pgid":"2.f","version":"42'6435","reported_seq":6499,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.727334+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:59.727334+0000","last_peered":"2024-09-16T10:24:59.727334+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:59.727334+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6435,"log_dups_size":0,"ondisk_log_size":6435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26357760,"num_objects":6435,"num_object_clones":0,"num_object_copies":12870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":25740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2243},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6303","reported_seq":6457,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:55.521437+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:24:55.521437+0000","last_peered":"2024-09-16T10:24:55.521437+0000","last_clean":"2024-09-16T10:24:55.521437+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:24:55.521437+0000","last_undegraded":"2024-09-16T10:24:55.521437+0000","last_fullsized":"2024-09-16T10:24:55.521437+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25792512,"num_objects":6297,"num_object_clones":0,"num_object_copies":12594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2516,"num_write_kb":10064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6372","reported_seq":6554,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:55.583548+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:24:55.583548+0000","last_peered":"2024-09-16T10:24:55.583548+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:24:55.583548+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6372,"log_dups_size":0,"ondisk_log_size":6372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":6368,"num_object_clones":0,"num_object_copies":12736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":9716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2033}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6322","reported_seq":6375,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:55.551437+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:24:55.551437+0000","last_peered":"2024-09-16T10:24:55.551437+0000","last_clean":"2024-09-16T10:24:55.551437+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:24:55.551437+0000","last_undegraded":"2024-09-16T10:24:55.551437+0000","last_fullsized":"2024-09-16T10:24:55.551437+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25894912,"num_objects":6322,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":25288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6446","reported_seq":6600,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:59.568906+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:59.568906+0000","last_peered":"2024-09-16T10:24:59.568906+0000","last_clean":"2024-09-16T10:24:59.568906+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:59.568906+0000","last_undegraded":"2024-09-16T10:24:59.568906+0000","last_fullsized":"2024-09-16T10:24:59.568906+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26374144,"num_objects":6439,"num_object_clones":0,"num_object_copies":12878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":12352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6364","reported_seq":6525,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.666191+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:59.666191+0000","last_peered":"2024-09-16T10:24:59.666191+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:59.666191+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26046464,"num_objects":6359,"num_object_clones":0,"num_object_copies":12718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":11956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2142},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6659","reported_seq":6829,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.690215+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:59.690215+0000","last_peered":"2024-09-16T10:24:59.690215+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:59.690215+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6659,"log_dups_size":0,"ondisk_log_size":6659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27250688,"num_objects":6653,"num_object_clones":0,"num_object_copies":13306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2901,"num_write_kb":11604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2250},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6399","reported_seq":6571,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:59.617062+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:59.617062+0000","last_peered":"2024-09-16T10:24:59.617062+0000","last_clean":"2024-09-16T10:24:59.617062+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:59.617062+0000","last_undegraded":"2024-09-16T10:24:59.617062+0000","last_fullsized":"2024-09-16T10:24:59.617062+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26202112,"num_objects":6397,"num_object_clones":0,"num_object_copies":12794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":8808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6214","reported_seq":9661,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:24:55.576488+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:24:55.576488+0000","last_peered":"2024-09-16T10:24:55.576488+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:24:55.576488+0000","last_undegraded":"2024-09-16T10:24:55.576488+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6214,"log_dups_size":0,"ondisk_log_size":6214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25424838,"num_objects":6209,"num_object_clones":0,"num_object_copies":12418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2520,"num_objects_unfound":0,"num_objects_dirty":6209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6212,"num_write_kb":24832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1679,"num_bytes_recovered":6868992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2520},{"shards":"2","objects":3689}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6647","reported_seq":6723,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.652877+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:59.652877+0000","last_peered":"2024-09-16T10:24:59.652877+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:59.652877+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6647,"log_dups_size":0,"ondisk_log_size":6647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27194310,"num_objects":6641,"num_object_clones":0,"num_object_copies":13282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6645,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2207},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6259","reported_seq":6361,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:55.562238+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:24:55.562238+0000","last_peered":"2024-09-16T10:24:55.562238+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:24:55.562238+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25617350,"num_objects":6256,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6257,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":1979}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6304","reported_seq":6385,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:55.509002+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:24:55.509002+0000","last_peered":"2024-09-16T10:24:55.509002+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:24:55.509002+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6304,"log_dups_size":0,"ondisk_log_size":6304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25797574,"num_objects":6300,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6302,"num_write_kb":25195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6295","reported_seq":6370,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.729363+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:59.729363+0000","last_peered":"2024-09-16T10:24:59.729363+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:59.729363+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25776611,"num_objects":6294,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":25173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2135},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6589","reported_seq":6719,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.684344+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:59.684344+0000","last_peered":"2024-09-16T10:24:59.684344+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:59.684344+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6589,"log_dups_size":0,"ondisk_log_size":6589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26957225,"num_objects":6584,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6586,"num_write_kb":26328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2247},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":53,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-16T10:23:55.369308+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:55.369308+0000","last_peered":"2024-09-16T10:23:55.369308+0000","last_clean":"2024-09-16T10:23:55.369308+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:55.369308+0000","last_undegraded":"2024-09-16T10:23:55.369308+0000","last_fullsized":"2024-09-16T10:23:55.369308+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6322","reported_seq":6400,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.678650+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:59.678650+0000","last_peered":"2024-09-16T10:24:59.678650+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:59.678650+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25879494,"num_objects":6320,"num_object_clones":0,"num_object_copies":12640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":25274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2153},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6634","reported_seq":6724,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.698084+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:59.698084+0000","last_peered":"2024-09-16T10:24:59.698084+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:59.698084+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6634,"log_dups_size":0,"ondisk_log_size":6634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27125644,"num_objects":6626,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6631,"num_write_kb":26495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2231},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":419774966,"num_objects":102500,"num_object_clones":0,"num_object_copies":205000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2520,"num_objects_unfound":0,"num_objects_dirty":102500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80129,"num_write_kb":320404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1696,"num_bytes_recovered":6873571,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":846155776,"data_stored":846065451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102564,"ondisk_log_size":102564,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018909,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510268,"kb_used_data":338356,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93861572,"statfs":{"total":96636764160,"available":96114249728,"internally_reserved":0,"allocated":346476544,"data_stored":346233305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":176025264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,3,1,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":3.992,"5min":3.992,"15min":3.992},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":212020,"kb_used_data":108204,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94159820,"statfs":{"total":96636764160,"available":96419655680,"internally_reserved":0,"allocated":110800896,"data_stored":110478171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":106294954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0140000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8149999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215138,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571564,"kb_used_data":377572,"kb_used_omap":4,"kb_used_meta":193979,"kb_avail":93800276,"statfs":{"total":96636764160,"available":96051482624,"internally_reserved":0,"allocated":386633728,"data_stored":386394103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":198635176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,2,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":6.7380000000000004},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.5760000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":9.5060000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":387735552,"data_stored":387731939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111841280,"data_stored":111758181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":346578944,"data_stored":346575331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:01.538 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 109 20 104217 104197 3.73332 2.08984 0.0124221 0.0209195 2024-09-16T10:25:02.538 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 110 20 104761 104741 3.7187 2.125 0.0320416 0.0210017 2024-09-16T10:25:03.538 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 111 20 105300 105280 3.70417 2.10547 0.0841335 0.0210832 2024-09-16T10:25:03.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:04.461 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:04.462 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:04.489 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-16T10:25:02.590757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":423657993,"num_objects":103450,"num_object_clones":0,"num_object_copies":206900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2358,"num_objects_unfound":0,"num_objects_dirty":103450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81079,"num_write_kb":324198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1859,"num_bytes_recovered":7537142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103515,"ondisk_log_size":103515,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1303104,"kb_used_data":829288,"kb_used_omap":12,"kb_used_meta":473779,"kb_avail":281812416,"statfs":{"total":289910292480,"available":288575913984,"internally_reserved":0,"allocated":849190912,"data_stored":848385323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":485149698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,7,10,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21614592,"num_objects":5277,"num_object_clones":0,"num_object_copies":10554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-321,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5277,"num_write_kb":21108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":1314816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003019"},"pg_stats":[{"pgid":"2.f","version":"42'6435","reported_seq":6499,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.727334+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:59.727334+0000","last_peered":"2024-09-16T10:24:59.727334+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:59.727334+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6435,"log_dups_size":0,"ondisk_log_size":6435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26357760,"num_objects":6435,"num_object_clones":0,"num_object_copies":12870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":25740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2243},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6487","reported_seq":6641,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:00.544959+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:00.544959+0000","last_peered":"2024-09-16T10:25:00.544959+0000","last_clean":"2024-09-16T10:25:00.544959+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:00.544959+0000","last_undegraded":"2024-09-16T10:25:00.544959+0000","last_fullsized":"2024-09-16T10:25:00.544959+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6487,"log_dups_size":0,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26546176,"num_objects":6481,"num_object_clones":0,"num_object_copies":12962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2700,"num_write_kb":10800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6522","reported_seq":6704,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:00.607137+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:00.607137+0000","last_peered":"2024-09-16T10:25:00.607137+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:00.607137+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26697728,"num_objects":6518,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":10316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2183}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6483","reported_seq":6536,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:00.583725+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:00.583725+0000","last_peered":"2024-09-16T10:25:00.583725+0000","last_clean":"2024-09-16T10:25:00.583725+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:00.583725+0000","last_undegraded":"2024-09-16T10:25:00.583725+0000","last_fullsized":"2024-09-16T10:25:00.583725+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6483,"log_dups_size":0,"ondisk_log_size":6483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554368,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6483,"num_write_kb":25932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6446","reported_seq":6600,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:59.568906+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:59.568906+0000","last_peered":"2024-09-16T10:24:59.568906+0000","last_clean":"2024-09-16T10:24:59.568906+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:59.568906+0000","last_undegraded":"2024-09-16T10:24:59.568906+0000","last_fullsized":"2024-09-16T10:24:59.568906+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26374144,"num_objects":6439,"num_object_clones":0,"num_object_copies":12878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":12352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6364","reported_seq":6525,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.666191+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:59.666191+0000","last_peered":"2024-09-16T10:24:59.666191+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:59.666191+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26046464,"num_objects":6359,"num_object_clones":0,"num_object_copies":12718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":11956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2142},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6659","reported_seq":6829,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.690215+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:59.690215+0000","last_peered":"2024-09-16T10:24:59.690215+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:59.690215+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6659,"log_dups_size":0,"ondisk_log_size":6659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27250688,"num_objects":6653,"num_object_clones":0,"num_object_copies":13306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2901,"num_write_kb":11604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2250},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6399","reported_seq":6571,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:59.617062+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:59.617062+0000","last_peered":"2024-09-16T10:24:59.617062+0000","last_clean":"2024-09-16T10:24:59.617062+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:59.617062+0000","last_undegraded":"2024-09-16T10:24:59.617062+0000","last_fullsized":"2024-09-16T10:24:59.617062+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26202112,"num_objects":6397,"num_object_clones":0,"num_object_copies":12794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":8808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6371","reported_seq":10142,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:00.596333+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:00.596333+0000","last_peered":"2024-09-16T10:25:00.596333+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:00.596333+0000","last_undegraded":"2024-09-16T10:25:00.596333+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26067910,"num_objects":6366,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2358,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6369,"num_write_kb":25460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1841,"num_bytes_recovered":7532544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2358},{"shards":"2","objects":4008}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6647","reported_seq":6723,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.652877+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:59.652877+0000","last_peered":"2024-09-16T10:24:59.652877+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:59.652877+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6647,"log_dups_size":0,"ondisk_log_size":6647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27194310,"num_objects":6641,"num_object_clones":0,"num_object_copies":13282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6645,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2207},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6402","reported_seq":6505,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:00.574470+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:00.574470+0000","last_peered":"2024-09-16T10:25:00.574470+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:00.574470+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26203078,"num_objects":6399,"num_object_clones":0,"num_object_copies":12798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6400,"num_write_kb":25591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6457","reported_seq":6541,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:00.523848+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:00.523848+0000","last_peered":"2024-09-16T10:25:00.523848+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:00.523848+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6457,"log_dups_size":0,"ondisk_log_size":6457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26424262,"num_objects":6453,"num_object_clones":0,"num_object_copies":12906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6455,"num_write_kb":25807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6295","reported_seq":6370,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.729363+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:59.729363+0000","last_peered":"2024-09-16T10:24:59.729363+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:59.729363+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25776611,"num_objects":6294,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":25173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2135},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6589","reported_seq":6719,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.684344+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:59.684344+0000","last_peered":"2024-09-16T10:24:59.684344+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:59.684344+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6589,"log_dups_size":0,"ondisk_log_size":6589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26957225,"num_objects":6584,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6586,"num_write_kb":26328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2247},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6322","reported_seq":6400,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.678650+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:59.678650+0000","last_peered":"2024-09-16T10:24:59.678650+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:59.678650+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25879494,"num_objects":6320,"num_object_clones":0,"num_object_copies":12640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":25274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2153},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6634","reported_seq":6724,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.698084+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:59.698084+0000","last_peered":"2024-09-16T10:24:59.698084+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:59.698084+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6634,"log_dups_size":0,"ondisk_log_size":6634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27125644,"num_objects":6626,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6631,"num_write_kb":26495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2231},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":423657974,"num_objects":103448,"num_object_clones":0,"num_object_copies":206896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2358,"num_objects_unfound":0,"num_objects_dirty":103448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81077,"num_write_kb":324196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1858,"num_bytes_recovered":7537123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":851378176,"data_stored":851287851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103512,"ondisk_log_size":103512,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018910,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519520,"kb_used_data":343512,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93852320,"statfs":{"total":96636764160,"available":96104775680,"internally_reserved":0,"allocated":351756288,"data_stored":351513049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":180219568},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":1.423},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.51600000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.4430000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":212020,"kb_used_data":108204,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94159820,"statfs":{"total":96636764160,"available":96419655680,"internally_reserved":0,"allocated":110800896,"data_stored":110478171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":106294954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.0140000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8149999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215138,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571564,"kb_used_data":377572,"kb_used_omap":4,"kb_used_meta":193979,"kb_avail":93800276,"statfs":{"total":96636764160,"available":96051482624,"internally_reserved":0,"allocated":386633728,"data_stored":386394103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":198635176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,2,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":6.7380000000000004},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.5760000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":9.5060000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":387735552,"data_stored":387731939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111841280,"data_stored":111758181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351801344,"data_stored":351797731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:04.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:04.538 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 112 20 105800 105780 3.68853 1.95312 0.0222188 0.0211707 2024-09-16T10:25:05.041 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:05.043 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:05.069 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-16T10:25:04.591277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":423657993,"num_objects":103450,"num_object_clones":0,"num_object_copies":206900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2358,"num_objects_unfound":0,"num_objects_dirty":103450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81079,"num_write_kb":324198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1859,"num_bytes_recovered":7537142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103515,"ondisk_log_size":103515,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1319828,"kb_used_data":837820,"kb_used_omap":12,"kb_used_meta":481971,"kb_avail":281795692,"statfs":{"total":289910292480,"available":288558788608,"internally_reserved":0,"allocated":857927680,"data_stored":857122091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":493538306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,5,6,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21614592,"num_objects":5277,"num_object_clones":0,"num_object_copies":10554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-321,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5277,"num_write_kb":21108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":1314816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002999"},"pg_stats":[{"pgid":"2.f","version":"42'6435","reported_seq":6499,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.727334+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:59.727334+0000","last_peered":"2024-09-16T10:24:59.727334+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:59.727334+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6435,"log_dups_size":0,"ondisk_log_size":6435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26357760,"num_objects":6435,"num_object_clones":0,"num_object_copies":12870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":25740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2243},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6487","reported_seq":6641,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:00.544959+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:00.544959+0000","last_peered":"2024-09-16T10:25:00.544959+0000","last_clean":"2024-09-16T10:25:00.544959+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:00.544959+0000","last_undegraded":"2024-09-16T10:25:00.544959+0000","last_fullsized":"2024-09-16T10:25:00.544959+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6487,"log_dups_size":0,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26546176,"num_objects":6481,"num_object_clones":0,"num_object_copies":12962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2700,"num_write_kb":10800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6522","reported_seq":6704,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:00.607137+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:00.607137+0000","last_peered":"2024-09-16T10:25:00.607137+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:00.607137+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26697728,"num_objects":6518,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":10316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2183}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6483","reported_seq":6536,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:00.583725+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:00.583725+0000","last_peered":"2024-09-16T10:25:00.583725+0000","last_clean":"2024-09-16T10:25:00.583725+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:00.583725+0000","last_undegraded":"2024-09-16T10:25:00.583725+0000","last_fullsized":"2024-09-16T10:25:00.583725+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6483,"log_dups_size":0,"ondisk_log_size":6483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554368,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6483,"num_write_kb":25932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6446","reported_seq":6600,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:59.568906+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:59.568906+0000","last_peered":"2024-09-16T10:24:59.568906+0000","last_clean":"2024-09-16T10:24:59.568906+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:59.568906+0000","last_undegraded":"2024-09-16T10:24:59.568906+0000","last_fullsized":"2024-09-16T10:24:59.568906+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26374144,"num_objects":6439,"num_object_clones":0,"num_object_copies":12878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":12352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6364","reported_seq":6525,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.666191+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:59.666191+0000","last_peered":"2024-09-16T10:24:59.666191+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:59.666191+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26046464,"num_objects":6359,"num_object_clones":0,"num_object_copies":12718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":11956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2142},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6659","reported_seq":6829,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.690215+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:59.690215+0000","last_peered":"2024-09-16T10:24:59.690215+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:59.690215+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6659,"log_dups_size":0,"ondisk_log_size":6659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27250688,"num_objects":6653,"num_object_clones":0,"num_object_copies":13306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2901,"num_write_kb":11604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2250},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6399","reported_seq":6571,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:59.617062+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:59.617062+0000","last_peered":"2024-09-16T10:24:59.617062+0000","last_clean":"2024-09-16T10:24:59.617062+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:59.617062+0000","last_undegraded":"2024-09-16T10:24:59.617062+0000","last_fullsized":"2024-09-16T10:24:59.617062+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26202112,"num_objects":6397,"num_object_clones":0,"num_object_copies":12794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":8808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6371","reported_seq":10142,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:00.596333+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:00.596333+0000","last_peered":"2024-09-16T10:25:00.596333+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:00.596333+0000","last_undegraded":"2024-09-16T10:25:00.596333+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26067910,"num_objects":6366,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2358,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6369,"num_write_kb":25460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1841,"num_bytes_recovered":7532544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2358},{"shards":"2","objects":4008}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6647","reported_seq":6723,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.652877+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:59.652877+0000","last_peered":"2024-09-16T10:24:59.652877+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:59.652877+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6647,"log_dups_size":0,"ondisk_log_size":6647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27194310,"num_objects":6641,"num_object_clones":0,"num_object_copies":13282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6645,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2207},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6402","reported_seq":6505,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:00.574470+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:00.574470+0000","last_peered":"2024-09-16T10:25:00.574470+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:00.574470+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26203078,"num_objects":6399,"num_object_clones":0,"num_object_copies":12798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6400,"num_write_kb":25591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6457","reported_seq":6541,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:00.523848+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:00.523848+0000","last_peered":"2024-09-16T10:25:00.523848+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:00.523848+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6457,"log_dups_size":0,"ondisk_log_size":6457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26424262,"num_objects":6453,"num_object_clones":0,"num_object_copies":12906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6455,"num_write_kb":25807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6295","reported_seq":6370,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.729363+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:59.729363+0000","last_peered":"2024-09-16T10:24:59.729363+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:59.729363+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25776611,"num_objects":6294,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":25173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2135},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6589","reported_seq":6719,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.684344+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:59.684344+0000","last_peered":"2024-09-16T10:24:59.684344+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:59.684344+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6589,"log_dups_size":0,"ondisk_log_size":6589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26957225,"num_objects":6584,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6586,"num_write_kb":26328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2247},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6322","reported_seq":6400,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.678650+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:59.678650+0000","last_peered":"2024-09-16T10:24:59.678650+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:59.678650+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25879494,"num_objects":6320,"num_object_clones":0,"num_object_copies":12640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":25274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2153},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6634","reported_seq":6724,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.698084+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:59.698084+0000","last_peered":"2024-09-16T10:24:59.698084+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:59.698084+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6634,"log_dups_size":0,"ondisk_log_size":6634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27125644,"num_objects":6626,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6631,"num_write_kb":26495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2231},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":423657974,"num_objects":103448,"num_object_clones":0,"num_object_copies":206896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2358,"num_objects_unfound":0,"num_objects_dirty":103448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81077,"num_write_kb":324196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1858,"num_bytes_recovered":7537123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":860008448,"data_stored":859918123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103512,"ondisk_log_size":103512,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018910,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519520,"kb_used_data":343512,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93852320,"statfs":{"total":96636764160,"available":96104775680,"internally_reserved":0,"allocated":351756288,"data_stored":351513049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":180219568},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":1.423},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.51600000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.4430000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228744,"kb_used_data":116736,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94143096,"statfs":{"total":96636764160,"available":96402530304,"internally_reserved":0,"allocated":119537664,"data_stored":119214939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":114683562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.61399999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":5.9219999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":5.9820000000000002},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":2.1059999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215138,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571564,"kb_used_data":377572,"kb_used_omap":4,"kb_used_meta":193979,"kb_avail":93800276,"statfs":{"total":96636764160,"available":96051482624,"internally_reserved":0,"allocated":386633728,"data_stored":386394103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":198635176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,2,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":6.7380000000000004},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.5760000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":9.5060000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":387735552,"data_stored":387731939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120471552,"data_stored":120388453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351801344,"data_stored":351797731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:05.070 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:05.070 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'6435", 'reported_seq': 6499, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:59.727334+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:24:59.727334+0000', 'last_peered': '2024-09-16T10:24:59.727334+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:24:59.727334+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6435, 'log_dups_size': 0, 'ondisk_log_size': 6435, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26357760, 'num_objects': 6435, 'num_object_clones': 0, 'num_object_copies': 12870, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6435, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6435, 'num_write_kb': 25740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2243}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:05.070 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:05.070 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'6522", 'reported_seq': 6704, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:00.607137+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:00.607137+0000', 'last_peered': '2024-09-16T10:25:00.607137+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:00.607137+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 6522, 'log_dups_size': 0, 'ondisk_log_size': 6522, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26697728, 'num_objects': 6518, 'num_object_clones': 0, 'num_object_copies': 13036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2579, 'num_write_kb': 10316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 2183}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'6364", 'reported_seq': 6525, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:59.666191+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:24:59.666191+0000', 'last_peered': '2024-09-16T10:24:59.666191+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:24:59.666191+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 6364, 'log_dups_size': 0, 'ondisk_log_size': 6364, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26046464, 'num_objects': 6359, 'num_object_clones': 0, 'num_object_copies': 12718, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2989, 'num_write_kb': 11956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2142}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'6659", 'reported_seq': 6829, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:59.690215+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:24:59.690215+0000', 'last_peered': '2024-09-16T10:24:59.690215+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:24:59.690215+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 6659, 'log_dups_size': 0, 'ondisk_log_size': 6659, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27250688, 'num_objects': 6653, 'num_object_clones': 0, 'num_object_copies': 13306, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2901, 'num_write_kb': 11604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2250}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'6371", 'reported_seq': 10142, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:00.596333+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:00.596333+0000', 'last_peered': '2024-09-16T10:25:00.596333+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:00.596333+0000', 'last_undegraded': '2024-09-16T10:25:00.596333+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6371, 'log_dups_size': 0, 'ondisk_log_size': 6371, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26067910, 'num_objects': 6366, 'num_object_clones': 0, 'num_object_copies': 12732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2358, 'num_objects_unfound': 0, 'num_objects_dirty': 6366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6369, 'num_write_kb': 25460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1841, 'num_bytes_recovered': 7532544, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2358}, {'shards': '2', 'objects': 4008}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'6647", 'reported_seq': 6723, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:59.652877+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:24:59.652877+0000', 'last_peered': '2024-09-16T10:24:59.652877+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:24:59.652877+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6647, 'log_dups_size': 0, 'ondisk_log_size': 6647, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27194310, 'num_objects': 6641, 'num_object_clones': 0, 'num_object_copies': 13282, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6641, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6645, 'num_write_kb': 26560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2207}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'6402", 'reported_seq': 6505, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:00.574470+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:00.574470+0000', 'last_peered': '2024-09-16T10:25:00.574470+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:00.574470+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 6402, 'log_dups_size': 0, 'ondisk_log_size': 6402, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26203078, 'num_objects': 6399, 'num_object_clones': 0, 'num_object_copies': 12798, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6400, 'num_write_kb': 25591, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 2122}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:05.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:05.072 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'6457", 'reported_seq': 6541, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:00.523848+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:00.523848+0000', 'last_peered': '2024-09-16T10:25:00.523848+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:00.523848+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6457, 'log_dups_size': 0, 'ondisk_log_size': 6457, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26424262, 'num_objects': 6453, 'num_object_clones': 0, 'num_object_copies': 12906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6455, 'num_write_kb': 25807, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 2261}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:05.072 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:05.072 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'6295", 'reported_seq': 6370, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:59.729363+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:24:59.729363+0000', 'last_peered': '2024-09-16T10:24:59.729363+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:24:59.729363+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6295, 'log_dups_size': 0, 'ondisk_log_size': 6295, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25776611, 'num_objects': 6294, 'num_object_clones': 0, 'num_object_copies': 12588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6294, 'num_write_kb': 25173, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2135}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:05.072 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:05.072 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'6589", 'reported_seq': 6719, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:59.684344+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:24:59.684344+0000', 'last_peered': '2024-09-16T10:24:59.684344+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:24:59.684344+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 6589, 'log_dups_size': 0, 'ondisk_log_size': 6589, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26957225, 'num_objects': 6584, 'num_object_clones': 0, 'num_object_copies': 13168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6584, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6586, 'num_write_kb': 26328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2247}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:05.072 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:05.072 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'6322", 'reported_seq': 6400, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:59.678650+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:24:59.678650+0000', 'last_peered': '2024-09-16T10:24:59.678650+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:24:59.678650+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6322, 'log_dups_size': 0, 'ondisk_log_size': 6322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25879494, 'num_objects': 6320, 'num_object_clones': 0, 'num_object_copies': 12640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6320, 'num_write_kb': 25274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2153}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:05.072 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:05.072 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'6634", 'reported_seq': 6724, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:24:59.698084+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:24:59.698084+0000', 'last_peered': '2024-09-16T10:24:59.698084+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:24:59.698084+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6634, 'log_dups_size': 0, 'ondisk_log_size': 6634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27125644, 'num_objects': 6626, 'num_object_clones': 0, 'num_object_copies': 13252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6631, 'num_write_kb': 26495, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2231}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:05.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:05.538 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 113 20 106275 106255 3.67231 1.85547 0.0421832 0.0212649 2024-09-16T10:25:05.727 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:05.727 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/206900 objects degraded (22.778%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":194,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":103450,"data_bytes":423657993,"bytes_used":1351503872,"bytes_avail":288558788608,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":206900,"degraded_ratio":0.22778153697438375,"misplaced_objects":2358,"misplaced_total":206900,"misplaced_ratio":0.011396810053165779,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":109540,"recovering_keys_per_sec":0,"num_objects_recovered":321,"num_bytes_recovered":1314816,"num_keys_recovered":0,"write_bytes_sec":1800765,"read_op_per_sec":0,"write_op_per_sec":439},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:05.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:05.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:06.309 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:06.339 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-16T10:25:04.591277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":423657993,"num_objects":103450,"num_object_clones":0,"num_object_copies":206900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2358,"num_objects_unfound":0,"num_objects_dirty":103450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81079,"num_write_kb":324198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1859,"num_bytes_recovered":7537142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103515,"ondisk_log_size":103515,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1319828,"kb_used_data":837820,"kb_used_omap":12,"kb_used_meta":481971,"kb_avail":281795692,"statfs":{"total":289910292480,"available":288558788608,"internally_reserved":0,"allocated":857927680,"data_stored":857122091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":493538306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,5,6,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21614592,"num_objects":5277,"num_object_clones":0,"num_object_copies":10554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-321,"num_objects_unfound":0,"num_objects_dirty":5277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5277,"num_write_kb":21108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":1314816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002999"},"pg_stats":[{"pgid":"2.f","version":"42'6435","reported_seq":6499,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.727334+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:24:59.727334+0000","last_peered":"2024-09-16T10:24:59.727334+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:24:59.727334+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6435,"log_dups_size":0,"ondisk_log_size":6435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26357760,"num_objects":6435,"num_object_clones":0,"num_object_copies":12870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6435,"num_write_kb":25740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2243},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6487","reported_seq":6641,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:00.544959+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:00.544959+0000","last_peered":"2024-09-16T10:25:00.544959+0000","last_clean":"2024-09-16T10:25:00.544959+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:00.544959+0000","last_undegraded":"2024-09-16T10:25:00.544959+0000","last_fullsized":"2024-09-16T10:25:00.544959+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6487,"log_dups_size":0,"ondisk_log_size":6487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26546176,"num_objects":6481,"num_object_clones":0,"num_object_copies":12962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2700,"num_write_kb":10800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6522","reported_seq":6704,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:00.607137+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:00.607137+0000","last_peered":"2024-09-16T10:25:00.607137+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:00.607137+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26697728,"num_objects":6518,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2579,"num_write_kb":10316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2183}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6483","reported_seq":6536,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:00.583725+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:00.583725+0000","last_peered":"2024-09-16T10:25:00.583725+0000","last_clean":"2024-09-16T10:25:00.583725+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:00.583725+0000","last_undegraded":"2024-09-16T10:25:00.583725+0000","last_fullsized":"2024-09-16T10:25:00.583725+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6483,"log_dups_size":0,"ondisk_log_size":6483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554368,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6483,"num_write_kb":25932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6446","reported_seq":6600,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:59.568906+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:24:59.568906+0000","last_peered":"2024-09-16T10:24:59.568906+0000","last_clean":"2024-09-16T10:24:59.568906+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:24:59.568906+0000","last_undegraded":"2024-09-16T10:24:59.568906+0000","last_fullsized":"2024-09-16T10:24:59.568906+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26374144,"num_objects":6439,"num_object_clones":0,"num_object_copies":12878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":12352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6364","reported_seq":6525,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.666191+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:24:59.666191+0000","last_peered":"2024-09-16T10:24:59.666191+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:24:59.666191+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6364,"log_dups_size":0,"ondisk_log_size":6364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26046464,"num_objects":6359,"num_object_clones":0,"num_object_copies":12718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2989,"num_write_kb":11956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2142},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6659","reported_seq":6829,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.690215+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:24:59.690215+0000","last_peered":"2024-09-16T10:24:59.690215+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:24:59.690215+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6659,"log_dups_size":0,"ondisk_log_size":6659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27250688,"num_objects":6653,"num_object_clones":0,"num_object_copies":13306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2901,"num_write_kb":11604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2250},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6399","reported_seq":6571,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:24:59.617062+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:24:59.617062+0000","last_peered":"2024-09-16T10:24:59.617062+0000","last_clean":"2024-09-16T10:24:59.617062+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:24:59.617062+0000","last_undegraded":"2024-09-16T10:24:59.617062+0000","last_fullsized":"2024-09-16T10:24:59.617062+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6399,"log_dups_size":0,"ondisk_log_size":6399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26202112,"num_objects":6397,"num_object_clones":0,"num_object_copies":12794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2202,"num_write_kb":8808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6371","reported_seq":10142,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:00.596333+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:00.596333+0000","last_peered":"2024-09-16T10:25:00.596333+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:00.596333+0000","last_undegraded":"2024-09-16T10:25:00.596333+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6371,"log_dups_size":0,"ondisk_log_size":6371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26067910,"num_objects":6366,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2358,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6369,"num_write_kb":25460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1841,"num_bytes_recovered":7532544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2358},{"shards":"2","objects":4008}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6647","reported_seq":6723,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.652877+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:24:59.652877+0000","last_peered":"2024-09-16T10:24:59.652877+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:24:59.652877+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6647,"log_dups_size":0,"ondisk_log_size":6647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27194310,"num_objects":6641,"num_object_clones":0,"num_object_copies":13282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6645,"num_write_kb":26560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2207},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6402","reported_seq":6505,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:00.574470+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:00.574470+0000","last_peered":"2024-09-16T10:25:00.574470+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:00.574470+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26203078,"num_objects":6399,"num_object_clones":0,"num_object_copies":12798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6400,"num_write_kb":25591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2122}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6457","reported_seq":6541,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:00.523848+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:00.523848+0000","last_peered":"2024-09-16T10:25:00.523848+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:00.523848+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6457,"log_dups_size":0,"ondisk_log_size":6457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26424262,"num_objects":6453,"num_object_clones":0,"num_object_copies":12906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6455,"num_write_kb":25807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2261}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6295","reported_seq":6370,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.729363+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:24:59.729363+0000","last_peered":"2024-09-16T10:24:59.729363+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:24:59.729363+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25776611,"num_objects":6294,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":25173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2135},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6589","reported_seq":6719,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.684344+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:24:59.684344+0000","last_peered":"2024-09-16T10:24:59.684344+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:24:59.684344+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6589,"log_dups_size":0,"ondisk_log_size":6589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26957225,"num_objects":6584,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6586,"num_write_kb":26328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2247},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6322","reported_seq":6400,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.678650+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:24:59.678650+0000","last_peered":"2024-09-16T10:24:59.678650+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:24:59.678650+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6322,"log_dups_size":0,"ondisk_log_size":6322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25879494,"num_objects":6320,"num_object_clones":0,"num_object_copies":12640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6320,"num_write_kb":25274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2153},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6634","reported_seq":6724,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:24:59.698084+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:24:59.698084+0000","last_peered":"2024-09-16T10:24:59.698084+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:24:59.698084+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6634,"log_dups_size":0,"ondisk_log_size":6634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27125644,"num_objects":6626,"num_object_clones":0,"num_object_copies":13252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6631,"num_write_kb":26495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2231},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":423657974,"num_objects":103448,"num_object_clones":0,"num_object_copies":206896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2358,"num_objects_unfound":0,"num_objects_dirty":103448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81077,"num_write_kb":324196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1858,"num_bytes_recovered":7537123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":860008448,"data_stored":859918123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103512,"ondisk_log_size":103512,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018910,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":519520,"kb_used_data":343512,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93852320,"statfs":{"total":96636764160,"available":96104775680,"internally_reserved":0,"allocated":351756288,"data_stored":351513049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":180219568},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":1.423},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.51600000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.4430000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691855,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228744,"kb_used_data":116736,"kb_used_omap":4,"kb_used_meta":111995,"kb_avail":94143096,"statfs":{"total":96636764160,"available":96402530304,"internally_reserved":0,"allocated":119537664,"data_stored":119214939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":114683562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.61399999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":5.9219999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":5.9820000000000002},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":2.1059999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215138,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571564,"kb_used_data":377572,"kb_used_omap":4,"kb_used_meta":193979,"kb_avail":93800276,"statfs":{"total":96636764160,"available":96051482624,"internally_reserved":0,"allocated":386633728,"data_stored":386394103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":198635176},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,2,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":6.7380000000000004},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.5760000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":9.5060000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.313}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":387735552,"data_stored":387731939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120471552,"data_stored":120388453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":351801344,"data_stored":351797731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:06.538 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 114 20 106765 106745 3.65689 1.91406 0.089212 0.0213557 2024-09-16T10:25:07.541 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 115 20 107318 107298 3.64379 2.16016 0.0904183 0.0214336 2024-09-16T10:25:08.541 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 116 20 107863 107843 3.63073 2.12891 0.0187592 0.0215104 2024-09-16T10:25:09.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:09.541 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 117 20 108380 108360 3.61696 2.01953 0.0423418 0.0215898 2024-09-16T10:25:09.916 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:09.919 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:09.943 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-09-16T10:25:08.592262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":434319881,"num_objects":106053,"num_object_clones":0,"num_object_copies":212106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2189,"num_objects_unfound":0,"num_objects_dirty":106053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83682,"num_write_kb":334610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2028,"num_bytes_recovered":8229366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106118,"ondisk_log_size":106118,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1378792,"kb_used_data":860688,"kb_used_omap":12,"kb_used_meta":518067,"kb_avail":281736728,"statfs":{"total":289910292480,"available":288498409472,"internally_reserved":0,"allocated":881344512,"data_stored":880538923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":530500610},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,17,4,7,6,11,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21417984,"num_objects":5229,"num_object_clones":0,"num_object_copies":10458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-331,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":20916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":1355776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003069"},"pg_stats":[{"pgid":"2.f","version":"42'6604","reported_seq":6668,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.751212+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:04.751212+0000","last_peered":"2024-09-16T10:25:04.751212+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:04.751212+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27049984,"num_objects":6604,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":26416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2412},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6661","reported_seq":6815,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:05.561076+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:05.561076+0000","last_peered":"2024-09-16T10:25:05.561076+0000","last_clean":"2024-09-16T10:25:05.561076+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:05.561076+0000","last_undegraded":"2024-09-16T10:25:05.561076+0000","last_fullsized":"2024-09-16T10:25:05.561076+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6661,"log_dups_size":0,"ondisk_log_size":6661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27258880,"num_objects":6655,"num_object_clones":0,"num_object_copies":13310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":11496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6663","reported_seq":6845,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:05.523043+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:05.523043+0000","last_peered":"2024-09-16T10:25:05.523043+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:05.523043+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27275264,"num_objects":6659,"num_object_clones":0,"num_object_copies":13318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":10880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2324}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6635","reported_seq":6688,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:05.588939+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:05.588939+0000","last_peered":"2024-09-16T10:25:05.588939+0000","last_clean":"2024-09-16T10:25:05.588939+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:05.588939+0000","last_undegraded":"2024-09-16T10:25:05.588939+0000","last_fullsized":"2024-09-16T10:25:05.588939+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6635,"log_dups_size":0,"ondisk_log_size":6635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27176960,"num_objects":6635,"num_object_clones":0,"num_object_copies":13270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6635,"num_write_kb":26540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6601","reported_seq":6755,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:04.613186+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:04.613186+0000","last_peered":"2024-09-16T10:25:04.613186+0000","last_clean":"2024-09-16T10:25:04.613186+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:04.613186+0000","last_undegraded":"2024-09-16T10:25:04.613186+0000","last_fullsized":"2024-09-16T10:25:04.613186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6601,"log_dups_size":0,"ondisk_log_size":6601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27009024,"num_objects":6594,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3243,"num_write_kb":12972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6522","reported_seq":6683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.659627+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:04.659627+0000","last_peered":"2024-09-16T10:25:04.659627+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:04.659627+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26693632,"num_objects":6517,"num_object_clones":0,"num_object_copies":13034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3147,"num_write_kb":12588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2300},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6817","reported_seq":6988,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.618910+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:04.618910+0000","last_peered":"2024-09-16T10:25:04.618910+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:04.618910+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6817,"log_dups_size":0,"ondisk_log_size":6817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27897856,"num_objects":6811,"num_object_clones":0,"num_object_copies":13622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3059,"num_write_kb":12236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2408},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6563","reported_seq":6735,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:04.707576+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:04.707576+0000","last_peered":"2024-09-16T10:25:04.707576+0000","last_clean":"2024-09-16T10:25:04.707576+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:04.707576+0000","last_undegraded":"2024-09-16T10:25:04.707576+0000","last_fullsized":"2024-09-16T10:25:04.707576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26873856,"num_objects":6561,"num_object_clones":0,"num_object_copies":13122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2366,"num_write_kb":9464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6530","reported_seq":10638,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:05.619943+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:05.619943+0000","last_peered":"2024-09-16T10:25:05.619943+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:05.619943+0000","last_undegraded":"2024-09-16T10:25:05.619943+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719174,"num_objects":6525,"num_object_clones":0,"num_object_copies":13050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2189,"num_objects_unfound":0,"num_objects_dirty":6525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":26096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2189},{"shards":"2","objects":4336}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6809","reported_seq":6885,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.586993+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:04.586993+0000","last_peered":"2024-09-16T10:25:04.586993+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:04.586993+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6809,"log_dups_size":0,"ondisk_log_size":6809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27857862,"num_objects":6803,"num_object_clones":0,"num_object_copies":13606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6807,"num_write_kb":27208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2369},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6562","reported_seq":6665,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:05.579528+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:05.579528+0000","last_peered":"2024-09-16T10:25:05.579528+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:05.579528+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26858438,"num_objects":6559,"num_object_clones":0,"num_object_copies":13118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":26231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2282}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6631","reported_seq":6715,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:05.585566+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:05.585566+0000","last_peered":"2024-09-16T10:25:05.585566+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:05.585566+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6631,"log_dups_size":0,"ondisk_log_size":6631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27136966,"num_objects":6627,"num_object_clones":0,"num_object_copies":13254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":26503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2435}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6455","reported_seq":6532,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.604068+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:04.604068+0000","last_peered":"2024-09-16T10:25:04.604068+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:04.604068+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26431971,"num_objects":6454,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6454,"num_write_kb":25813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2295},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6764","reported_seq":6894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.690598+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:04.690598+0000","last_peered":"2024-09-16T10:25:04.690598+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:04.690598+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6764,"log_dups_size":0,"ondisk_log_size":6764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27674025,"num_objects":6759,"num_object_clones":0,"num_object_copies":13518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6761,"num_write_kb":27028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2422},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6493","reported_seq":6571,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.747213+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:04.747213+0000","last_peered":"2024-09-16T10:25:04.747213+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:04.747213+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6493,"log_dups_size":0,"ondisk_log_size":6493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26579910,"num_objects":6491,"num_object_clones":0,"num_object_copies":12982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":25958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2324},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6805","reported_seq":6894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.720572+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:04.720572+0000","last_peered":"2024-09-16T10:25:04.720572+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:04.720572+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6805,"log_dups_size":0,"ondisk_log_size":6805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27826060,"num_objects":6797,"num_object_clones":0,"num_object_copies":13594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6802,"num_write_kb":27179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2402},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":434319862,"num_objects":106051,"num_object_clones":0,"num_object_copies":212102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2189,"num_objects_unfound":0,"num_objects_dirty":106051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83680,"num_write_kb":334608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2027,"num_bytes_recovered":8229347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":881971200,"data_stored":881888101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106115,"ondisk_log_size":106115,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018911,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528724,"kb_used_data":348620,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93843116,"statfs":{"total":96636764160,"available":96095350784,"internally_reserved":0,"allocated":356986880,"data_stored":356743641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":184413872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,2,1,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.498}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.248}]}]},{"osd":1,"up_from":40,"seq":171798691856,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228996,"kb_used_data":125180,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94142844,"statfs":{"total":96636764160,"available":96402272256,"internally_reserved":0,"allocated":128184320,"data_stored":127861595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":106294954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":3.7639999999999998},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":3.7930000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":3.3340000000000001},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":5.5149999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621072,"kb_used_data":386888,"kb_used_omap":4,"kb_used_meta":234171,"kb_avail":93750768,"statfs":{"total":96636764160,"available":96000786432,"internally_reserved":0,"allocated":396173312,"data_stored":395933687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":239791784},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,3,1,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":1.4950000000000001},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.1469999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.4560000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395829248,"data_stored":395825635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129069056,"data_stored":128993183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357072896,"data_stored":357069283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:09.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:10.506 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:10.507 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:10.532 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2024-09-16T10:25:08.592262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":434319881,"num_objects":106053,"num_object_clones":0,"num_object_copies":212106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2189,"num_objects_unfound":0,"num_objects_dirty":106053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83682,"num_write_kb":334610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2028,"num_bytes_recovered":8229366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106118,"ondisk_log_size":106118,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1378792,"kb_used_data":860688,"kb_used_omap":12,"kb_used_meta":518067,"kb_avail":281736728,"statfs":{"total":289910292480,"available":288498409472,"internally_reserved":0,"allocated":881344512,"data_stored":880538923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":530500610},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,17,4,7,6,11,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21417984,"num_objects":5229,"num_object_clones":0,"num_object_copies":10458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-331,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":20916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":1355776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003069"},"pg_stats":[{"pgid":"2.f","version":"42'6604","reported_seq":6668,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.751212+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:04.751212+0000","last_peered":"2024-09-16T10:25:04.751212+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:04.751212+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27049984,"num_objects":6604,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":26416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2412},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6661","reported_seq":6815,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:05.561076+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:05.561076+0000","last_peered":"2024-09-16T10:25:05.561076+0000","last_clean":"2024-09-16T10:25:05.561076+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:05.561076+0000","last_undegraded":"2024-09-16T10:25:05.561076+0000","last_fullsized":"2024-09-16T10:25:05.561076+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6661,"log_dups_size":0,"ondisk_log_size":6661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27258880,"num_objects":6655,"num_object_clones":0,"num_object_copies":13310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":11496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6663","reported_seq":6845,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:05.523043+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:05.523043+0000","last_peered":"2024-09-16T10:25:05.523043+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:05.523043+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27275264,"num_objects":6659,"num_object_clones":0,"num_object_copies":13318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":10880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2324}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6635","reported_seq":6688,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:05.588939+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:05.588939+0000","last_peered":"2024-09-16T10:25:05.588939+0000","last_clean":"2024-09-16T10:25:05.588939+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:05.588939+0000","last_undegraded":"2024-09-16T10:25:05.588939+0000","last_fullsized":"2024-09-16T10:25:05.588939+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6635,"log_dups_size":0,"ondisk_log_size":6635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27176960,"num_objects":6635,"num_object_clones":0,"num_object_copies":13270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6635,"num_write_kb":26540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6601","reported_seq":6755,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:04.613186+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:04.613186+0000","last_peered":"2024-09-16T10:25:04.613186+0000","last_clean":"2024-09-16T10:25:04.613186+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:04.613186+0000","last_undegraded":"2024-09-16T10:25:04.613186+0000","last_fullsized":"2024-09-16T10:25:04.613186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6601,"log_dups_size":0,"ondisk_log_size":6601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27009024,"num_objects":6594,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3243,"num_write_kb":12972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6522","reported_seq":6683,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.659627+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:04.659627+0000","last_peered":"2024-09-16T10:25:04.659627+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:04.659627+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26693632,"num_objects":6517,"num_object_clones":0,"num_object_copies":13034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3147,"num_write_kb":12588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2300},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6817","reported_seq":6988,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.618910+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:04.618910+0000","last_peered":"2024-09-16T10:25:04.618910+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:04.618910+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6817,"log_dups_size":0,"ondisk_log_size":6817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27897856,"num_objects":6811,"num_object_clones":0,"num_object_copies":13622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3059,"num_write_kb":12236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2408},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6563","reported_seq":6735,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:04.707576+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:04.707576+0000","last_peered":"2024-09-16T10:25:04.707576+0000","last_clean":"2024-09-16T10:25:04.707576+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:04.707576+0000","last_undegraded":"2024-09-16T10:25:04.707576+0000","last_fullsized":"2024-09-16T10:25:04.707576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26873856,"num_objects":6561,"num_object_clones":0,"num_object_copies":13122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2366,"num_write_kb":9464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6530","reported_seq":10638,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:05.619943+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:05.619943+0000","last_peered":"2024-09-16T10:25:05.619943+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:05.619943+0000","last_undegraded":"2024-09-16T10:25:05.619943+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719174,"num_objects":6525,"num_object_clones":0,"num_object_copies":13050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2189,"num_objects_unfound":0,"num_objects_dirty":6525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":26096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2189},{"shards":"2","objects":4336}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6809","reported_seq":6885,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.586993+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:04.586993+0000","last_peered":"2024-09-16T10:25:04.586993+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:04.586993+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6809,"log_dups_size":0,"ondisk_log_size":6809,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27857862,"num_objects":6803,"num_object_clones":0,"num_object_copies":13606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6807,"num_write_kb":27208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2369},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6562","reported_seq":6665,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:05.579528+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:05.579528+0000","last_peered":"2024-09-16T10:25:05.579528+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:05.579528+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26858438,"num_objects":6559,"num_object_clones":0,"num_object_copies":13118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":26231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2282}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6631","reported_seq":6715,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:05.585566+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:05.585566+0000","last_peered":"2024-09-16T10:25:05.585566+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:05.585566+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6631,"log_dups_size":0,"ondisk_log_size":6631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27136966,"num_objects":6627,"num_object_clones":0,"num_object_copies":13254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":26503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2435}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6455","reported_seq":6532,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.604068+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:04.604068+0000","last_peered":"2024-09-16T10:25:04.604068+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:04.604068+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26431971,"num_objects":6454,"num_object_clones":0,"num_object_copies":12908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6454,"num_write_kb":25813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2295},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6764","reported_seq":6894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.690598+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:04.690598+0000","last_peered":"2024-09-16T10:25:04.690598+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:04.690598+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6764,"log_dups_size":0,"ondisk_log_size":6764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27674025,"num_objects":6759,"num_object_clones":0,"num_object_copies":13518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6761,"num_write_kb":27028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2422},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6493","reported_seq":6571,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.747213+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:04.747213+0000","last_peered":"2024-09-16T10:25:04.747213+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:04.747213+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6493,"log_dups_size":0,"ondisk_log_size":6493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26579910,"num_objects":6491,"num_object_clones":0,"num_object_copies":12982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":25958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2324},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6805","reported_seq":6894,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:04.720572+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:04.720572+0000","last_peered":"2024-09-16T10:25:04.720572+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:04.720572+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6805,"log_dups_size":0,"ondisk_log_size":6805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27826060,"num_objects":6797,"num_object_clones":0,"num_object_copies":13594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6802,"num_write_kb":27179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2402},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":434319862,"num_objects":106051,"num_object_clones":0,"num_object_copies":212102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2189,"num_objects_unfound":0,"num_objects_dirty":106051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83680,"num_write_kb":334608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2027,"num_bytes_recovered":8229347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":881971200,"data_stored":881888101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106115,"ondisk_log_size":106115,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018911,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528724,"kb_used_data":348620,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93843116,"statfs":{"total":96636764160,"available":96095350784,"internally_reserved":0,"allocated":356986880,"data_stored":356743641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":184413872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,2,1,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.498}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.248}]}]},{"osd":1,"up_from":40,"seq":171798691856,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228996,"kb_used_data":125180,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94142844,"statfs":{"total":96636764160,"available":96402272256,"internally_reserved":0,"allocated":128184320,"data_stored":127861595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":106294954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":3.7639999999999998},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":3.7930000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":3.3340000000000001},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":5.5149999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215139,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":621072,"kb_used_data":386888,"kb_used_omap":4,"kb_used_meta":234171,"kb_avail":93750768,"statfs":{"total":96636764160,"available":96000786432,"internally_reserved":0,"allocated":396173312,"data_stored":395933687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":239791784},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,3,1,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":1.4950000000000001},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.1469999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.4560000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395829248,"data_stored":395825635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129069056,"data_stored":128993183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357072896,"data_stored":357069283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:10.533 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:10.533 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'6604", 'reported_seq': 6668, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:04.751212+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:25:04.751212+0000', 'last_peered': '2024-09-16T10:25:04.751212+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:25:04.751212+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6604, 'log_dups_size': 0, 'ondisk_log_size': 6604, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27049984, 'num_objects': 6604, 'num_object_clones': 0, 'num_object_copies': 13208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6604, 'num_write_kb': 26416, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2412}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:10.533 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:10.533 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'6663", 'reported_seq': 6845, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:05.523043+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:05.523043+0000', 'last_peered': '2024-09-16T10:25:05.523043+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:05.523043+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 6663, 'log_dups_size': 0, 'ondisk_log_size': 6663, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27275264, 'num_objects': 6659, 'num_object_clones': 0, 'num_object_copies': 13318, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2720, 'num_write_kb': 10880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 2324}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:10.534 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:10.534 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'6522", 'reported_seq': 6683, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:04.659627+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:25:04.659627+0000', 'last_peered': '2024-09-16T10:25:04.659627+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:25:04.659627+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 6522, 'log_dups_size': 0, 'ondisk_log_size': 6522, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26693632, 'num_objects': 6517, 'num_object_clones': 0, 'num_object_copies': 13034, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3147, 'num_write_kb': 12588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2300}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:10.534 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:10.534 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'6817", 'reported_seq': 6988, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:04.618910+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:25:04.618910+0000', 'last_peered': '2024-09-16T10:25:04.618910+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:25:04.618910+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 6817, 'log_dups_size': 0, 'ondisk_log_size': 6817, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27897856, 'num_objects': 6811, 'num_object_clones': 0, 'num_object_copies': 13622, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6811, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3059, 'num_write_kb': 12236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2408}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:10.534 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:10.534 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'6530", 'reported_seq': 10638, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:05.619943+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:05.619943+0000', 'last_peered': '2024-09-16T10:25:05.619943+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:05.619943+0000', 'last_undegraded': '2024-09-16T10:25:05.619943+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6530, 'log_dups_size': 0, 'ondisk_log_size': 6530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26719174, 'num_objects': 6525, 'num_object_clones': 0, 'num_object_copies': 13050, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2189, 'num_objects_unfound': 0, 'num_objects_dirty': 6525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6528, 'num_write_kb': 26096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2010, 'num_bytes_recovered': 8224768, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2189}, {'shards': '2', 'objects': 4336}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:10.534 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:10.534 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'6809", 'reported_seq': 6885, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:04.586993+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:25:04.586993+0000', 'last_peered': '2024-09-16T10:25:04.586993+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:25:04.586993+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6809, 'log_dups_size': 0, 'ondisk_log_size': 6809, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27857862, 'num_objects': 6803, 'num_object_clones': 0, 'num_object_copies': 13606, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6807, 'num_write_kb': 27208, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2369}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:10.534 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:10.534 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'6562", 'reported_seq': 6665, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:05.579528+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:05.579528+0000', 'last_peered': '2024-09-16T10:25:05.579528+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:05.579528+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 6562, 'log_dups_size': 0, 'ondisk_log_size': 6562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26858438, 'num_objects': 6559, 'num_object_clones': 0, 'num_object_copies': 13118, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6560, 'num_write_kb': 26231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 2282}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:10.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:10.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'6631", 'reported_seq': 6715, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:05.585566+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:05.585566+0000', 'last_peered': '2024-09-16T10:25:05.585566+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:05.585566+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6631, 'log_dups_size': 0, 'ondisk_log_size': 6631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27136966, 'num_objects': 6627, 'num_object_clones': 0, 'num_object_copies': 13254, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6629, 'num_write_kb': 26503, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 2435}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:10.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:10.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'6455", 'reported_seq': 6532, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:04.604068+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:25:04.604068+0000', 'last_peered': '2024-09-16T10:25:04.604068+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:25:04.604068+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6455, 'log_dups_size': 0, 'ondisk_log_size': 6455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26431971, 'num_objects': 6454, 'num_object_clones': 0, 'num_object_copies': 12908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6454, 'num_write_kb': 25813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2295}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:10.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:10.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'6764", 'reported_seq': 6894, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:04.690598+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:04.690598+0000', 'last_peered': '2024-09-16T10:25:04.690598+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:04.690598+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 6764, 'log_dups_size': 0, 'ondisk_log_size': 6764, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27674025, 'num_objects': 6759, 'num_object_clones': 0, 'num_object_copies': 13518, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6761, 'num_write_kb': 27028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2422}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:10.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:10.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'6493", 'reported_seq': 6571, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:04.747213+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:04.747213+0000', 'last_peered': '2024-09-16T10:25:04.747213+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:04.747213+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6493, 'log_dups_size': 0, 'ondisk_log_size': 6493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26579910, 'num_objects': 6491, 'num_object_clones': 0, 'num_object_copies': 12982, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6491, 'num_write_kb': 25958, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2324}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:10.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:10.535 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'6805", 'reported_seq': 6894, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:04.720572+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:25:04.720572+0000', 'last_peered': '2024-09-16T10:25:04.720572+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:25:04.720572+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6805, 'log_dups_size': 0, 'ondisk_log_size': 6805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27826060, 'num_objects': 6797, 'num_object_clones': 0, 'num_object_copies': 13594, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6797, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6802, 'num_write_kb': 27179, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2402}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:10.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:10.541 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 118 20 108831 108811 3.60124 1.76172 0.0200953 0.0216839 2024-09-16T10:25:11.236 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:11.237 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/215394 objects degraded (21.880%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":200,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":107697,"data_bytes":441053705,"bytes_used":1441001472,"bytes_avail":288469291008,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":215394,"degraded_ratio":0.21879903804191389,"misplaced_objects":2189,"misplaced_total":215394,"misplaced_ratio":0.01016277147924269,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":112953,"recovering_keys_per_sec":0,"num_objects_recovered":331,"num_bytes_recovered":1355776,"num_keys_recovered":0,"write_bytes_sec":2345405,"read_op_per_sec":0,"write_op_per_sec":572},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:11.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:11.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:11.542 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 119 20 109298 109278 3.5863 1.82422 0.0277235 0.0217753 2024-09-16T10:25:11.870 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:11.896 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-16T10:25:10.592741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":441053705,"num_objects":107697,"num_object_clones":0,"num_object_copies":215394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2189,"num_objects_unfound":0,"num_objects_dirty":107697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85326,"num_write_kb":341186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2028,"num_bytes_recovered":8229366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107762,"ondisk_log_size":107762,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1407228,"kb_used_data":866980,"kb_used_omap":12,"kb_used_meta":540211,"kb_avail":281708292,"statfs":{"total":289910292480,"available":288469291008,"internally_reserved":0,"allocated":887787520,"data_stored":886981931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":553176066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,17,4,7,6,11,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28151808,"num_objects":6873,"num_object_clones":0,"num_object_copies":13746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-331,"num_objects_unfound":0,"num_objects_dirty":6873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6873,"num_write_kb":27492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":331,"num_bytes_recovered":1355776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002958"},"pg_stats":[{"pgid":"2.f","version":"42'6754","reported_seq":6820,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.680090+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:09.680090+0000","last_peered":"2024-09-16T10:25:09.680090+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:09.680090+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6754,"log_dups_size":0,"ondisk_log_size":6754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27664384,"num_objects":6754,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":27016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2562},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6661","reported_seq":6815,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:05.561076+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:05.561076+0000","last_peered":"2024-09-16T10:25:05.561076+0000","last_clean":"2024-09-16T10:25:05.561076+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:05.561076+0000","last_undegraded":"2024-09-16T10:25:05.561076+0000","last_fullsized":"2024-09-16T10:25:05.561076+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6661,"log_dups_size":0,"ondisk_log_size":6661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27258880,"num_objects":6655,"num_object_clones":0,"num_object_copies":13310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":11496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6663","reported_seq":6845,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:05.523043+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:05.523043+0000","last_peered":"2024-09-16T10:25:05.523043+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:05.523043+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6663,"log_dups_size":0,"ondisk_log_size":6663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27275264,"num_objects":6659,"num_object_clones":0,"num_object_copies":13318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":10880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2324}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6635","reported_seq":6688,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:05.588939+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:05.588939+0000","last_peered":"2024-09-16T10:25:05.588939+0000","last_clean":"2024-09-16T10:25:05.588939+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:05.588939+0000","last_undegraded":"2024-09-16T10:25:05.588939+0000","last_fullsized":"2024-09-16T10:25:05.588939+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6635,"log_dups_size":0,"ondisk_log_size":6635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27176960,"num_objects":6635,"num_object_clones":0,"num_object_copies":13270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6635,"num_write_kb":26540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6775","reported_seq":6929,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:09.760220+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:09.760220+0000","last_peered":"2024-09-16T10:25:09.760220+0000","last_clean":"2024-09-16T10:25:09.760220+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:09.760220+0000","last_undegraded":"2024-09-16T10:25:09.760220+0000","last_fullsized":"2024-09-16T10:25:09.760220+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6775,"log_dups_size":0,"ondisk_log_size":6775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":6768,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3417,"num_write_kb":13668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6683","reported_seq":6844,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.637922+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:09.637922+0000","last_peered":"2024-09-16T10:25:09.637922+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:09.637922+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6683,"log_dups_size":0,"ondisk_log_size":6683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27353088,"num_objects":6678,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3308,"num_write_kb":13232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2461},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6959","reported_seq":7130,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.765226+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:09.765226+0000","last_peered":"2024-09-16T10:25:09.765226+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:09.765226+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28479488,"num_objects":6953,"num_object_clones":0,"num_object_copies":13906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3201,"num_write_kb":12804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2550},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6738","reported_seq":6910,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:09.758891+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:09.758891+0000","last_peered":"2024-09-16T10:25:09.758891+0000","last_clean":"2024-09-16T10:25:09.758891+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:09.758891+0000","last_undegraded":"2024-09-16T10:25:09.758891+0000","last_fullsized":"2024-09-16T10:25:09.758891+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6738,"log_dups_size":0,"ondisk_log_size":6738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":6736,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2541,"num_write_kb":10164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6530","reported_seq":10638,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:05.619943+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:05.619943+0000","last_peered":"2024-09-16T10:25:05.619943+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:05.619943+0000","last_undegraded":"2024-09-16T10:25:05.619943+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719174,"num_objects":6525,"num_object_clones":0,"num_object_copies":13050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2189,"num_objects_unfound":0,"num_objects_dirty":6525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6528,"num_write_kb":26096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2010,"num_bytes_recovered":8224768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2189},{"shards":"2","objects":4336}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6982","reported_seq":7058,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.736077+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:09.736077+0000","last_peered":"2024-09-16T10:25:09.736077+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:09.736077+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6982,"log_dups_size":0,"ondisk_log_size":6982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28566470,"num_objects":6976,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6980,"num_write_kb":27900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2542},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6562","reported_seq":6665,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:05.579528+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:05.579528+0000","last_peered":"2024-09-16T10:25:05.579528+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:05.579528+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6562,"log_dups_size":0,"ondisk_log_size":6562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26858438,"num_objects":6559,"num_object_clones":0,"num_object_copies":13118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6560,"num_write_kb":26231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2282}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6631","reported_seq":6715,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:05.585566+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:05.585566+0000","last_peered":"2024-09-16T10:25:05.585566+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:05.585566+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6631,"log_dups_size":0,"ondisk_log_size":6631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27136966,"num_objects":6627,"num_object_clones":0,"num_object_copies":13254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6629,"num_write_kb":26503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2435}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6611","reported_seq":6688,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.681214+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:09.681214+0000","last_peered":"2024-09-16T10:25:09.681214+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:09.681214+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27070947,"num_objects":6610,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6610,"num_write_kb":26437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2451},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6937","reported_seq":7067,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.648825+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:09.648825+0000","last_peered":"2024-09-16T10:25:09.648825+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:09.648825+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6937,"log_dups_size":0,"ondisk_log_size":6937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28382633,"num_objects":6932,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6934,"num_write_kb":27720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2595},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6667","reported_seq":6742,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.738029+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:09.738029+0000","last_peered":"2024-09-16T10:25:09.738029+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:09.738029+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6667,"log_dups_size":0,"ondisk_log_size":6667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27292614,"num_objects":6665,"num_object_clones":0,"num_object_copies":13330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6665,"num_write_kb":26654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2498},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6971","reported_seq":7061,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.650528+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:09.650528+0000","last_peered":"2024-09-16T10:25:09.650528+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:09.650528+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6971,"log_dups_size":0,"ondisk_log_size":6971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505996,"num_objects":6963,"num_object_clones":0,"num_object_copies":13926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6968,"num_write_kb":27843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2568},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":441053686,"num_objects":107695,"num_object_clones":0,"num_object_copies":215390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2189,"num_objects_unfound":0,"num_objects_dirty":107695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85324,"num_write_kb":341184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2027,"num_bytes_recovered":8229347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":889958400,"data_stored":889875301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107759,"ondisk_log_size":107759,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018911,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528724,"kb_used_data":348620,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93843116,"statfs":{"total":96636764160,"available":96095350784,"internally_reserved":0,"allocated":356986880,"data_stored":356743641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":184413872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,9,2,1,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.498}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.248}]}]},{"osd":1,"up_from":40,"seq":171798691856,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228996,"kb_used_data":125180,"kb_used_omap":4,"kb_used_meta":103803,"kb_avail":94142844,"statfs":{"total":96636764160,"available":96402272256,"internally_reserved":0,"allocated":128184320,"data_stored":127861595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":106294954},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":3.7639999999999998},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":3.7930000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":3.3340000000000001},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":5.5149999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215140,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649508,"kb_used_data":393180,"kb_used_omap":4,"kb_used_meta":256315,"kb_avail":93722332,"statfs":{"total":96636764160,"available":95971667968,"internally_reserved":0,"allocated":402616320,"data_stored":402376695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":262467240},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,3,1,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":1.4950000000000001},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.1469999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.4560000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":403816448,"data_stored":403812835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129069056,"data_stored":128993183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357072896,"data_stored":357069283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:12.542 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:25:12.541229+0000 min lat: 0.00139595 max lat: 2.61294 avg lat: 0.0218557 2024-09-16T10:25:12.542 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:25:12.542 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 120 20 109814 109794 3.57322 2.01562 0.0260445 0.0218557 2024-09-16T10:25:13.542 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 121 20 110348 110328 3.56092 2.08594 0.0526077 0.0219317 2024-09-16T10:25:14.542 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 122 20 110879 110859 3.54874 2.07422 0.042614 0.0220073 2024-09-16T10:25:14.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:15.408 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:15.409 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:15.435 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-16T10:25:14.593752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":445039113,"num_objects":108670,"num_object_clones":0,"num_object_copies":217340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2030,"num_objects_unfound":0,"num_objects_dirty":108670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86299,"num_write_kb":345078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2189,"num_bytes_recovered":8880630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108735,"ondisk_log_size":108735,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1446932,"kb_used_data":880444,"kb_used_omap":12,"kb_used_meta":566451,"kb_avail":281668588,"statfs":{"total":289910292480,"available":288428634112,"internally_reserved":0,"allocated":901574656,"data_stored":900776293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":580045826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,2,6,4,8,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21381120,"num_objects":5220,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-328,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":20880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":330,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002995"},"pg_stats":[{"pgid":"2.f","version":"42'6754","reported_seq":6820,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.680090+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:09.680090+0000","last_peered":"2024-09-16T10:25:09.680090+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:09.680090+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6754,"log_dups_size":0,"ondisk_log_size":6754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27664384,"num_objects":6754,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":27016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2562},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6825","reported_seq":6979,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:10.692071+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:10.692071+0000","last_peered":"2024-09-16T10:25:10.692071+0000","last_clean":"2024-09-16T10:25:10.692071+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:10.692071+0000","last_undegraded":"2024-09-16T10:25:10.692071+0000","last_fullsized":"2024-09-16T10:25:10.692071+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27930624,"num_objects":6819,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3038,"num_write_kb":12152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6820","reported_seq":6997,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:10.594817+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:10.594817+0000","last_peered":"2024-09-16T10:25:10.594817+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:10.594817+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6820,"log_dups_size":0,"ondisk_log_size":6820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2877,"num_write_kb":11508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2481}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6801","reported_seq":6855,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:10.700877+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:10.700877+0000","last_peered":"2024-09-16T10:25:10.700877+0000","last_clean":"2024-09-16T10:25:10.700877+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:10.700877+0000","last_undegraded":"2024-09-16T10:25:10.700877+0000","last_fullsized":"2024-09-16T10:25:10.700877+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6801,"log_dups_size":0,"ondisk_log_size":6801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27856896,"num_objects":6801,"num_object_clones":0,"num_object_copies":13602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6801,"num_write_kb":27204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6775","reported_seq":6929,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:09.760220+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:09.760220+0000","last_peered":"2024-09-16T10:25:09.760220+0000","last_clean":"2024-09-16T10:25:09.760220+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:09.760220+0000","last_undegraded":"2024-09-16T10:25:09.760220+0000","last_fullsized":"2024-09-16T10:25:09.760220+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6775,"log_dups_size":0,"ondisk_log_size":6775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":6768,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3417,"num_write_kb":13668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6683","reported_seq":6844,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.637922+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:09.637922+0000","last_peered":"2024-09-16T10:25:09.637922+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:09.637922+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6683,"log_dups_size":0,"ondisk_log_size":6683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27353088,"num_objects":6678,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3308,"num_write_kb":13232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2461},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6959","reported_seq":7130,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.765226+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:09.765226+0000","last_peered":"2024-09-16T10:25:09.765226+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:09.765226+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28479488,"num_objects":6953,"num_object_clones":0,"num_object_copies":13906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3201,"num_write_kb":12804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2550},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6738","reported_seq":6910,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:09.758891+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:09.758891+0000","last_peered":"2024-09-16T10:25:09.758891+0000","last_clean":"2024-09-16T10:25:09.758891+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:09.758891+0000","last_undegraded":"2024-09-16T10:25:09.758891+0000","last_fullsized":"2024-09-16T10:25:09.758891+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6738,"log_dups_size":0,"ondisk_log_size":6738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":6736,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2541,"num_write_kb":10164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6700","reported_seq":11132,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:10.701699+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:10.701699+0000","last_peered":"2024-09-16T10:25:10.701699+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:10.701699+0000","last_undegraded":"2024-09-16T10:25:10.701699+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6700,"log_dups_size":0,"ondisk_log_size":6700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27415494,"num_objects":6695,"num_object_clones":0,"num_object_copies":13390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2030,"num_objects_unfound":0,"num_objects_dirty":6695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6698,"num_write_kb":26776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2171,"num_bytes_recovered":8876032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2030},{"shards":"2","objects":4665}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6982","reported_seq":7058,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.736077+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:09.736077+0000","last_peered":"2024-09-16T10:25:09.736077+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:09.736077+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6982,"log_dups_size":0,"ondisk_log_size":6982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28566470,"num_objects":6976,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6980,"num_write_kb":27900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2542},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6721","reported_seq":6824,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:10.663844+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:10.663844+0000","last_peered":"2024-09-16T10:25:10.663844+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:10.663844+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6721,"log_dups_size":0,"ondisk_log_size":6721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27509702,"num_objects":6718,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6719,"num_write_kb":26867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2441}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6788","reported_seq":6872,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:10.560163+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:10.560163+0000","last_peered":"2024-09-16T10:25:10.560163+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:10.560163+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6788,"log_dups_size":0,"ondisk_log_size":6788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27780038,"num_objects":6784,"num_object_clones":0,"num_object_copies":13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6786,"num_write_kb":27131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2592}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6611","reported_seq":6688,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.681214+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:09.681214+0000","last_peered":"2024-09-16T10:25:09.681214+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:09.681214+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27070947,"num_objects":6610,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6610,"num_write_kb":26437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2451},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6937","reported_seq":7067,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.648825+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:09.648825+0000","last_peered":"2024-09-16T10:25:09.648825+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:09.648825+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6937,"log_dups_size":0,"ondisk_log_size":6937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28382633,"num_objects":6932,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6934,"num_write_kb":27720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2595},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6667","reported_seq":6742,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.738029+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:09.738029+0000","last_peered":"2024-09-16T10:25:09.738029+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:09.738029+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6667,"log_dups_size":0,"ondisk_log_size":6667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27292614,"num_objects":6665,"num_object_clones":0,"num_object_copies":13330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6665,"num_write_kb":26654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2498},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6971","reported_seq":7061,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.650528+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:09.650528+0000","last_peered":"2024-09-16T10:25:09.650528+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:09.650528+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6971,"log_dups_size":0,"ondisk_log_size":6971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505996,"num_objects":6963,"num_object_clones":0,"num_object_copies":13926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6968,"num_write_kb":27843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2568},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":445039094,"num_objects":108668,"num_object_clones":0,"num_object_copies":217336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2030,"num_objects_unfound":0,"num_objects_dirty":108668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86297,"num_write_kb":345076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2188,"num_bytes_recovered":8880611,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":903778304,"data_stored":903695205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108732,"ondisk_log_size":108732,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018912,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538040,"kb_used_data":353840,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93833800,"statfs":{"total":96636764160,"available":96085811200,"internally_reserved":0,"allocated":362332160,"data_stored":362088921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":188608176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":6.6299999999999999},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":3.0369999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":6.7709999999999999},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":6.5179999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691857,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259384,"kb_used_data":133424,"kb_used_omap":4,"kb_used_meta":125947,"kb_avail":94112456,"statfs":{"total":96636764160,"available":96371154944,"internally_reserved":0,"allocated":136626176,"data_stored":136310677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":128970410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.58699999999999997},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.45200000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":0.41299999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215140,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649508,"kb_used_data":393180,"kb_used_omap":4,"kb_used_meta":256315,"kb_avail":93722332,"statfs":{"total":96636764160,"available":95971667968,"internally_reserved":0,"allocated":402616320,"data_stored":402376695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":262467240},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,3,1,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":1.4950000000000001},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.1469999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.4560000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":403816448,"data_stored":403812835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137510912,"data_stored":137435039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362450944,"data_stored":362447331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:15.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:15.543 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 123 20 111370 111350 3.53544 1.91797 0.0543199 0.0220897 2024-09-16T10:25:15.990 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:15.993 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:16.015 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-16T10:25:14.593752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":445039113,"num_objects":108670,"num_object_clones":0,"num_object_copies":217340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2030,"num_objects_unfound":0,"num_objects_dirty":108670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86299,"num_write_kb":345078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2189,"num_bytes_recovered":8880630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108735,"ondisk_log_size":108735,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1446932,"kb_used_data":880444,"kb_used_omap":12,"kb_used_meta":566451,"kb_avail":281668588,"statfs":{"total":289910292480,"available":288428634112,"internally_reserved":0,"allocated":901574656,"data_stored":900776293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":580045826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,10,2,6,4,8,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21381120,"num_objects":5220,"num_object_clones":0,"num_object_copies":10440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-328,"num_objects_unfound":0,"num_objects_dirty":5220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5220,"num_write_kb":20880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":330,"num_bytes_recovered":1343488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002995"},"pg_stats":[{"pgid":"2.f","version":"42'6754","reported_seq":6820,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.680090+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:09.680090+0000","last_peered":"2024-09-16T10:25:09.680090+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:09.680090+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6754,"log_dups_size":0,"ondisk_log_size":6754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27664384,"num_objects":6754,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6754,"num_write_kb":27016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2562},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6825","reported_seq":6979,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:10.692071+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:10.692071+0000","last_peered":"2024-09-16T10:25:10.692071+0000","last_clean":"2024-09-16T10:25:10.692071+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:10.692071+0000","last_undegraded":"2024-09-16T10:25:10.692071+0000","last_fullsized":"2024-09-16T10:25:10.692071+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6825,"log_dups_size":0,"ondisk_log_size":6825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27930624,"num_objects":6819,"num_object_clones":0,"num_object_copies":13638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3038,"num_write_kb":12152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6820","reported_seq":6997,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:10.594817+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:10.594817+0000","last_peered":"2024-09-16T10:25:10.594817+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:10.594817+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6820,"log_dups_size":0,"ondisk_log_size":6820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2877,"num_write_kb":11508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2481}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6801","reported_seq":6855,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:10.700877+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:10.700877+0000","last_peered":"2024-09-16T10:25:10.700877+0000","last_clean":"2024-09-16T10:25:10.700877+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:10.700877+0000","last_undegraded":"2024-09-16T10:25:10.700877+0000","last_fullsized":"2024-09-16T10:25:10.700877+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6801,"log_dups_size":0,"ondisk_log_size":6801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27856896,"num_objects":6801,"num_object_clones":0,"num_object_copies":13602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6801,"num_write_kb":27204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6775","reported_seq":6929,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:09.760220+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:09.760220+0000","last_peered":"2024-09-16T10:25:09.760220+0000","last_clean":"2024-09-16T10:25:09.760220+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:09.760220+0000","last_undegraded":"2024-09-16T10:25:09.760220+0000","last_fullsized":"2024-09-16T10:25:09.760220+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6775,"log_dups_size":0,"ondisk_log_size":6775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":6768,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3417,"num_write_kb":13668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6683","reported_seq":6844,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.637922+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:09.637922+0000","last_peered":"2024-09-16T10:25:09.637922+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:09.637922+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6683,"log_dups_size":0,"ondisk_log_size":6683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27353088,"num_objects":6678,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3308,"num_write_kb":13232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2461},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'6959","reported_seq":7130,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.765226+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:09.765226+0000","last_peered":"2024-09-16T10:25:09.765226+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:09.765226+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":6959,"log_dups_size":0,"ondisk_log_size":6959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28479488,"num_objects":6953,"num_object_clones":0,"num_object_copies":13906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3201,"num_write_kb":12804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2550},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6738","reported_seq":6910,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:09.758891+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:09.758891+0000","last_peered":"2024-09-16T10:25:09.758891+0000","last_clean":"2024-09-16T10:25:09.758891+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:09.758891+0000","last_undegraded":"2024-09-16T10:25:09.758891+0000","last_fullsized":"2024-09-16T10:25:09.758891+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6738,"log_dups_size":0,"ondisk_log_size":6738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27590656,"num_objects":6736,"num_object_clones":0,"num_object_copies":13472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2541,"num_write_kb":10164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6700","reported_seq":11132,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:10.701699+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:10.701699+0000","last_peered":"2024-09-16T10:25:10.701699+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:10.701699+0000","last_undegraded":"2024-09-16T10:25:10.701699+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6700,"log_dups_size":0,"ondisk_log_size":6700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27415494,"num_objects":6695,"num_object_clones":0,"num_object_copies":13390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2030,"num_objects_unfound":0,"num_objects_dirty":6695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6698,"num_write_kb":26776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2171,"num_bytes_recovered":8876032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2030},{"shards":"2","objects":4665}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'6982","reported_seq":7058,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.736077+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:09.736077+0000","last_peered":"2024-09-16T10:25:09.736077+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:09.736077+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6982,"log_dups_size":0,"ondisk_log_size":6982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28566470,"num_objects":6976,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6980,"num_write_kb":27900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2542},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6721","reported_seq":6824,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:10.663844+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:10.663844+0000","last_peered":"2024-09-16T10:25:10.663844+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:10.663844+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6721,"log_dups_size":0,"ondisk_log_size":6721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27509702,"num_objects":6718,"num_object_clones":0,"num_object_copies":13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6719,"num_write_kb":26867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2441}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6788","reported_seq":6872,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:10.560163+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:10.560163+0000","last_peered":"2024-09-16T10:25:10.560163+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:10.560163+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6788,"log_dups_size":0,"ondisk_log_size":6788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27780038,"num_objects":6784,"num_object_clones":0,"num_object_copies":13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6786,"num_write_kb":27131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2592}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6611","reported_seq":6688,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.681214+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:09.681214+0000","last_peered":"2024-09-16T10:25:09.681214+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:09.681214+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6611,"log_dups_size":0,"ondisk_log_size":6611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27070947,"num_objects":6610,"num_object_clones":0,"num_object_copies":13220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6610,"num_write_kb":26437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2451},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'6937","reported_seq":7067,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.648825+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:09.648825+0000","last_peered":"2024-09-16T10:25:09.648825+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:09.648825+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":6937,"log_dups_size":0,"ondisk_log_size":6937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28382633,"num_objects":6932,"num_object_clones":0,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6934,"num_write_kb":27720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2595},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6667","reported_seq":6742,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.738029+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:09.738029+0000","last_peered":"2024-09-16T10:25:09.738029+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:09.738029+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6667,"log_dups_size":0,"ondisk_log_size":6667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27292614,"num_objects":6665,"num_object_clones":0,"num_object_copies":13330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6665,"num_write_kb":26654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2498},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'6971","reported_seq":7061,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:09.650528+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:09.650528+0000","last_peered":"2024-09-16T10:25:09.650528+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:09.650528+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6971,"log_dups_size":0,"ondisk_log_size":6971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28505996,"num_objects":6963,"num_object_clones":0,"num_object_copies":13926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6968,"num_write_kb":27843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2568},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":445039094,"num_objects":108668,"num_object_clones":0,"num_object_copies":217336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":2030,"num_objects_unfound":0,"num_objects_dirty":108668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86297,"num_write_kb":345076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2188,"num_bytes_recovered":8880611,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":903778304,"data_stored":903695205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108732,"ondisk_log_size":108732,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018912,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538040,"kb_used_data":353840,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93833800,"statfs":{"total":96636764160,"available":96085811200,"internally_reserved":0,"allocated":362332160,"data_stored":362088921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":188608176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":6.6299999999999999},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":3.0369999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":6.7709999999999999},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":6.5179999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691857,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259384,"kb_used_data":133424,"kb_used_omap":4,"kb_used_meta":125947,"kb_avail":94112456,"statfs":{"total":96636764160,"available":96371154944,"internally_reserved":0,"allocated":136626176,"data_stored":136310677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":128970410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.58699999999999997},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.45200000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":0.41299999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215140,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":649508,"kb_used_data":393180,"kb_used_omap":4,"kb_used_meta":256315,"kb_avail":93722332,"statfs":{"total":96636764160,"available":95971667968,"internally_reserved":0,"allocated":402616320,"data_stored":402376695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":262467240},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,3,1,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":1.4950000000000001},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.1469999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.83499999999999996},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.4560000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":403816448,"data_stored":403812835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137510912,"data_stored":137435039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362450944,"data_stored":362447331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:16.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:16.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'6754", 'reported_seq': 6820, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:09.680090+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:25:09.680090+0000', 'last_peered': '2024-09-16T10:25:09.680090+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:25:09.680090+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6754, 'log_dups_size': 0, 'ondisk_log_size': 6754, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27664384, 'num_objects': 6754, 'num_object_clones': 0, 'num_object_copies': 13508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6754, 'num_write_kb': 27016, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2562}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:16.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:16.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'6820", 'reported_seq': 6997, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:10.594817+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:10.594817+0000', 'last_peered': '2024-09-16T10:25:10.594817+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:10.594817+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 6820, 'log_dups_size': 0, 'ondisk_log_size': 6820, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27918336, 'num_objects': 6816, 'num_object_clones': 0, 'num_object_copies': 13632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6816, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2877, 'num_write_kb': 11508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 2481}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:16.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:16.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'6683", 'reported_seq': 6844, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:09.637922+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:25:09.637922+0000', 'last_peered': '2024-09-16T10:25:09.637922+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:25:09.637922+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 6683, 'log_dups_size': 0, 'ondisk_log_size': 6683, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27353088, 'num_objects': 6678, 'num_object_clones': 0, 'num_object_copies': 13356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3308, 'num_write_kb': 13232, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2461}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:16.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:16.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'6959", 'reported_seq': 7130, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:09.765226+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:25:09.765226+0000', 'last_peered': '2024-09-16T10:25:09.765226+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:25:09.765226+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 6959, 'log_dups_size': 0, 'ondisk_log_size': 6959, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28479488, 'num_objects': 6953, 'num_object_clones': 0, 'num_object_copies': 13906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3201, 'num_write_kb': 12804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2550}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:16.017 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'6700", 'reported_seq': 11132, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:10.701699+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:10.701699+0000', 'last_peered': '2024-09-16T10:25:10.701699+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:10.701699+0000', 'last_undegraded': '2024-09-16T10:25:10.701699+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6700, 'log_dups_size': 0, 'ondisk_log_size': 6700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27415494, 'num_objects': 6695, 'num_object_clones': 0, 'num_object_copies': 13390, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2030, 'num_objects_unfound': 0, 'num_objects_dirty': 6695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6698, 'num_write_kb': 26776, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2171, 'num_bytes_recovered': 8876032, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2030}, {'shards': '2', 'objects': 4665}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'6982", 'reported_seq': 7058, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:09.736077+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:25:09.736077+0000', 'last_peered': '2024-09-16T10:25:09.736077+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:25:09.736077+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6982, 'log_dups_size': 0, 'ondisk_log_size': 6982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28566470, 'num_objects': 6976, 'num_object_clones': 0, 'num_object_copies': 13952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6980, 'num_write_kb': 27900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2542}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'6721", 'reported_seq': 6824, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:10.663844+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:10.663844+0000', 'last_peered': '2024-09-16T10:25:10.663844+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:10.663844+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 6721, 'log_dups_size': 0, 'ondisk_log_size': 6721, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27509702, 'num_objects': 6718, 'num_object_clones': 0, 'num_object_copies': 13436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6719, 'num_write_kb': 26867, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 2441}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'6788", 'reported_seq': 6872, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:10.560163+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:10.560163+0000', 'last_peered': '2024-09-16T10:25:10.560163+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:10.560163+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6788, 'log_dups_size': 0, 'ondisk_log_size': 6788, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27780038, 'num_objects': 6784, 'num_object_clones': 0, 'num_object_copies': 13568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6786, 'num_write_kb': 27131, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 2592}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'6611", 'reported_seq': 6688, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:09.681214+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:25:09.681214+0000', 'last_peered': '2024-09-16T10:25:09.681214+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:25:09.681214+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6611, 'log_dups_size': 0, 'ondisk_log_size': 6611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27070947, 'num_objects': 6610, 'num_object_clones': 0, 'num_object_copies': 13220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6610, 'num_write_kb': 26437, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2451}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:16.018 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'6937", 'reported_seq': 7067, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:09.648825+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:09.648825+0000', 'last_peered': '2024-09-16T10:25:09.648825+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:09.648825+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 6937, 'log_dups_size': 0, 'ondisk_log_size': 6937, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28382633, 'num_objects': 6932, 'num_object_clones': 0, 'num_object_copies': 13864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6934, 'num_write_kb': 27720, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2595}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:16.019 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:16.019 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'6667", 'reported_seq': 6742, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:09.738029+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:09.738029+0000', 'last_peered': '2024-09-16T10:25:09.738029+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:09.738029+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6667, 'log_dups_size': 0, 'ondisk_log_size': 6667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27292614, 'num_objects': 6665, 'num_object_clones': 0, 'num_object_copies': 13330, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6665, 'num_write_kb': 26654, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2498}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:16.019 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:16.019 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'6971", 'reported_seq': 7061, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:09.650528+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:25:09.650528+0000', 'last_peered': '2024-09-16T10:25:09.650528+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:25:09.650528+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6971, 'log_dups_size': 0, 'ondisk_log_size': 6971, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28505996, 'num_objects': 6963, 'num_object_clones': 0, 'num_object_copies': 13926, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6963, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6968, 'num_write_kb': 27843, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2568}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:16.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:16.543 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 124 20 111823 111803 3.5212 1.76953 0.0139817 0.0221779 2024-09-16T10:25:16.698 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:16.698 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/217340 objects degraded (21.684%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":205,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":108670,"data_bytes":445039113,"bytes_used":1481658368,"bytes_avail":288428634112,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":217340,"degraded_ratio":0.21683997423391921,"misplaced_objects":2030,"misplaced_total":217340,"misplaced_ratio":0.009340204288212018,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":111929,"recovering_keys_per_sec":0,"num_objects_recovered":330,"num_bytes_recovered":1343488,"num_keys_recovered":0,"write_bytes_sec":1781315,"read_op_per_sec":0,"write_op_per_sec":434},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:16.721 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:16.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:17.268 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:17.271 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:17.293 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-16T10:25:16.597846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":455442953,"num_objects":111210,"num_object_clones":0,"num_object_copies":222420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1864,"num_objects_unfound":0,"num_objects_dirty":111210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88839,"num_write_kb":355238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2355,"num_bytes_recovered":9560566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111275,"ondisk_log_size":111275,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1457140,"kb_used_data":893084,"kb_used_omap":12,"kb_used_meta":564019,"kb_avail":281658380,"statfs":{"total":289910292480,"available":288418181120,"internally_reserved":0,"allocated":914518016,"data_stored":913719653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":577555458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,6,6,7,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31784960,"num_objects":7760,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-494,"num_objects_unfound":0,"num_objects_dirty":7760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7760,"num_write_kb":31040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":2023424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006568"},"pg_stats":[{"pgid":"2.f","version":"42'6922","reported_seq":6988,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:14.703462+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:14.703462+0000","last_peered":"2024-09-16T10:25:14.703462+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:14.703462+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6922,"log_dups_size":0,"ondisk_log_size":6922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28352512,"num_objects":6922,"num_object_clones":0,"num_object_copies":13844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6922,"num_write_kb":27688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2730},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6995","reported_seq":7149,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:15.646184+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:15.646184+0000","last_peered":"2024-09-16T10:25:15.646184+0000","last_clean":"2024-09-16T10:25:15.646184+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:15.646184+0000","last_undegraded":"2024-09-16T10:25:15.646184+0000","last_fullsized":"2024-09-16T10:25:15.646184+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6995,"log_dups_size":0,"ondisk_log_size":6995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28626944,"num_objects":6989,"num_object_clones":0,"num_object_copies":13978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3208,"num_write_kb":12832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6968","reported_seq":7150,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:15.668347+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:15.668347+0000","last_peered":"2024-09-16T10:25:15.668347+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:15.668347+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6968,"log_dups_size":0,"ondisk_log_size":6968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":6964,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3025,"num_write_kb":12100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6957","reported_seq":7008,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:15.675765+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:15.675765+0000","last_peered":"2024-09-16T10:25:15.675765+0000","last_clean":"2024-09-16T10:25:15.675765+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:15.675765+0000","last_undegraded":"2024-09-16T10:25:15.675765+0000","last_fullsized":"2024-09-16T10:25:15.675765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6957,"log_dups_size":0,"ondisk_log_size":6957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28495872,"num_objects":6957,"num_object_clones":0,"num_object_copies":13914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6957,"num_write_kb":27828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'6933","reported_seq":7086,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:14.760318+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:14.760318+0000","last_peered":"2024-09-16T10:25:14.760318+0000","last_clean":"2024-09-16T10:25:14.760318+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:14.760318+0000","last_undegraded":"2024-09-16T10:25:14.760318+0000","last_fullsized":"2024-09-16T10:25:14.760318+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":6933,"log_dups_size":0,"ondisk_log_size":6933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28368896,"num_objects":6926,"num_object_clones":0,"num_object_copies":13852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3575,"num_write_kb":14300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6832","reported_seq":6992,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:14.760826+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:14.760826+0000","last_peered":"2024-09-16T10:25:14.760826+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:14.760826+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6832,"log_dups_size":0,"ondisk_log_size":6832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27963392,"num_objects":6827,"num_object_clones":0,"num_object_copies":13654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3457,"num_write_kb":13828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2610},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7134","reported_seq":7304,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:14.799384+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:14.799384+0000","last_peered":"2024-09-16T10:25:14.799384+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:14.799384+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7134,"log_dups_size":0,"ondisk_log_size":7134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29196288,"num_objects":7128,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3376,"num_write_kb":13504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2725},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'6897","reported_seq":7069,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:14.760298+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:14.760298+0000","last_peered":"2024-09-16T10:25:14.760298+0000","last_clean":"2024-09-16T10:25:14.760298+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:14.760298+0000","last_undegraded":"2024-09-16T10:25:14.760298+0000","last_fullsized":"2024-09-16T10:25:14.760298+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":6897,"log_dups_size":0,"ondisk_log_size":6897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28241920,"num_objects":6895,"num_object_clones":0,"num_object_copies":13790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2700,"num_write_kb":10800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6874","reported_seq":11638,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:15.721472+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:15.721472+0000","last_peered":"2024-09-16T10:25:15.721472+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:15.721472+0000","last_undegraded":"2024-09-16T10:25:15.721472+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6874,"log_dups_size":0,"ondisk_log_size":6874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28128198,"num_objects":6869,"num_object_clones":0,"num_object_copies":13738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1864,"num_objects_unfound":0,"num_objects_dirty":6869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":27472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2337,"num_bytes_recovered":9555968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1864},{"shards":"2","objects":5005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7112","reported_seq":7188,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:14.512381+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:14.512381+0000","last_peered":"2024-09-16T10:25:14.512381+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:14.512381+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7112,"log_dups_size":0,"ondisk_log_size":7112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29098950,"num_objects":7106,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":28420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2672},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6890","reported_seq":6993,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:15.710016+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:15.710016+0000","last_peered":"2024-09-16T10:25:15.710016+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:15.710016+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6890,"log_dups_size":0,"ondisk_log_size":6890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28201926,"num_objects":6887,"num_object_clones":0,"num_object_copies":13774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6888,"num_write_kb":27543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6950","reported_seq":7034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:15.698850+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:15.698850+0000","last_peered":"2024-09-16T10:25:15.698850+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:15.698850+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28443590,"num_objects":6946,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6948,"num_write_kb":27779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2754}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6777","reported_seq":6854,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:14.741187+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:14.741187+0000","last_peered":"2024-09-16T10:25:14.741187+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:14.741187+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6777,"log_dups_size":0,"ondisk_log_size":6777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27750883,"num_objects":6776,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":27101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2617},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7102","reported_seq":7231,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:14.780675+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:14.780675+0000","last_peered":"2024-09-16T10:25:14.780675+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:14.780675+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7102,"log_dups_size":0,"ondisk_log_size":7102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29058473,"num_objects":7097,"num_object_clones":0,"num_object_copies":14194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7099,"num_write_kb":28380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2760},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6815","reported_seq":6893,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:14.724958+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:14.724958+0000","last_peered":"2024-09-16T10:25:14.724958+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:14.724958+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6815,"log_dups_size":0,"ondisk_log_size":6815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27898822,"num_objects":6813,"num_object_clones":0,"num_object_copies":13626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6813,"num_write_kb":27246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2646},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7114","reported_seq":7204,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:14.752766+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:14.752766+0000","last_peered":"2024-09-16T10:25:14.752766+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:14.752766+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7114,"log_dups_size":0,"ondisk_log_size":7114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29091724,"num_objects":7106,"num_object_clones":0,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7111,"num_write_kb":28415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2711},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":455442934,"num_objects":111208,"num_object_clones":0,"num_object_copies":222416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1864,"num_objects_unfound":0,"num_objects_dirty":111208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88837,"num_write_kb":355236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2354,"num_bytes_recovered":9560547,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":916848640,"data_stored":916765541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111272,"ondisk_log_size":111272,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018913,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530972,"kb_used_data":359060,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93840868,"statfs":{"total":96636764160,"available":96093048832,"internally_reserved":0,"allocated":367677440,"data_stored":367434201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":176025264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.26800000000000002},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":2.0270000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":1.9430000000000001},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.315}]}]},{"osd":1,"up_from":40,"seq":171798691857,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259384,"kb_used_data":133424,"kb_used_omap":4,"kb_used_meta":125947,"kb_avail":94112456,"statfs":{"total":96636764160,"available":96371154944,"internally_reserved":0,"allocated":136626176,"data_stored":136310677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":128970410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.58699999999999997},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.45200000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":0.41299999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215141,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":666784,"kb_used_data":400600,"kb_used_omap":4,"kb_used_meta":266171,"kb_avail":93705056,"statfs":{"total":96636764160,"available":95953977344,"internally_reserved":0,"allocated":410214400,"data_stored":409974775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":272559784},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":1.222},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.92500000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.83899999999999997},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.89000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":411545600,"data_stored":411541987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":137510912,"data_stored":137435039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367792128,"data_stored":367788515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:17.543 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 125 20 112295 112275 3.50778 1.84375 0.0183406 0.0222645 2024-09-16T10:25:18.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 126 20 112810 112790 3.49591 2.01172 0.0279654 0.0223391 2024-09-16T10:25:19.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 127 20 113337 113317 3.48459 2.05859 0.0108323 0.0224103 2024-09-16T10:25:20.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:20.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 128 20 113852 113832 3.47309 2.01172 0.115648 0.0224857 2024-09-16T10:25:20.851 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:20.853 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:20.876 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-16T10:25:20.598917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":461681161,"num_objects":112733,"num_object_clones":0,"num_object_copies":225466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1864,"num_objects_unfound":0,"num_objects_dirty":112733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90362,"num_write_kb":361330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2355,"num_bytes_recovered":9560566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112798,"ondisk_log_size":112798,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1468736,"kb_used_data":908776,"kb_used_omap":12,"kb_used_meta":559923,"kb_avail":281646784,"statfs":{"total":289910292480,"available":288406306816,"internally_reserved":0,"allocated":930586624,"data_stored":929788261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":573361154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,12,3,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27361280,"num_objects":6680,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-325,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":26720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":1331200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006654"},"pg_stats":[{"pgid":"2.f","version":"42'7087","reported_seq":7151,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.841581+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:19.841581+0000","last_peered":"2024-09-16T10:25:19.841581+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:19.841581+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29028352,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":28348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6995","reported_seq":7149,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:15.646184+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:15.646184+0000","last_peered":"2024-09-16T10:25:15.646184+0000","last_clean":"2024-09-16T10:25:15.646184+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:15.646184+0000","last_undegraded":"2024-09-16T10:25:15.646184+0000","last_fullsized":"2024-09-16T10:25:15.646184+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6995,"log_dups_size":0,"ondisk_log_size":6995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28626944,"num_objects":6989,"num_object_clones":0,"num_object_copies":13978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3208,"num_write_kb":12832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6968","reported_seq":7150,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:15.668347+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:15.668347+0000","last_peered":"2024-09-16T10:25:15.668347+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:15.668347+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6968,"log_dups_size":0,"ondisk_log_size":6968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":6964,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3025,"num_write_kb":12100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6957","reported_seq":7008,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:15.675765+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:15.675765+0000","last_peered":"2024-09-16T10:25:15.675765+0000","last_clean":"2024-09-16T10:25:15.675765+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:15.675765+0000","last_undegraded":"2024-09-16T10:25:15.675765+0000","last_fullsized":"2024-09-16T10:25:15.675765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6957,"log_dups_size":0,"ondisk_log_size":6957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28495872,"num_objects":6957,"num_object_clones":0,"num_object_copies":13914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6957,"num_write_kb":27828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7083","reported_seq":7237,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:19.794137+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:19.794137+0000","last_peered":"2024-09-16T10:25:19.794137+0000","last_clean":"2024-09-16T10:25:19.794137+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:19.794137+0000","last_undegraded":"2024-09-16T10:25:19.794137+0000","last_fullsized":"2024-09-16T10:25:19.794137+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7083,"log_dups_size":0,"ondisk_log_size":7083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28983296,"num_objects":7076,"num_object_clones":0,"num_object_copies":14152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":14900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6992","reported_seq":7152,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.777229+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:19.777229+0000","last_peered":"2024-09-16T10:25:19.777229+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:19.777229+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6992,"log_dups_size":0,"ondisk_log_size":6992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28618752,"num_objects":6987,"num_object_clones":0,"num_object_copies":13974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":14468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2770},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7292","reported_seq":7463,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.827896+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:19.827896+0000","last_peered":"2024-09-16T10:25:19.827896+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:19.827896+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7292,"log_dups_size":0,"ondisk_log_size":7292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29843456,"num_objects":7286,"num_object_clones":0,"num_object_copies":14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3534,"num_write_kb":14136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2883},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7054","reported_seq":7226,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:19.719169+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:19.719169+0000","last_peered":"2024-09-16T10:25:19.719169+0000","last_clean":"2024-09-16T10:25:19.719169+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:19.719169+0000","last_undegraded":"2024-09-16T10:25:19.719169+0000","last_fullsized":"2024-09-16T10:25:19.719169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28884992,"num_objects":7052,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6874","reported_seq":11638,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:15.721472+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:15.721472+0000","last_peered":"2024-09-16T10:25:15.721472+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:15.721472+0000","last_undegraded":"2024-09-16T10:25:15.721472+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6874,"log_dups_size":0,"ondisk_log_size":6874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28128198,"num_objects":6869,"num_object_clones":0,"num_object_copies":13738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1864,"num_objects_unfound":0,"num_objects_dirty":6869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":27472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2337,"num_bytes_recovered":9555968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1864},{"shards":"2","objects":5005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7242","reported_seq":7318,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.585909+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:19.585909+0000","last_peered":"2024-09-16T10:25:19.585909+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:19.585909+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29631430,"num_objects":7236,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":28940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2802},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6890","reported_seq":6993,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:15.710016+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:15.710016+0000","last_peered":"2024-09-16T10:25:15.710016+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:15.710016+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6890,"log_dups_size":0,"ondisk_log_size":6890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28201926,"num_objects":6887,"num_object_clones":0,"num_object_copies":13774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6888,"num_write_kb":27543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6950","reported_seq":7034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:15.698850+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:15.698850+0000","last_peered":"2024-09-16T10:25:15.698850+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:15.698850+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28443590,"num_objects":6946,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6948,"num_write_kb":27779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2754}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6929","reported_seq":7006,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.747622+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:19.747622+0000","last_peered":"2024-09-16T10:25:19.747622+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:19.747622+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6929,"log_dups_size":0,"ondisk_log_size":6929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28373475,"num_objects":6928,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6928,"num_write_kb":27709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2769},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7272","reported_seq":7402,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.761106+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:19.761106+0000","last_peered":"2024-09-16T10:25:19.761106+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:19.761106+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29754793,"num_objects":7267,"num_object_clones":0,"num_object_copies":14534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7269,"num_write_kb":29060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2930},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6946","reported_seq":7024,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.666765+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:19.666765+0000","last_peered":"2024-09-16T10:25:19.666765+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:19.666765+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6946,"log_dups_size":0,"ondisk_log_size":6946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28435398,"num_objects":6944,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6944,"num_write_kb":27770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2777},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7264","reported_seq":7354,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.812289+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:19.812289+0000","last_peered":"2024-09-16T10:25:19.812289+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:19.812289+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29706124,"num_objects":7256,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7261,"num_write_kb":29015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2861},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":461681142,"num_objects":112731,"num_object_clones":0,"num_object_copies":225462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1864,"num_objects_unfound":0,"num_objects_dirty":112731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90360,"num_write_kb":361328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2354,"num_bytes_recovered":9560547,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":932601856,"data_stored":932518757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112795,"ondisk_log_size":112795,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018913,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530972,"kb_used_data":359060,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93840868,"statfs":{"total":96636764160,"available":96093048832,"internally_reserved":0,"allocated":367677440,"data_stored":367434201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":176025264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.26800000000000002},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":2.0270000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":1.9430000000000001},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.315}]}]},{"osd":1,"up_from":40,"seq":171798691858,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259412,"kb_used_data":141644,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94112428,"statfs":{"total":96636764160,"available":96371126272,"internally_reserved":0,"allocated":145043456,"data_stored":144727957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":120581802},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":2.9569999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":2.9900000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":3.1930000000000001},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":3.2330000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215142,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678352,"kb_used_data":408072,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93693488,"statfs":{"total":96636764160,"available":95942131712,"internally_reserved":0,"allocated":417865728,"data_stored":417626103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":276754088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":4.0460000000000003},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.57199999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.0049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":419065856,"data_stored":419062243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":145743872,"data_stored":145667999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367792128,"data_stored":367788515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:20.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:21.444 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:21.447 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:21.471 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-16T10:25:20.598917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":461681161,"num_objects":112733,"num_object_clones":0,"num_object_copies":225466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1864,"num_objects_unfound":0,"num_objects_dirty":112733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90362,"num_write_kb":361330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2355,"num_bytes_recovered":9560566,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112798,"ondisk_log_size":112798,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1468736,"kb_used_data":908776,"kb_used_omap":12,"kb_used_meta":559923,"kb_avail":281646784,"statfs":{"total":289910292480,"available":288406306816,"internally_reserved":0,"allocated":930586624,"data_stored":929788261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":573361154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,12,3,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27361280,"num_objects":6680,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-325,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6680,"num_write_kb":26720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":1331200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006654"},"pg_stats":[{"pgid":"2.f","version":"42'7087","reported_seq":7151,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.841581+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:19.841581+0000","last_peered":"2024-09-16T10:25:19.841581+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:19.841581+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29028352,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":28348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'6995","reported_seq":7149,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:15.646184+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:15.646184+0000","last_peered":"2024-09-16T10:25:15.646184+0000","last_clean":"2024-09-16T10:25:15.646184+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:15.646184+0000","last_undegraded":"2024-09-16T10:25:15.646184+0000","last_fullsized":"2024-09-16T10:25:15.646184+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":6995,"log_dups_size":0,"ondisk_log_size":6995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28626944,"num_objects":6989,"num_object_clones":0,"num_object_copies":13978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3208,"num_write_kb":12832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'6968","reported_seq":7150,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:15.668347+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:15.668347+0000","last_peered":"2024-09-16T10:25:15.668347+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:15.668347+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":6968,"log_dups_size":0,"ondisk_log_size":6968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28524544,"num_objects":6964,"num_object_clones":0,"num_object_copies":13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3025,"num_write_kb":12100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'6957","reported_seq":7008,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:15.675765+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:15.675765+0000","last_peered":"2024-09-16T10:25:15.675765+0000","last_clean":"2024-09-16T10:25:15.675765+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:15.675765+0000","last_undegraded":"2024-09-16T10:25:15.675765+0000","last_fullsized":"2024-09-16T10:25:15.675765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6957,"log_dups_size":0,"ondisk_log_size":6957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28495872,"num_objects":6957,"num_object_clones":0,"num_object_copies":13914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6957,"num_write_kb":27828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7083","reported_seq":7237,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:19.794137+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:19.794137+0000","last_peered":"2024-09-16T10:25:19.794137+0000","last_clean":"2024-09-16T10:25:19.794137+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:19.794137+0000","last_undegraded":"2024-09-16T10:25:19.794137+0000","last_fullsized":"2024-09-16T10:25:19.794137+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7083,"log_dups_size":0,"ondisk_log_size":7083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28983296,"num_objects":7076,"num_object_clones":0,"num_object_copies":14152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":14900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6992","reported_seq":7152,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.777229+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:19.777229+0000","last_peered":"2024-09-16T10:25:19.777229+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:19.777229+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6992,"log_dups_size":0,"ondisk_log_size":6992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28618752,"num_objects":6987,"num_object_clones":0,"num_object_copies":13974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":14468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2770},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7292","reported_seq":7463,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.827896+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:19.827896+0000","last_peered":"2024-09-16T10:25:19.827896+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:19.827896+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7292,"log_dups_size":0,"ondisk_log_size":7292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29843456,"num_objects":7286,"num_object_clones":0,"num_object_copies":14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3534,"num_write_kb":14136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2883},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7054","reported_seq":7226,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:19.719169+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:19.719169+0000","last_peered":"2024-09-16T10:25:19.719169+0000","last_clean":"2024-09-16T10:25:19.719169+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:19.719169+0000","last_undegraded":"2024-09-16T10:25:19.719169+0000","last_fullsized":"2024-09-16T10:25:19.719169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28884992,"num_objects":7052,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'6874","reported_seq":11638,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:15.721472+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:15.721472+0000","last_peered":"2024-09-16T10:25:15.721472+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:15.721472+0000","last_undegraded":"2024-09-16T10:25:15.721472+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6874,"log_dups_size":0,"ondisk_log_size":6874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28128198,"num_objects":6869,"num_object_clones":0,"num_object_copies":13738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1864,"num_objects_unfound":0,"num_objects_dirty":6869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":27472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2337,"num_bytes_recovered":9555968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1864},{"shards":"2","objects":5005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7242","reported_seq":7318,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.585909+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:19.585909+0000","last_peered":"2024-09-16T10:25:19.585909+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:19.585909+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29631430,"num_objects":7236,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":28940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2802},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'6890","reported_seq":6993,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:15.710016+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:15.710016+0000","last_peered":"2024-09-16T10:25:15.710016+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:15.710016+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":6890,"log_dups_size":0,"ondisk_log_size":6890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28201926,"num_objects":6887,"num_object_clones":0,"num_object_copies":13774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6888,"num_write_kb":27543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2610}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'6950","reported_seq":7034,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:15.698850+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:15.698850+0000","last_peered":"2024-09-16T10:25:15.698850+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:15.698850+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6950,"log_dups_size":0,"ondisk_log_size":6950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28443590,"num_objects":6946,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6948,"num_write_kb":27779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2754}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6929","reported_seq":7006,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.747622+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:19.747622+0000","last_peered":"2024-09-16T10:25:19.747622+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:19.747622+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6929,"log_dups_size":0,"ondisk_log_size":6929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28373475,"num_objects":6928,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6928,"num_write_kb":27709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2769},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7272","reported_seq":7402,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.761106+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:19.761106+0000","last_peered":"2024-09-16T10:25:19.761106+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:19.761106+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29754793,"num_objects":7267,"num_object_clones":0,"num_object_copies":14534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7269,"num_write_kb":29060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2930},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6946","reported_seq":7024,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.666765+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:19.666765+0000","last_peered":"2024-09-16T10:25:19.666765+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:19.666765+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6946,"log_dups_size":0,"ondisk_log_size":6946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28435398,"num_objects":6944,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6944,"num_write_kb":27770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2777},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7264","reported_seq":7354,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.812289+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:19.812289+0000","last_peered":"2024-09-16T10:25:19.812289+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:19.812289+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29706124,"num_objects":7256,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7261,"num_write_kb":29015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2861},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":461681142,"num_objects":112731,"num_object_clones":0,"num_object_copies":225462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1864,"num_objects_unfound":0,"num_objects_dirty":112731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90360,"num_write_kb":361328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2354,"num_bytes_recovered":9560547,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":932601856,"data_stored":932518757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112795,"ondisk_log_size":112795,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018913,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530972,"kb_used_data":359060,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93840868,"statfs":{"total":96636764160,"available":96093048832,"internally_reserved":0,"allocated":367677440,"data_stored":367434201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":176025264},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.26800000000000002},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":2.0270000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":1.9430000000000001},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.315}]}]},{"osd":1,"up_from":40,"seq":171798691858,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259412,"kb_used_data":141644,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94112428,"statfs":{"total":96636764160,"available":96371126272,"internally_reserved":0,"allocated":145043456,"data_stored":144727957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":120581802},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":2.9569999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":2.9900000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":3.1930000000000001},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":3.2330000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215142,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678352,"kb_used_data":408072,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93693488,"statfs":{"total":96636764160,"available":95942131712,"internally_reserved":0,"allocated":417865728,"data_stored":417626103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":276754088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":4.0460000000000003},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.57199999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.0049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":419065856,"data_stored":419062243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":145743872,"data_stored":145667999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":367792128,"data_stored":367788515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:21.472 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:21.472 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'7087", 'reported_seq': 7151, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:19.841581+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:25:19.841581+0000', 'last_peered': '2024-09-16T10:25:19.841581+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:25:19.841581+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7087, 'log_dups_size': 0, 'ondisk_log_size': 7087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29028352, 'num_objects': 7087, 'num_object_clones': 0, 'num_object_copies': 14174, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7087, 'num_write_kb': 28348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2895}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:21.473 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:21.473 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'6968", 'reported_seq': 7150, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:15.668347+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:15.668347+0000', 'last_peered': '2024-09-16T10:25:15.668347+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:15.668347+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 6968, 'log_dups_size': 0, 'ondisk_log_size': 6968, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28524544, 'num_objects': 6964, 'num_object_clones': 0, 'num_object_copies': 13928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3025, 'num_write_kb': 12100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 2629}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:21.473 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:21.473 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'6992", 'reported_seq': 7152, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:19.777229+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:25:19.777229+0000', 'last_peered': '2024-09-16T10:25:19.777229+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:25:19.777229+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 6992, 'log_dups_size': 0, 'ondisk_log_size': 6992, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28618752, 'num_objects': 6987, 'num_object_clones': 0, 'num_object_copies': 13974, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3617, 'num_write_kb': 14468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2770}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:21.473 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:21.473 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'7292", 'reported_seq': 7463, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:19.827896+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:25:19.827896+0000', 'last_peered': '2024-09-16T10:25:19.827896+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:25:19.827896+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 7292, 'log_dups_size': 0, 'ondisk_log_size': 7292, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29843456, 'num_objects': 7286, 'num_object_clones': 0, 'num_object_copies': 14572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3534, 'num_write_kb': 14136, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2883}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:21.473 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:21.473 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'6874", 'reported_seq': 11638, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:15.721472+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:15.721472+0000', 'last_peered': '2024-09-16T10:25:15.721472+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:15.721472+0000', 'last_undegraded': '2024-09-16T10:25:15.721472+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6874, 'log_dups_size': 0, 'ondisk_log_size': 6874, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28128198, 'num_objects': 6869, 'num_object_clones': 0, 'num_object_copies': 13738, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1864, 'num_objects_unfound': 0, 'num_objects_dirty': 6869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6872, 'num_write_kb': 27472, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2337, 'num_bytes_recovered': 9555968, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1864}, {'shards': '2', 'objects': 5005}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:21.473 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:21.474 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'7242", 'reported_seq': 7318, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:19.585909+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:25:19.585909+0000', 'last_peered': '2024-09-16T10:25:19.585909+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:25:19.585909+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7242, 'log_dups_size': 0, 'ondisk_log_size': 7242, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29631430, 'num_objects': 7236, 'num_object_clones': 0, 'num_object_copies': 14472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7240, 'num_write_kb': 28940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2802}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:21.474 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:21.474 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'6890", 'reported_seq': 6993, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:15.710016+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:15.710016+0000', 'last_peered': '2024-09-16T10:25:15.710016+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:15.710016+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 6890, 'log_dups_size': 0, 'ondisk_log_size': 6890, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28201926, 'num_objects': 6887, 'num_object_clones': 0, 'num_object_copies': 13774, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6888, 'num_write_kb': 27543, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 2610}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:21.474 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:21.474 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'6950", 'reported_seq': 7034, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:15.698850+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:15.698850+0000', 'last_peered': '2024-09-16T10:25:15.698850+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:15.698850+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6950, 'log_dups_size': 0, 'ondisk_log_size': 6950, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28443590, 'num_objects': 6946, 'num_object_clones': 0, 'num_object_copies': 13892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6948, 'num_write_kb': 27779, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 2754}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:21.474 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:21.474 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'6929", 'reported_seq': 7006, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:19.747622+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:25:19.747622+0000', 'last_peered': '2024-09-16T10:25:19.747622+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:25:19.747622+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6929, 'log_dups_size': 0, 'ondisk_log_size': 6929, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28373475, 'num_objects': 6928, 'num_object_clones': 0, 'num_object_copies': 13856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6928, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6928, 'num_write_kb': 27709, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2769}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:21.474 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:21.474 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'7272", 'reported_seq': 7402, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:19.761106+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:19.761106+0000', 'last_peered': '2024-09-16T10:25:19.761106+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:19.761106+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 7272, 'log_dups_size': 0, 'ondisk_log_size': 7272, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29754793, 'num_objects': 7267, 'num_object_clones': 0, 'num_object_copies': 14534, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7269, 'num_write_kb': 29060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2930}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:21.474 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:21.475 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'6946", 'reported_seq': 7024, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:19.666765+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:19.666765+0000', 'last_peered': '2024-09-16T10:25:19.666765+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:19.666765+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 6946, 'log_dups_size': 0, 'ondisk_log_size': 6946, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28435398, 'num_objects': 6944, 'num_object_clones': 0, 'num_object_copies': 13888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6944, 'num_write_kb': 27770, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2777}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:21.475 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:21.475 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'7264", 'reported_seq': 7354, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:19.812289+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:25:19.812289+0000', 'last_peered': '2024-09-16T10:25:19.812289+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:25:19.812289+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7264, 'log_dups_size': 0, 'ondisk_log_size': 7264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29706124, 'num_objects': 7256, 'num_object_clones': 0, 'num_object_copies': 14512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7261, 'num_write_kb': 29015, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2861}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:21.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:21.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 129 20 114318 114298 3.46027 1.82031 0.0983843 0.0225674 2024-09-16T10:25:22.138 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:22.138 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/225466 objects degraded (20.902%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":210,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":112733,"data_bytes":461681161,"bytes_used":1503985664,"bytes_avail":288406306816,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":225466,"degraded_ratio":0.20902486405932602,"misplaced_objects":1864,"misplaced_total":225466,"misplaced_ratio":0.0082673219021936785,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":110871,"recovering_keys_per_sec":0,"num_objects_recovered":327,"num_bytes_recovered":1331200,"num_keys_recovered":0,"write_bytes_sec":2278842,"read_op_per_sec":0,"write_op_per_sec":556},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:22.161 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:22.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:22.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 130 20 114768 114748 3.44718 1.75781 0.034555 0.0226547 2024-09-16T10:25:22.712 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:22.713 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:22.741 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-16T10:25:22.599412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465547785,"num_objects":113677,"num_object_clones":0,"num_object_copies":227354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1672,"num_objects_unfound":0,"num_objects_dirty":113677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91306,"num_write_kb":365106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":10346998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113742,"ondisk_log_size":113742,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1496020,"kb_used_data":913916,"kb_used_omap":12,"kb_used_meta":582067,"kb_avail":281619500,"statfs":{"total":289910292480,"available":288378368000,"internally_reserved":0,"allocated":935849984,"data_stored":935051621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":596036610},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,10,11,10,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24494080,"num_objects":5980,"num_object_clones":0,"num_object_copies":11960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-517,"num_objects_unfound":0,"num_objects_dirty":5980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5980,"num_write_kb":23920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":519,"num_bytes_recovered":2117632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006670"},"pg_stats":[{"pgid":"2.f","version":"42'7087","reported_seq":7151,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.841581+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:19.841581+0000","last_peered":"2024-09-16T10:25:19.841581+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:19.841581+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29028352,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":28348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7142","reported_seq":7296,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:20.683540+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:20.683540+0000","last_peered":"2024-09-16T10:25:20.683540+0000","last_clean":"2024-09-16T10:25:20.683540+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:20.683540+0000","last_undegraded":"2024-09-16T10:25:20.683540+0000","last_fullsized":"2024-09-16T10:25:20.683540+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7142,"log_dups_size":0,"ondisk_log_size":7142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":7136,"num_object_clones":0,"num_object_copies":14272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3355,"num_write_kb":13420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7133","reported_seq":7316,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:20.711955+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:20.711955+0000","last_peered":"2024-09-16T10:25:20.711955+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:20.711955+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7133,"log_dups_size":0,"ondisk_log_size":7133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29200384,"num_objects":7129,"num_object_clones":0,"num_object_copies":14258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":12760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2794}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7117","reported_seq":7171,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:20.663070+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:20.663070+0000","last_peered":"2024-09-16T10:25:20.663070+0000","last_clean":"2024-09-16T10:25:20.663070+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:20.663070+0000","last_undegraded":"2024-09-16T10:25:20.663070+0000","last_fullsized":"2024-09-16T10:25:20.663070+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29151232,"num_objects":7117,"num_object_clones":0,"num_object_copies":14234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":28468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7083","reported_seq":7237,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:19.794137+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:19.794137+0000","last_peered":"2024-09-16T10:25:19.794137+0000","last_clean":"2024-09-16T10:25:19.794137+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:19.794137+0000","last_undegraded":"2024-09-16T10:25:19.794137+0000","last_fullsized":"2024-09-16T10:25:19.794137+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7083,"log_dups_size":0,"ondisk_log_size":7083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28983296,"num_objects":7076,"num_object_clones":0,"num_object_copies":14152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":14900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6992","reported_seq":7152,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.777229+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:19.777229+0000","last_peered":"2024-09-16T10:25:19.777229+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:19.777229+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6992,"log_dups_size":0,"ondisk_log_size":6992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28618752,"num_objects":6987,"num_object_clones":0,"num_object_copies":13974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":14468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2770},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7292","reported_seq":7463,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.827896+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:19.827896+0000","last_peered":"2024-09-16T10:25:19.827896+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:19.827896+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7292,"log_dups_size":0,"ondisk_log_size":7292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29843456,"num_objects":7286,"num_object_clones":0,"num_object_copies":14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3534,"num_write_kb":14136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2883},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7054","reported_seq":7226,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:19.719169+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:19.719169+0000","last_peered":"2024-09-16T10:25:19.719169+0000","last_clean":"2024-09-16T10:25:19.719169+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:19.719169+0000","last_undegraded":"2024-09-16T10:25:19.719169+0000","last_fullsized":"2024-09-16T10:25:19.719169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28884992,"num_objects":7052,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7022","reported_seq":12170,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:20.712694+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:20.712694+0000","last_peered":"2024-09-16T10:25:20.712694+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:20.712694+0000","last_undegraded":"2024-09-16T10:25:20.712694+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28734406,"num_objects":7017,"num_object_clones":0,"num_object_copies":14034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1672,"num_objects_unfound":0,"num_objects_dirty":7017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":28064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2529,"num_bytes_recovered":10342400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1672},{"shards":"2","objects":5345}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7242","reported_seq":7318,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.585909+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:19.585909+0000","last_peered":"2024-09-16T10:25:19.585909+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:19.585909+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29631430,"num_objects":7236,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":28940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2802},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7045","reported_seq":7147,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:20.616609+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:20.616609+0000","last_peered":"2024-09-16T10:25:20.616609+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:20.616609+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28836806,"num_objects":7042,"num_object_clones":0,"num_object_copies":14084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7043,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2765}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7119","reported_seq":7202,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:20.705834+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:20.705834+0000","last_peered":"2024-09-16T10:25:20.705834+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:20.705834+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7119,"log_dups_size":0,"ondisk_log_size":7119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29135814,"num_objects":7115,"num_object_clones":0,"num_object_copies":14230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":28455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2923}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6929","reported_seq":7006,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.747622+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:19.747622+0000","last_peered":"2024-09-16T10:25:19.747622+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:19.747622+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6929,"log_dups_size":0,"ondisk_log_size":6929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28373475,"num_objects":6928,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6928,"num_write_kb":27709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2769},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7272","reported_seq":7402,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.761106+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:19.761106+0000","last_peered":"2024-09-16T10:25:19.761106+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:19.761106+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29754793,"num_objects":7267,"num_object_clones":0,"num_object_copies":14534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7269,"num_write_kb":29060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2930},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6946","reported_seq":7024,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.666765+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:19.666765+0000","last_peered":"2024-09-16T10:25:19.666765+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:19.666765+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6946,"log_dups_size":0,"ondisk_log_size":6946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28435398,"num_objects":6944,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6944,"num_write_kb":27770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2777},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7264","reported_seq":7354,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.812289+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:19.812289+0000","last_peered":"2024-09-16T10:25:19.812289+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:19.812289+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29706124,"num_objects":7256,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7261,"num_write_kb":29015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2861},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":465547766,"num_objects":113675,"num_object_clones":0,"num_object_copies":227350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1672,"num_objects_unfound":0,"num_objects_dirty":113675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91304,"num_write_kb":365104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2546,"num_bytes_recovered":10346979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":937713664,"data_stored":937630565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113739,"ondisk_log_size":113739,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018914,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558256,"kb_used_data":364200,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93813584,"statfs":{"total":96636764160,"available":96065110016,"internally_reserved":0,"allocated":372940800,"data_stored":372697561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":198700720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,10,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":1.52},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":3.911}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":4.899},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.5620000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691858,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":259412,"kb_used_data":141644,"kb_used_omap":4,"kb_used_meta":117755,"kb_avail":94112428,"statfs":{"total":96636764160,"available":96371126272,"internally_reserved":0,"allocated":145043456,"data_stored":144727957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":120581802},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":2.9569999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":2.9900000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":3.1930000000000001},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":3.2330000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215142,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678352,"kb_used_data":408072,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93693488,"statfs":{"total":96636764160,"available":95942131712,"internally_reserved":0,"allocated":417865728,"data_stored":417626103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":276754088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":4.0460000000000003},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.57199999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.0049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":419065856,"data_stored":419062243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":145743872,"data_stored":145667999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":372903936,"data_stored":372900323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:23.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 131 20 115280 115260 3.43613 2 0.0434137 0.0227278 2024-09-16T10:25:24.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 132 20 115808 115788 3.42573 2.0625 0.0772706 0.0227971 2024-09-16T10:25:25.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 133 20 116334 116314 3.41542 2.05469 0.0202209 0.0228676 2024-09-16T10:25:25.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:26.278 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:26.303 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-16T10:25:24.599918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":465547785,"num_objects":113677,"num_object_clones":0,"num_object_copies":227354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1672,"num_objects_unfound":0,"num_objects_dirty":113677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91306,"num_write_kb":365106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":10346998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113742,"ondisk_log_size":113742,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1530632,"kb_used_data":922288,"kb_used_omap":12,"kb_used_meta":608307,"kb_avail":281584888,"statfs":{"total":289910292480,"available":288342925312,"internally_reserved":0,"allocated":944422912,"data_stored":943624549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":622906370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,7,16,12,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20508672,"num_objects":5007,"num_object_clones":0,"num_object_copies":10014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-358,"num_objects_unfound":0,"num_objects_dirty":5007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5007,"num_write_kb":20028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006664"},"pg_stats":[{"pgid":"2.f","version":"42'7087","reported_seq":7151,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.841581+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:19.841581+0000","last_peered":"2024-09-16T10:25:19.841581+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:19.841581+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7087,"log_dups_size":0,"ondisk_log_size":7087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29028352,"num_objects":7087,"num_object_clones":0,"num_object_copies":14174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7087,"num_write_kb":28348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7142","reported_seq":7296,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:20.683540+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:20.683540+0000","last_peered":"2024-09-16T10:25:20.683540+0000","last_clean":"2024-09-16T10:25:20.683540+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:20.683540+0000","last_undegraded":"2024-09-16T10:25:20.683540+0000","last_fullsized":"2024-09-16T10:25:20.683540+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7142,"log_dups_size":0,"ondisk_log_size":7142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":7136,"num_object_clones":0,"num_object_copies":14272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3355,"num_write_kb":13420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7133","reported_seq":7316,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:20.711955+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:20.711955+0000","last_peered":"2024-09-16T10:25:20.711955+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:20.711955+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7133,"log_dups_size":0,"ondisk_log_size":7133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29200384,"num_objects":7129,"num_object_clones":0,"num_object_copies":14258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":12760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2794}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7117","reported_seq":7171,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:20.663070+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:20.663070+0000","last_peered":"2024-09-16T10:25:20.663070+0000","last_clean":"2024-09-16T10:25:20.663070+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:20.663070+0000","last_undegraded":"2024-09-16T10:25:20.663070+0000","last_fullsized":"2024-09-16T10:25:20.663070+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7117,"log_dups_size":0,"ondisk_log_size":7117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29151232,"num_objects":7117,"num_object_clones":0,"num_object_copies":14234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":28468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7083","reported_seq":7237,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:19.794137+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:19.794137+0000","last_peered":"2024-09-16T10:25:19.794137+0000","last_clean":"2024-09-16T10:25:19.794137+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:19.794137+0000","last_undegraded":"2024-09-16T10:25:19.794137+0000","last_fullsized":"2024-09-16T10:25:19.794137+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7083,"log_dups_size":0,"ondisk_log_size":7083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28983296,"num_objects":7076,"num_object_clones":0,"num_object_copies":14152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3725,"num_write_kb":14900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'6992","reported_seq":7152,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.777229+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:19.777229+0000","last_peered":"2024-09-16T10:25:19.777229+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:19.777229+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":6992,"log_dups_size":0,"ondisk_log_size":6992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28618752,"num_objects":6987,"num_object_clones":0,"num_object_copies":13974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3617,"num_write_kb":14468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2770},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7292","reported_seq":7463,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.827896+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:19.827896+0000","last_peered":"2024-09-16T10:25:19.827896+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:19.827896+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7292,"log_dups_size":0,"ondisk_log_size":7292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29843456,"num_objects":7286,"num_object_clones":0,"num_object_copies":14572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3534,"num_write_kb":14136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2883},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7054","reported_seq":7226,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:19.719169+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:19.719169+0000","last_peered":"2024-09-16T10:25:19.719169+0000","last_clean":"2024-09-16T10:25:19.719169+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:19.719169+0000","last_undegraded":"2024-09-16T10:25:19.719169+0000","last_fullsized":"2024-09-16T10:25:19.719169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28884992,"num_objects":7052,"num_object_clones":0,"num_object_copies":14104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7022","reported_seq":12170,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:20.712694+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:20.712694+0000","last_peered":"2024-09-16T10:25:20.712694+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:20.712694+0000","last_undegraded":"2024-09-16T10:25:20.712694+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7022,"log_dups_size":0,"ondisk_log_size":7022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28734406,"num_objects":7017,"num_object_clones":0,"num_object_copies":14034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1672,"num_objects_unfound":0,"num_objects_dirty":7017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":28064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2529,"num_bytes_recovered":10342400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1672},{"shards":"2","objects":5345}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7242","reported_seq":7318,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.585909+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:19.585909+0000","last_peered":"2024-09-16T10:25:19.585909+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:19.585909+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7242,"log_dups_size":0,"ondisk_log_size":7242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29631430,"num_objects":7236,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":28940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2802},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7045","reported_seq":7147,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:20.616609+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:20.616609+0000","last_peered":"2024-09-16T10:25:20.616609+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:20.616609+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7045,"log_dups_size":0,"ondisk_log_size":7045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28836806,"num_objects":7042,"num_object_clones":0,"num_object_copies":14084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7043,"num_write_kb":28163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2765}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7119","reported_seq":7202,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:20.705834+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:20.705834+0000","last_peered":"2024-09-16T10:25:20.705834+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:20.705834+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7119,"log_dups_size":0,"ondisk_log_size":7119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29135814,"num_objects":7115,"num_object_clones":0,"num_object_copies":14230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":28455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":2923}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'6929","reported_seq":7006,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.747622+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:19.747622+0000","last_peered":"2024-09-16T10:25:19.747622+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:19.747622+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6929,"log_dups_size":0,"ondisk_log_size":6929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28373475,"num_objects":6928,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6928,"num_write_kb":27709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2769},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7272","reported_seq":7402,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.761106+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:19.761106+0000","last_peered":"2024-09-16T10:25:19.761106+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:19.761106+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7272,"log_dups_size":0,"ondisk_log_size":7272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29754793,"num_objects":7267,"num_object_clones":0,"num_object_copies":14534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7269,"num_write_kb":29060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2930},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'6946","reported_seq":7024,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.666765+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:19.666765+0000","last_peered":"2024-09-16T10:25:19.666765+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:19.666765+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":6946,"log_dups_size":0,"ondisk_log_size":6946,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28435398,"num_objects":6944,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6944,"num_write_kb":27770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2777},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7264","reported_seq":7354,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:19.812289+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:19.812289+0000","last_peered":"2024-09-16T10:25:19.812289+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:19.812289+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7264,"log_dups_size":0,"ondisk_log_size":7264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29706124,"num_objects":7256,"num_object_clones":0,"num_object_copies":14512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7261,"num_write_kb":29015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2861},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":465547766,"num_objects":113675,"num_object_clones":0,"num_object_copies":227350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1672,"num_objects_unfound":0,"num_objects_dirty":113675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91304,"num_write_kb":365104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2546,"num_bytes_recovered":10346979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":946167808,"data_stored":946084709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113739,"ondisk_log_size":113739,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018914,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558256,"kb_used_data":364200,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93813584,"statfs":{"total":96636764160,"available":96065110016,"internally_reserved":0,"allocated":372940800,"data_stored":372697561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":198700720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,10,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":1.52},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":3.911}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":4.899},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.5620000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691859,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294024,"kb_used_data":150016,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94077816,"statfs":{"total":96636764160,"available":96335683584,"internally_reserved":0,"allocated":153616384,"data_stored":153300885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":147451562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":12.819000000000001},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":12.851000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":5.4080000000000004},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":4.4039999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215142,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":678352,"kb_used_data":408072,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93693488,"statfs":{"total":96636764160,"available":95942131712,"internally_reserved":0,"allocated":417865728,"data_stored":417626103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":276754088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":4.0460000000000003},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.57199999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.0049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":419065856,"data_stored":419062243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154198016,"data_stored":154122143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":372903936,"data_stored":372900323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:26.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:26.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 134 20 116794 116774 3.40334 1.79688 0.0265176 0.0229462 2024-09-16T10:25:26.857 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:26.858 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:26.888 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-16T10:25:26.600397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":475636233,"num_objects":116140,"num_object_clones":0,"num_object_copies":232280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":116140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93769,"num_write_kb":374958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2736,"num_bytes_recovered":11121142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116205,"ondisk_log_size":116205,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586988,"kb_used_data":934356,"kb_used_omap":12,"kb_used_meta":652595,"kb_avail":281528532,"statfs":{"total":289910292480,"available":288285216768,"internally_reserved":0,"allocated":956780544,"data_stored":955982181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":668257282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,2,11,15,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30597120,"num_objects":7470,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-547,"num_objects_unfound":0,"num_objects_dirty":7470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":29880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":547,"num_bytes_recovered":2240512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006644"},"pg_stats":[{"pgid":"2.f","version":"42'7237","reported_seq":7303,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.861467+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:24.861467+0000","last_peered":"2024-09-16T10:25:24.861467+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:24.861467+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29642752,"num_objects":7237,"num_object_clones":0,"num_object_copies":14474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7237,"num_write_kb":28948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3045},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7291","reported_seq":7445,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:25.670030+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:25.670030+0000","last_peered":"2024-09-16T10:25:25.670030+0000","last_clean":"2024-09-16T10:25:25.670030+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:25.670030+0000","last_undegraded":"2024-09-16T10:25:25.670030+0000","last_fullsized":"2024-09-16T10:25:25.670030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7291,"log_dups_size":0,"ondisk_log_size":7291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29839360,"num_objects":7285,"num_object_clones":0,"num_object_copies":14570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":14016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7276","reported_seq":7459,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.704098+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:25.704098+0000","last_peered":"2024-09-16T10:25:25.704098+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:25.704098+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7276,"log_dups_size":0,"ondisk_log_size":7276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29786112,"num_objects":7272,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3333,"num_write_kb":13332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7266","reported_seq":7320,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:25.698889+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:25.698889+0000","last_peered":"2024-09-16T10:25:25.698889+0000","last_clean":"2024-09-16T10:25:25.698889+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:25.698889+0000","last_undegraded":"2024-09-16T10:25:25.698889+0000","last_fullsized":"2024-09-16T10:25:25.698889+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7266,"log_dups_size":0,"ondisk_log_size":7266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761536,"num_objects":7266,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":29064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7235","reported_seq":7389,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:24.847036+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:24.847036+0000","last_peered":"2024-09-16T10:25:24.847036+0000","last_clean":"2024-09-16T10:25:24.847036+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:24.847036+0000","last_undegraded":"2024-09-16T10:25:24.847036+0000","last_fullsized":"2024-09-16T10:25:24.847036+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7235,"log_dups_size":0,"ondisk_log_size":7235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29605888,"num_objects":7228,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3877,"num_write_kb":15508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7144","reported_seq":7305,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.785307+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:24.785307+0000","last_peered":"2024-09-16T10:25:24.785307+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:24.785307+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7144,"log_dups_size":0,"ondisk_log_size":7144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29241344,"num_objects":7139,"num_object_clones":0,"num_object_copies":14278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":15076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2922},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7481","reported_seq":7652,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.840943+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:24.840943+0000","last_peered":"2024-09-16T10:25:24.840943+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:24.840943+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7481,"log_dups_size":0,"ondisk_log_size":7481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30617600,"num_objects":7475,"num_object_clones":0,"num_object_copies":14950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3723,"num_write_kb":14892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3072},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7214","reported_seq":7386,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:24.864332+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:24.864332+0000","last_peered":"2024-09-16T10:25:24.864332+0000","last_clean":"2024-09-16T10:25:24.864332+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:24.864332+0000","last_undegraded":"2024-09-16T10:25:24.864332+0000","last_fullsized":"2024-09-16T10:25:24.864332+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7214,"log_dups_size":0,"ondisk_log_size":7214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29540352,"num_objects":7212,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":12068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7164","reported_seq":12690,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:25.723491+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:25.723491+0000","last_peered":"2024-09-16T10:25:25.723491+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:25.723491+0000","last_undegraded":"2024-09-16T10:25:25.723491+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7164,"log_dups_size":0,"ondisk_log_size":7164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29316038,"num_objects":7159,"num_object_clones":0,"num_object_copies":14318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":7159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7162,"num_write_kb":28632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2718,"num_bytes_recovered":11116544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1483},{"shards":"2","objects":5676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7410","reported_seq":7483,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.774284+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:24.774284+0000","last_peered":"2024-09-16T10:25:24.774284+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:24.774284+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7410,"log_dups_size":0,"ondisk_log_size":7410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30319558,"num_objects":7404,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7408,"num_write_kb":29612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2970},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7199","reported_seq":7301,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.728552+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:25.728552+0000","last_peered":"2024-09-16T10:25:25.728552+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:25.728552+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7199,"log_dups_size":0,"ondisk_log_size":7199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29467590,"num_objects":7196,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7197,"num_write_kb":28779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7267","reported_seq":7351,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.632079+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:25.632079+0000","last_peered":"2024-09-16T10:25:25.632079+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:25.632079+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7267,"log_dups_size":0,"ondisk_log_size":7267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29742022,"num_objects":7263,"num_object_clones":0,"num_object_copies":14526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":29047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3071}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7080","reported_seq":7157,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.821973+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:24.821973+0000","last_peered":"2024-09-16T10:25:24.821973+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:24.821973+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7080,"log_dups_size":0,"ondisk_log_size":7080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28991971,"num_objects":7079,"num_object_clones":0,"num_object_copies":14158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7079,"num_write_kb":28313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2920},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7411","reported_seq":7541,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.813171+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:24.813171+0000","last_peered":"2024-09-16T10:25:24.813171+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:24.813171+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7411,"log_dups_size":0,"ondisk_log_size":7411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30324137,"num_objects":7406,"num_object_clones":0,"num_object_copies":14812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7408,"num_write_kb":29616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3069},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7106","reported_seq":7182,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.823884+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:24.823884+0000","last_peered":"2024-09-16T10:25:24.823884+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:24.823884+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7106,"log_dups_size":0,"ondisk_log_size":7106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29090758,"num_objects":7104,"num_object_clones":0,"num_object_copies":14208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7104,"num_write_kb":28410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2937},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7421","reported_seq":7511,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.869004+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:24.869004+0000","last_peered":"2024-09-16T10:25:24.869004+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:24.869004+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7421,"log_dups_size":0,"ondisk_log_size":7421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30349196,"num_objects":7413,"num_object_clones":0,"num_object_copies":14826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7418,"num_write_kb":29643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3018},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":475636214,"num_objects":116138,"num_object_clones":0,"num_object_copies":232276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":116138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93767,"num_write_kb":374956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2735,"num_bytes_recovered":11121123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":958869504,"data_stored":958786405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116202,"ondisk_log_size":116202,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018915,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603112,"kb_used_data":368864,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93768728,"statfs":{"total":96636764160,"available":96019177472,"internally_reserved":0,"allocated":377716736,"data_stored":377473497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":239857328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.20599999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.246},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.6870000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691859,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294024,"kb_used_data":150016,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94077816,"statfs":{"total":96636764160,"available":96335683584,"internally_reserved":0,"allocated":153616384,"data_stored":153300885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":147451562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":12.819000000000001},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":12.851000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":5.4080000000000004},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":4.4039999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215143,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689852,"kb_used_data":415476,"kb_used_omap":4,"kb_used_meta":274363,"kb_avail":93681988,"statfs":{"total":96636764160,"available":95930355712,"internally_reserved":0,"allocated":425447424,"data_stored":425207799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":280948392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,5,11,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.32200000000000001},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.8440000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":3.2930000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":426807296,"data_stored":426803683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154198016,"data_stored":154122143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377864192,"data_stored":377860579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:26.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:26.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'7237", 'reported_seq': 7303, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:24.861467+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:25:24.861467+0000', 'last_peered': '2024-09-16T10:25:24.861467+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:25:24.861467+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7237, 'log_dups_size': 0, 'ondisk_log_size': 7237, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29642752, 'num_objects': 7237, 'num_object_clones': 0, 'num_object_copies': 14474, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7237, 'num_write_kb': 28948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3045}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:26.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:26.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'7276", 'reported_seq': 7459, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:25.704098+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:25.704098+0000', 'last_peered': '2024-09-16T10:25:25.704098+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:25.704098+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 7276, 'log_dups_size': 0, 'ondisk_log_size': 7276, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29786112, 'num_objects': 7272, 'num_object_clones': 0, 'num_object_copies': 14544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3333, 'num_write_kb': 13332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 2937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:26.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:26.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'7144", 'reported_seq': 7305, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:24.785307+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:25:24.785307+0000', 'last_peered': '2024-09-16T10:25:24.785307+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:25:24.785307+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 7144, 'log_dups_size': 0, 'ondisk_log_size': 7144, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29241344, 'num_objects': 7139, 'num_object_clones': 0, 'num_object_copies': 14278, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3769, 'num_write_kb': 15076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2922}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:26.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:26.890 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'7481", 'reported_seq': 7652, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:24.840943+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:25:24.840943+0000', 'last_peered': '2024-09-16T10:25:24.840943+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:25:24.840943+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 7481, 'log_dups_size': 0, 'ondisk_log_size': 7481, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30617600, 'num_objects': 7475, 'num_object_clones': 0, 'num_object_copies': 14950, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3723, 'num_write_kb': 14892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3072}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:26.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:26.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'7164", 'reported_seq': 12690, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:25.723491+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:25.723491+0000', 'last_peered': '2024-09-16T10:25:25.723491+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:25.723491+0000', 'last_undegraded': '2024-09-16T10:25:25.723491+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7164, 'log_dups_size': 0, 'ondisk_log_size': 7164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29316038, 'num_objects': 7159, 'num_object_clones': 0, 'num_object_copies': 14318, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1483, 'num_objects_unfound': 0, 'num_objects_dirty': 7159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7162, 'num_write_kb': 28632, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2718, 'num_bytes_recovered': 11116544, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1483}, {'shards': '2', 'objects': 5676}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:26.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:26.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'7410", 'reported_seq': 7483, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:24.774284+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:25:24.774284+0000', 'last_peered': '2024-09-16T10:25:24.774284+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:25:24.774284+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7410, 'log_dups_size': 0, 'ondisk_log_size': 7410, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30319558, 'num_objects': 7404, 'num_object_clones': 0, 'num_object_copies': 14808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7408, 'num_write_kb': 29612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2970}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:26.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:26.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'7199", 'reported_seq': 7301, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:25.728552+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:25.728552+0000', 'last_peered': '2024-09-16T10:25:25.728552+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:25.728552+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 7199, 'log_dups_size': 0, 'ondisk_log_size': 7199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29467590, 'num_objects': 7196, 'num_object_clones': 0, 'num_object_copies': 14392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7197, 'num_write_kb': 28779, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 2919}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:26.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:26.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'7267", 'reported_seq': 7351, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:25.632079+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:25.632079+0000', 'last_peered': '2024-09-16T10:25:25.632079+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:25.632079+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7267, 'log_dups_size': 0, 'ondisk_log_size': 7267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29742022, 'num_objects': 7263, 'num_object_clones': 0, 'num_object_copies': 14526, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7265, 'num_write_kb': 29047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 3071}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:26.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:26.891 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'7080", 'reported_seq': 7157, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:24.821973+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:25:24.821973+0000', 'last_peered': '2024-09-16T10:25:24.821973+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:25:24.821973+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7080, 'log_dups_size': 0, 'ondisk_log_size': 7080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28991971, 'num_objects': 7079, 'num_object_clones': 0, 'num_object_copies': 14158, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7079, 'num_write_kb': 28313, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2920}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:26.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:26.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'7411", 'reported_seq': 7541, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:24.813171+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:24.813171+0000', 'last_peered': '2024-09-16T10:25:24.813171+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:24.813171+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 7411, 'log_dups_size': 0, 'ondisk_log_size': 7411, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30324137, 'num_objects': 7406, 'num_object_clones': 0, 'num_object_copies': 14812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7408, 'num_write_kb': 29616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3069}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:26.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:26.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'7106", 'reported_seq': 7182, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:24.823884+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:24.823884+0000', 'last_peered': '2024-09-16T10:25:24.823884+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:24.823884+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7106, 'log_dups_size': 0, 'ondisk_log_size': 7106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29090758, 'num_objects': 7104, 'num_object_clones': 0, 'num_object_copies': 14208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7104, 'num_write_kb': 28410, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2937}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:26.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:26.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'7421", 'reported_seq': 7511, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:24.869004+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:25:24.869004+0000', 'last_peered': '2024-09-16T10:25:24.869004+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:25:24.869004+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7421, 'log_dups_size': 0, 'ondisk_log_size': 7421, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30349196, 'num_objects': 7413, 'num_object_clones': 0, 'num_object_copies': 14826, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7418, 'num_write_kb': 29643, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3018}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:26.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:27.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 135 20 117249 117229 3.39129 1.77734 0.0917181 0.0230285 2024-09-16T10:25:27.583 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:27.583 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/232280 objects degraded (20.289%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":216,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":116140,"data_bytes":475636233,"bytes_used":1625075712,"bytes_avail":288285216768,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":232280,"degraded_ratio":0.20289306009987945,"misplaced_objects":1483,"misplaced_total":232280,"misplaced_ratio":0.0063845359049423114,"recovering_objects_per_sec":45,"recovering_bytes_per_sec":186606,"recovering_keys_per_sec":0,"num_objects_recovered":547,"num_bytes_recovered":2240512,"num_keys_recovered":0,"write_bytes_sec":2548348,"read_op_per_sec":0,"write_op_per_sec":622},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:27.609 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:27.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:28.144 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:28.147 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:28.171 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-16T10:25:26.600397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":475636233,"num_objects":116140,"num_object_clones":0,"num_object_copies":232280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":116140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93769,"num_write_kb":374958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2736,"num_bytes_recovered":11121142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116205,"ondisk_log_size":116205,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1586988,"kb_used_data":934356,"kb_used_omap":12,"kb_used_meta":652595,"kb_avail":281528532,"statfs":{"total":289910292480,"available":288285216768,"internally_reserved":0,"allocated":956780544,"data_stored":955982181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":668257282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,2,11,15,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30597120,"num_objects":7470,"num_object_clones":0,"num_object_copies":14940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-547,"num_objects_unfound":0,"num_objects_dirty":7470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7470,"num_write_kb":29880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":547,"num_bytes_recovered":2240512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006644"},"pg_stats":[{"pgid":"2.f","version":"42'7237","reported_seq":7303,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.861467+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:24.861467+0000","last_peered":"2024-09-16T10:25:24.861467+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:24.861467+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29642752,"num_objects":7237,"num_object_clones":0,"num_object_copies":14474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7237,"num_write_kb":28948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3045},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7291","reported_seq":7445,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:25.670030+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:25.670030+0000","last_peered":"2024-09-16T10:25:25.670030+0000","last_clean":"2024-09-16T10:25:25.670030+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:25.670030+0000","last_undegraded":"2024-09-16T10:25:25.670030+0000","last_fullsized":"2024-09-16T10:25:25.670030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7291,"log_dups_size":0,"ondisk_log_size":7291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29839360,"num_objects":7285,"num_object_clones":0,"num_object_copies":14570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":14016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7276","reported_seq":7459,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.704098+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:25.704098+0000","last_peered":"2024-09-16T10:25:25.704098+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:25.704098+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7276,"log_dups_size":0,"ondisk_log_size":7276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29786112,"num_objects":7272,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3333,"num_write_kb":13332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7266","reported_seq":7320,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:25.698889+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:25.698889+0000","last_peered":"2024-09-16T10:25:25.698889+0000","last_clean":"2024-09-16T10:25:25.698889+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:25.698889+0000","last_undegraded":"2024-09-16T10:25:25.698889+0000","last_fullsized":"2024-09-16T10:25:25.698889+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7266,"log_dups_size":0,"ondisk_log_size":7266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761536,"num_objects":7266,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":29064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7235","reported_seq":7389,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:24.847036+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:24.847036+0000","last_peered":"2024-09-16T10:25:24.847036+0000","last_clean":"2024-09-16T10:25:24.847036+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:24.847036+0000","last_undegraded":"2024-09-16T10:25:24.847036+0000","last_fullsized":"2024-09-16T10:25:24.847036+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7235,"log_dups_size":0,"ondisk_log_size":7235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29605888,"num_objects":7228,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3877,"num_write_kb":15508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7144","reported_seq":7305,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.785307+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:24.785307+0000","last_peered":"2024-09-16T10:25:24.785307+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:24.785307+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7144,"log_dups_size":0,"ondisk_log_size":7144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29241344,"num_objects":7139,"num_object_clones":0,"num_object_copies":14278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":15076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2922},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7481","reported_seq":7652,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.840943+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:24.840943+0000","last_peered":"2024-09-16T10:25:24.840943+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:24.840943+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7481,"log_dups_size":0,"ondisk_log_size":7481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30617600,"num_objects":7475,"num_object_clones":0,"num_object_copies":14950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3723,"num_write_kb":14892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3072},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7214","reported_seq":7386,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:24.864332+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:24.864332+0000","last_peered":"2024-09-16T10:25:24.864332+0000","last_clean":"2024-09-16T10:25:24.864332+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:24.864332+0000","last_undegraded":"2024-09-16T10:25:24.864332+0000","last_fullsized":"2024-09-16T10:25:24.864332+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7214,"log_dups_size":0,"ondisk_log_size":7214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29540352,"num_objects":7212,"num_object_clones":0,"num_object_copies":14424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":12068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7164","reported_seq":12690,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:25.723491+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:25.723491+0000","last_peered":"2024-09-16T10:25:25.723491+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:25.723491+0000","last_undegraded":"2024-09-16T10:25:25.723491+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7164,"log_dups_size":0,"ondisk_log_size":7164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29316038,"num_objects":7159,"num_object_clones":0,"num_object_copies":14318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":7159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7162,"num_write_kb":28632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2718,"num_bytes_recovered":11116544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1483},{"shards":"2","objects":5676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7410","reported_seq":7483,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.774284+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:24.774284+0000","last_peered":"2024-09-16T10:25:24.774284+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:24.774284+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7410,"log_dups_size":0,"ondisk_log_size":7410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30319558,"num_objects":7404,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7408,"num_write_kb":29612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2970},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7199","reported_seq":7301,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.728552+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:25.728552+0000","last_peered":"2024-09-16T10:25:25.728552+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:25.728552+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7199,"log_dups_size":0,"ondisk_log_size":7199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29467590,"num_objects":7196,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7197,"num_write_kb":28779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7267","reported_seq":7351,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.632079+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:25.632079+0000","last_peered":"2024-09-16T10:25:25.632079+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:25.632079+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7267,"log_dups_size":0,"ondisk_log_size":7267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29742022,"num_objects":7263,"num_object_clones":0,"num_object_copies":14526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":29047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3071}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7080","reported_seq":7157,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.821973+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:24.821973+0000","last_peered":"2024-09-16T10:25:24.821973+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:24.821973+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7080,"log_dups_size":0,"ondisk_log_size":7080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28991971,"num_objects":7079,"num_object_clones":0,"num_object_copies":14158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7079,"num_write_kb":28313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2920},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7411","reported_seq":7541,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.813171+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:24.813171+0000","last_peered":"2024-09-16T10:25:24.813171+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:24.813171+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7411,"log_dups_size":0,"ondisk_log_size":7411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30324137,"num_objects":7406,"num_object_clones":0,"num_object_copies":14812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7408,"num_write_kb":29616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3069},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7106","reported_seq":7182,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.823884+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:24.823884+0000","last_peered":"2024-09-16T10:25:24.823884+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:24.823884+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7106,"log_dups_size":0,"ondisk_log_size":7106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29090758,"num_objects":7104,"num_object_clones":0,"num_object_copies":14208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7104,"num_write_kb":28410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2937},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7421","reported_seq":7511,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:24.869004+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:24.869004+0000","last_peered":"2024-09-16T10:25:24.869004+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:24.869004+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7421,"log_dups_size":0,"ondisk_log_size":7421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30349196,"num_objects":7413,"num_object_clones":0,"num_object_copies":14826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7418,"num_write_kb":29643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3018},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":475636214,"num_objects":116138,"num_object_clones":0,"num_object_copies":232276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":116138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93767,"num_write_kb":374956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2735,"num_bytes_recovered":11121123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":958869504,"data_stored":958786405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116202,"ondisk_log_size":116202,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018915,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603112,"kb_used_data":368864,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93768728,"statfs":{"total":96636764160,"available":96019177472,"internally_reserved":0,"allocated":377716736,"data_stored":377473497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":239857328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.20599999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.246},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.6870000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691859,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":294024,"kb_used_data":150016,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94077816,"statfs":{"total":96636764160,"available":96335683584,"internally_reserved":0,"allocated":153616384,"data_stored":153300885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":147451562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":12.819000000000001},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":12.851000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":5.4080000000000004},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":4.4039999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215143,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689852,"kb_used_data":415476,"kb_used_omap":4,"kb_used_meta":274363,"kb_avail":93681988,"statfs":{"total":96636764160,"available":95930355712,"internally_reserved":0,"allocated":425447424,"data_stored":425207799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":280948392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,5,11,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.32200000000000001},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.8440000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":3.2930000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":426807296,"data_stored":426803683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154198016,"data_stored":154122143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377864192,"data_stored":377860579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:28.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 136 20 117729 117709 3.38015 1.875 0.0162058 0.0231046 2024-09-16T10:25:29.544 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 137 20 118242 118222 3.3701 2.00391 0.0396676 0.0231751 2024-09-16T10:25:30.545 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 138 20 118760 118740 3.36034 2.02344 0.0175536 0.0232404 2024-09-16T10:25:31.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:31.545 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 139 20 119261 119241 3.35025 1.95703 0.0642988 0.023313 2024-09-16T10:25:31.757 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:31.760 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:31.779 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-16T10:25:30.601574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481858057,"num_objects":117659,"num_object_clones":0,"num_object_copies":235318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":117659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95288,"num_write_kb":381034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2736,"num_bytes_recovered":11121142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117724,"ondisk_log_size":117724,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1599848,"kb_used_data":951308,"kb_used_omap":12,"kb_used_meta":648499,"kb_avail":281515672,"statfs":{"total":289910292480,"available":288272048128,"internally_reserved":0,"allocated":974139392,"data_stored":973341029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":664062978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,3,4,6,6,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26415104,"num_objects":6449,"num_object_clones":0,"num_object_copies":12898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-381,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":25796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":381,"num_bytes_recovered":1560576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003151"},"pg_stats":[{"pgid":"2.f","version":"42'7405","reported_seq":7471,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.854551+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:29.854551+0000","last_peered":"2024-09-16T10:25:29.854551+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:29.854551+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7405,"log_dups_size":0,"ondisk_log_size":7405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30330880,"num_objects":7405,"num_object_clones":0,"num_object_copies":14810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7405,"num_write_kb":29620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3213},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7291","reported_seq":7445,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:25.670030+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:25.670030+0000","last_peered":"2024-09-16T10:25:25.670030+0000","last_clean":"2024-09-16T10:25:25.670030+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:25.670030+0000","last_undegraded":"2024-09-16T10:25:25.670030+0000","last_fullsized":"2024-09-16T10:25:25.670030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7291,"log_dups_size":0,"ondisk_log_size":7291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29839360,"num_objects":7285,"num_object_clones":0,"num_object_copies":14570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":14016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7276","reported_seq":7459,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.704098+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:25.704098+0000","last_peered":"2024-09-16T10:25:25.704098+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:25.704098+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7276,"log_dups_size":0,"ondisk_log_size":7276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29786112,"num_objects":7272,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3333,"num_write_kb":13332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7266","reported_seq":7320,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:25.698889+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:25.698889+0000","last_peered":"2024-09-16T10:25:25.698889+0000","last_clean":"2024-09-16T10:25:25.698889+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:25.698889+0000","last_undegraded":"2024-09-16T10:25:25.698889+0000","last_fullsized":"2024-09-16T10:25:25.698889+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7266,"log_dups_size":0,"ondisk_log_size":7266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761536,"num_objects":7266,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":29064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7392","reported_seq":7546,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:29.796589+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:29.796589+0000","last_peered":"2024-09-16T10:25:29.796589+0000","last_clean":"2024-09-16T10:25:29.796589+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:29.796589+0000","last_undegraded":"2024-09-16T10:25:29.796589+0000","last_fullsized":"2024-09-16T10:25:29.796589+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7392,"log_dups_size":0,"ondisk_log_size":7392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30248960,"num_objects":7385,"num_object_clones":0,"num_object_copies":14770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":16136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7300","reported_seq":7461,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.811119+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:29.811119+0000","last_peered":"2024-09-16T10:25:29.811119+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:29.811119+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7300,"log_dups_size":0,"ondisk_log_size":7300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29880320,"num_objects":7295,"num_object_clones":0,"num_object_copies":14590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":15700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3078},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7633","reported_seq":7804,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.819337+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:29.819337+0000","last_peered":"2024-09-16T10:25:29.819337+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:29.819337+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7633,"log_dups_size":0,"ondisk_log_size":7633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31240192,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3875,"num_write_kb":15500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3224},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7374","reported_seq":7546,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:29.851497+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:29.851497+0000","last_peered":"2024-09-16T10:25:29.851497+0000","last_clean":"2024-09-16T10:25:29.851497+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:29.851497+0000","last_undegraded":"2024-09-16T10:25:29.851497+0000","last_fullsized":"2024-09-16T10:25:29.851497+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7374,"log_dups_size":0,"ondisk_log_size":7374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30195712,"num_objects":7372,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":12708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7164","reported_seq":12690,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:25.723491+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:25.723491+0000","last_peered":"2024-09-16T10:25:25.723491+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:25.723491+0000","last_undegraded":"2024-09-16T10:25:25.723491+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7164,"log_dups_size":0,"ondisk_log_size":7164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29316038,"num_objects":7159,"num_object_clones":0,"num_object_copies":14318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":7159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7162,"num_write_kb":28632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2718,"num_bytes_recovered":11116544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1483},{"shards":"2","objects":5676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7578","reported_seq":7648,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.812547+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:29.812547+0000","last_peered":"2024-09-16T10:25:29.812547+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:29.812547+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7578,"log_dups_size":0,"ondisk_log_size":7578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31007686,"num_objects":7572,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7576,"num_write_kb":30284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3138},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7199","reported_seq":7301,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.728552+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:25.728552+0000","last_peered":"2024-09-16T10:25:25.728552+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:25.728552+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7199,"log_dups_size":0,"ondisk_log_size":7199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29467590,"num_objects":7196,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7197,"num_write_kb":28779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7267","reported_seq":7351,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.632079+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:25.632079+0000","last_peered":"2024-09-16T10:25:25.632079+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:25.632079+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7267,"log_dups_size":0,"ondisk_log_size":7267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29742022,"num_objects":7263,"num_object_clones":0,"num_object_copies":14526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":29047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3071}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7208","reported_seq":7285,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.681312+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:29.681312+0000","last_peered":"2024-09-16T10:25:29.681312+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:29.681312+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7208,"log_dups_size":0,"ondisk_log_size":7208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29516259,"num_objects":7207,"num_object_clones":0,"num_object_copies":14414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":28825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3048},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7554","reported_seq":7684,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.894964+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:29.894964+0000","last_peered":"2024-09-16T10:25:29.894964+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:29.894964+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30909865,"num_objects":7549,"num_object_clones":0,"num_object_copies":15098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":30188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3212},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7253","reported_seq":7331,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.861016+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:29.861016+0000","last_peered":"2024-09-16T10:25:29.861016+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:29.861016+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7253,"log_dups_size":0,"ondisk_log_size":7253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29692870,"num_objects":7251,"num_object_clones":0,"num_object_copies":14502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7251,"num_write_kb":28998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3084},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7561","reported_seq":7651,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.806972+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:29.806972+0000","last_peered":"2024-09-16T10:25:29.806972+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:29.806972+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7561,"log_dups_size":0,"ondisk_log_size":7561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30922636,"num_objects":7553,"num_object_clones":0,"num_object_copies":15106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":30203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3158},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":481858038,"num_objects":117657,"num_object_clones":0,"num_object_copies":235314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":117657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95286,"num_write_kb":381032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2735,"num_bytes_recovered":11121123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974688256,"data_stored":974605157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117721,"ondisk_log_size":117721,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018915,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603112,"kb_used_data":368864,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93768728,"statfs":{"total":96636764160,"available":96019177472,"internally_reserved":0,"allocated":377716736,"data_stored":377473497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":239857328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.20599999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.246},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.6870000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691860,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306208,"kb_used_data":158100,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":94065632,"statfs":{"total":96636764160,"available":96323207168,"internally_reserved":0,"allocated":161894400,"data_stored":161578901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":151645866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.69899999999999995},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":2.1890000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.98999999999999999},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":2.0070000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215144,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690528,"kb_used_data":424344,"kb_used_omap":4,"kb_used_meta":266171,"kb_avail":93681312,"statfs":{"total":96636764160,"available":95929663488,"internally_reserved":0,"allocated":434528256,"data_stored":434288631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":272559784},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,2,1,3,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":2.6419999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.339}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":2.0350000000000001},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.6789999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":434278400,"data_stored":434274787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":162545664,"data_stored":162469791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377864192,"data_stored":377860579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:31.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:32.350 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:32.352 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:32.372 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2024-09-16T10:25:30.601574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481858057,"num_objects":117659,"num_object_clones":0,"num_object_copies":235318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":117659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95288,"num_write_kb":381034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2736,"num_bytes_recovered":11121142,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117724,"ondisk_log_size":117724,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1599848,"kb_used_data":951308,"kb_used_omap":12,"kb_used_meta":648499,"kb_avail":281515672,"statfs":{"total":289910292480,"available":288272048128,"internally_reserved":0,"allocated":974139392,"data_stored":973341029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":664062978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,3,4,6,6,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26415104,"num_objects":6449,"num_object_clones":0,"num_object_copies":12898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-381,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":25796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":381,"num_bytes_recovered":1560576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003151"},"pg_stats":[{"pgid":"2.f","version":"42'7405","reported_seq":7471,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.854551+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:29.854551+0000","last_peered":"2024-09-16T10:25:29.854551+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:29.854551+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7405,"log_dups_size":0,"ondisk_log_size":7405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30330880,"num_objects":7405,"num_object_clones":0,"num_object_copies":14810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7405,"num_write_kb":29620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3213},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7291","reported_seq":7445,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:25.670030+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:25.670030+0000","last_peered":"2024-09-16T10:25:25.670030+0000","last_clean":"2024-09-16T10:25:25.670030+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:25.670030+0000","last_undegraded":"2024-09-16T10:25:25.670030+0000","last_fullsized":"2024-09-16T10:25:25.670030+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7291,"log_dups_size":0,"ondisk_log_size":7291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29839360,"num_objects":7285,"num_object_clones":0,"num_object_copies":14570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3504,"num_write_kb":14016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7276","reported_seq":7459,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.704098+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:25.704098+0000","last_peered":"2024-09-16T10:25:25.704098+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:25.704098+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7276,"log_dups_size":0,"ondisk_log_size":7276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29786112,"num_objects":7272,"num_object_clones":0,"num_object_copies":14544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3333,"num_write_kb":13332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7266","reported_seq":7320,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:25.698889+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:25.698889+0000","last_peered":"2024-09-16T10:25:25.698889+0000","last_clean":"2024-09-16T10:25:25.698889+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:25.698889+0000","last_undegraded":"2024-09-16T10:25:25.698889+0000","last_fullsized":"2024-09-16T10:25:25.698889+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7266,"log_dups_size":0,"ondisk_log_size":7266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29761536,"num_objects":7266,"num_object_clones":0,"num_object_copies":14532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":29064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7392","reported_seq":7546,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:29.796589+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:29.796589+0000","last_peered":"2024-09-16T10:25:29.796589+0000","last_clean":"2024-09-16T10:25:29.796589+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:29.796589+0000","last_undegraded":"2024-09-16T10:25:29.796589+0000","last_fullsized":"2024-09-16T10:25:29.796589+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7392,"log_dups_size":0,"ondisk_log_size":7392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30248960,"num_objects":7385,"num_object_clones":0,"num_object_copies":14770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":16136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7300","reported_seq":7461,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.811119+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:29.811119+0000","last_peered":"2024-09-16T10:25:29.811119+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:29.811119+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7300,"log_dups_size":0,"ondisk_log_size":7300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29880320,"num_objects":7295,"num_object_clones":0,"num_object_copies":14590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":15700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3078},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7633","reported_seq":7804,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.819337+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:29.819337+0000","last_peered":"2024-09-16T10:25:29.819337+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:29.819337+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7633,"log_dups_size":0,"ondisk_log_size":7633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31240192,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3875,"num_write_kb":15500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3224},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7374","reported_seq":7546,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:29.851497+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:29.851497+0000","last_peered":"2024-09-16T10:25:29.851497+0000","last_clean":"2024-09-16T10:25:29.851497+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:29.851497+0000","last_undegraded":"2024-09-16T10:25:29.851497+0000","last_fullsized":"2024-09-16T10:25:29.851497+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7374,"log_dups_size":0,"ondisk_log_size":7374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30195712,"num_objects":7372,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":12708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7164","reported_seq":12690,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:25.723491+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:25.723491+0000","last_peered":"2024-09-16T10:25:25.723491+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:25.723491+0000","last_undegraded":"2024-09-16T10:25:25.723491+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7164,"log_dups_size":0,"ondisk_log_size":7164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29316038,"num_objects":7159,"num_object_clones":0,"num_object_copies":14318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":7159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7162,"num_write_kb":28632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2718,"num_bytes_recovered":11116544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1483},{"shards":"2","objects":5676}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7578","reported_seq":7648,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.812547+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:29.812547+0000","last_peered":"2024-09-16T10:25:29.812547+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:29.812547+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7578,"log_dups_size":0,"ondisk_log_size":7578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31007686,"num_objects":7572,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7576,"num_write_kb":30284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3138},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7199","reported_seq":7301,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.728552+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:25.728552+0000","last_peered":"2024-09-16T10:25:25.728552+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:25.728552+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7199,"log_dups_size":0,"ondisk_log_size":7199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29467590,"num_objects":7196,"num_object_clones":0,"num_object_copies":14392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7197,"num_write_kb":28779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":2919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7267","reported_seq":7351,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:25.632079+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:25.632079+0000","last_peered":"2024-09-16T10:25:25.632079+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:25.632079+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7267,"log_dups_size":0,"ondisk_log_size":7267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29742022,"num_objects":7263,"num_object_clones":0,"num_object_copies":14526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7265,"num_write_kb":29047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3071}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7208","reported_seq":7285,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.681312+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:29.681312+0000","last_peered":"2024-09-16T10:25:29.681312+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:29.681312+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7208,"log_dups_size":0,"ondisk_log_size":7208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29516259,"num_objects":7207,"num_object_clones":0,"num_object_copies":14414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":28825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3048},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7554","reported_seq":7684,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.894964+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:29.894964+0000","last_peered":"2024-09-16T10:25:29.894964+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:29.894964+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30909865,"num_objects":7549,"num_object_clones":0,"num_object_copies":15098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":30188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3212},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7253","reported_seq":7331,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.861016+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:29.861016+0000","last_peered":"2024-09-16T10:25:29.861016+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:29.861016+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7253,"log_dups_size":0,"ondisk_log_size":7253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29692870,"num_objects":7251,"num_object_clones":0,"num_object_copies":14502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7251,"num_write_kb":28998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3084},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7561","reported_seq":7651,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.806972+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:29.806972+0000","last_peered":"2024-09-16T10:25:29.806972+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:29.806972+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7561,"log_dups_size":0,"ondisk_log_size":7561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30922636,"num_objects":7553,"num_object_clones":0,"num_object_copies":15106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":30203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3158},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":481858038,"num_objects":117657,"num_object_clones":0,"num_object_copies":235314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1483,"num_objects_unfound":0,"num_objects_dirty":117657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95286,"num_write_kb":381032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2735,"num_bytes_recovered":11121123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":974688256,"data_stored":974605157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117721,"ondisk_log_size":117721,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018915,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603112,"kb_used_data":368864,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93768728,"statfs":{"total":96636764160,"available":96019177472,"internally_reserved":0,"allocated":377716736,"data_stored":377473497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":239857328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.20599999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.246},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.6870000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691860,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306208,"kb_used_data":158100,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":94065632,"statfs":{"total":96636764160,"available":96323207168,"internally_reserved":0,"allocated":161894400,"data_stored":161578901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":151645866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.69899999999999995},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":2.1890000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.98999999999999999},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":2.0070000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215144,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690528,"kb_used_data":424344,"kb_used_omap":4,"kb_used_meta":266171,"kb_avail":93681312,"statfs":{"total":96636764160,"available":95929663488,"internally_reserved":0,"allocated":434528256,"data_stored":434288631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":272559784},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,2,1,3,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":2.6419999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.339}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":2.0350000000000001},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.6789999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":434278400,"data_stored":434274787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":162545664,"data_stored":162469791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":377864192,"data_stored":377860579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:32.374 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:32.374 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'7405", 'reported_seq': 7471, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:29.854551+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:25:29.854551+0000', 'last_peered': '2024-09-16T10:25:29.854551+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:25:29.854551+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7405, 'log_dups_size': 0, 'ondisk_log_size': 7405, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30330880, 'num_objects': 7405, 'num_object_clones': 0, 'num_object_copies': 14810, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7405, 'num_write_kb': 29620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3213}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:32.374 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:32.374 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'7276", 'reported_seq': 7459, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:25.704098+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:25.704098+0000', 'last_peered': '2024-09-16T10:25:25.704098+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:25.704098+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 7276, 'log_dups_size': 0, 'ondisk_log_size': 7276, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29786112, 'num_objects': 7272, 'num_object_clones': 0, 'num_object_copies': 14544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3333, 'num_write_kb': 13332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 2937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:32.374 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:32.374 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'7300", 'reported_seq': 7461, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:29.811119+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:25:29.811119+0000', 'last_peered': '2024-09-16T10:25:29.811119+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:25:29.811119+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 7300, 'log_dups_size': 0, 'ondisk_log_size': 7300, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29880320, 'num_objects': 7295, 'num_object_clones': 0, 'num_object_copies': 14590, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3925, 'num_write_kb': 15700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3078}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:32.374 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'7633", 'reported_seq': 7804, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:29.819337+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:25:29.819337+0000', 'last_peered': '2024-09-16T10:25:29.819337+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:25:29.819337+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 7633, 'log_dups_size': 0, 'ondisk_log_size': 7633, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31240192, 'num_objects': 7627, 'num_object_clones': 0, 'num_object_copies': 15254, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3875, 'num_write_kb': 15500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3224}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'7164", 'reported_seq': 12690, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:25.723491+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:25.723491+0000', 'last_peered': '2024-09-16T10:25:25.723491+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:25.723491+0000', 'last_undegraded': '2024-09-16T10:25:25.723491+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7164, 'log_dups_size': 0, 'ondisk_log_size': 7164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29316038, 'num_objects': 7159, 'num_object_clones': 0, 'num_object_copies': 14318, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1483, 'num_objects_unfound': 0, 'num_objects_dirty': 7159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7162, 'num_write_kb': 28632, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2718, 'num_bytes_recovered': 11116544, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1483}, {'shards': '2', 'objects': 5676}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'7578", 'reported_seq': 7648, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:29.812547+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:25:29.812547+0000', 'last_peered': '2024-09-16T10:25:29.812547+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:25:29.812547+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7578, 'log_dups_size': 0, 'ondisk_log_size': 7578, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31007686, 'num_objects': 7572, 'num_object_clones': 0, 'num_object_copies': 15144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7576, 'num_write_kb': 30284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3138}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'7199", 'reported_seq': 7301, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:25.728552+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:25.728552+0000', 'last_peered': '2024-09-16T10:25:25.728552+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:25.728552+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 7199, 'log_dups_size': 0, 'ondisk_log_size': 7199, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29467590, 'num_objects': 7196, 'num_object_clones': 0, 'num_object_copies': 14392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7197, 'num_write_kb': 28779, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 2919}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'7267", 'reported_seq': 7351, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:25.632079+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:25.632079+0000', 'last_peered': '2024-09-16T10:25:25.632079+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:25.632079+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7267, 'log_dups_size': 0, 'ondisk_log_size': 7267, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29742022, 'num_objects': 7263, 'num_object_clones': 0, 'num_object_copies': 14526, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7265, 'num_write_kb': 29047, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 3071}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:32.375 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'7208", 'reported_seq': 7285, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:29.681312+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:25:29.681312+0000', 'last_peered': '2024-09-16T10:25:29.681312+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:25:29.681312+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7208, 'log_dups_size': 0, 'ondisk_log_size': 7208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29516259, 'num_objects': 7207, 'num_object_clones': 0, 'num_object_copies': 14414, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7207, 'num_write_kb': 28825, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3048}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:32.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:32.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'7554", 'reported_seq': 7684, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:29.894964+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:29.894964+0000', 'last_peered': '2024-09-16T10:25:29.894964+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:29.894964+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 7554, 'log_dups_size': 0, 'ondisk_log_size': 7554, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30909865, 'num_objects': 7549, 'num_object_clones': 0, 'num_object_copies': 15098, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7549, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7551, 'num_write_kb': 30188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3212}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:32.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:32.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'7253", 'reported_seq': 7331, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:29.861016+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:29.861016+0000', 'last_peered': '2024-09-16T10:25:29.861016+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:29.861016+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7253, 'log_dups_size': 0, 'ondisk_log_size': 7253, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29692870, 'num_objects': 7251, 'num_object_clones': 0, 'num_object_copies': 14502, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7251, 'num_write_kb': 28998, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3084}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:32.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:32.376 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'7561", 'reported_seq': 7651, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:29.806972+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:25:29.806972+0000', 'last_peered': '2024-09-16T10:25:29.806972+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:25:29.806972+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7561, 'log_dups_size': 0, 'ondisk_log_size': 7561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30922636, 'num_objects': 7553, 'num_object_clones': 0, 'num_object_copies': 15106, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7553, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7558, 'num_write_kb': 30203, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3158}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:32.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:32.547 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:25:32.544298+0000 min lat: 0.00139595 max lat: 2.61294 avg lat: 0.0233901 2024-09-16T10:25:32.547 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:25:32.547 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 140 20 119710 119690 3.33884 1.75391 0.0309038 0.0233901 2024-09-16T10:25:33.047 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:33.047 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/237164 objects degraded (19.871%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":221,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":118582,"data_bytes":485638665,"bytes_used":1647636480,"bytes_avail":288262656000,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":237164,"degraded_ratio":0.19871481337808436,"misplaced_objects":1298,"misplaced_total":237164,"misplaced_ratio":0.005473006021149922,"recovering_objects_per_sec":47,"recovering_bytes_per_sec":193143,"recovering_keys_per_sec":0,"num_objects_recovered":566,"num_bytes_recovered":2318336,"num_keys_recovered":0,"write_bytes_sec":1995933,"read_op_per_sec":0,"write_op_per_sec":487},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:33.076 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:33.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:33.545 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 141 20 120167 120147 3.32783 1.78516 0.104525 0.0234654 2024-09-16T10:25:33.637 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:33.638 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:33.663 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-16T10:25:32.602071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":485638665,"num_objects":118582,"num_object_clones":0,"num_object_copies":237164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1298,"num_objects_unfound":0,"num_objects_dirty":118582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96211,"num_write_kb":384726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2921,"num_bytes_recovered":11878902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118647,"ondisk_log_size":118647,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1609020,"kb_used_data":956384,"kb_used_omap":12,"kb_used_meta":652595,"kb_avail":281506500,"statfs":{"total":289910292480,"available":288262656000,"internally_reserved":0,"allocated":979337216,"data_stored":978538853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":668257282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,8,3,7,7,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23957504,"num_objects":5849,"num_object_clones":0,"num_object_copies":11698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-566,"num_objects_unfound":0,"num_objects_dirty":5849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5849,"num_write_kb":23396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":566,"num_bytes_recovered":2318336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003154"},"pg_stats":[{"pgid":"2.f","version":"42'7405","reported_seq":7471,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.854551+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:29.854551+0000","last_peered":"2024-09-16T10:25:29.854551+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:29.854551+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7405,"log_dups_size":0,"ondisk_log_size":7405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30330880,"num_objects":7405,"num_object_clones":0,"num_object_copies":14810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7405,"num_write_kb":29620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3213},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7436","reported_seq":7590,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:30.687765+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:30.687765+0000","last_peered":"2024-09-16T10:25:30.687765+0000","last_clean":"2024-09-16T10:25:30.687765+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:30.687765+0000","last_undegraded":"2024-09-16T10:25:30.687765+0000","last_fullsized":"2024-09-16T10:25:30.687765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7436,"log_dups_size":0,"ondisk_log_size":7436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30433280,"num_objects":7430,"num_object_clones":0,"num_object_copies":14860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":14596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7411","reported_seq":7594,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:30.603568+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:30.603568+0000","last_peered":"2024-09-16T10:25:30.603568+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:30.603568+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7411,"log_dups_size":0,"ondisk_log_size":7411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30339072,"num_objects":7407,"num_object_clones":0,"num_object_copies":14814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3468,"num_write_kb":13872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3072}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7423","reported_seq":7477,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:30.727738+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:30.727738+0000","last_peered":"2024-09-16T10:25:30.727738+0000","last_clean":"2024-09-16T10:25:30.727738+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:30.727738+0000","last_undegraded":"2024-09-16T10:25:30.727738+0000","last_fullsized":"2024-09-16T10:25:30.727738+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7423,"log_dups_size":0,"ondisk_log_size":7423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30404608,"num_objects":7423,"num_object_clones":0,"num_object_copies":14846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7423,"num_write_kb":29692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7392","reported_seq":7546,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:29.796589+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:29.796589+0000","last_peered":"2024-09-16T10:25:29.796589+0000","last_clean":"2024-09-16T10:25:29.796589+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:29.796589+0000","last_undegraded":"2024-09-16T10:25:29.796589+0000","last_fullsized":"2024-09-16T10:25:29.796589+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7392,"log_dups_size":0,"ondisk_log_size":7392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30248960,"num_objects":7385,"num_object_clones":0,"num_object_copies":14770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":16136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7300","reported_seq":7461,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.811119+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:29.811119+0000","last_peered":"2024-09-16T10:25:29.811119+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:29.811119+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7300,"log_dups_size":0,"ondisk_log_size":7300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29880320,"num_objects":7295,"num_object_clones":0,"num_object_copies":14590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3925,"num_write_kb":15700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3078},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7633","reported_seq":7804,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.819337+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:29.819337+0000","last_peered":"2024-09-16T10:25:29.819337+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:29.819337+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7633,"log_dups_size":0,"ondisk_log_size":7633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31240192,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3875,"num_write_kb":15500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3224},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7374","reported_seq":7546,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:29.851497+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:29.851497+0000","last_peered":"2024-09-16T10:25:29.851497+0000","last_clean":"2024-09-16T10:25:29.851497+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:29.851497+0000","last_undegraded":"2024-09-16T10:25:29.851497+0000","last_fullsized":"2024-09-16T10:25:29.851497+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7374,"log_dups_size":0,"ondisk_log_size":7374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30195712,"num_objects":7372,"num_object_clones":0,"num_object_copies":14744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3177,"num_write_kb":12708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7327","reported_seq":13223,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:30.719684+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:30.719684+0000","last_peered":"2024-09-16T10:25:30.719684+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:30.719684+0000","last_undegraded":"2024-09-16T10:25:30.719684+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7327,"log_dups_size":0,"ondisk_log_size":7327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29983686,"num_objects":7322,"num_object_clones":0,"num_object_copies":14644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1298,"num_objects_unfound":0,"num_objects_dirty":7322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7325,"num_write_kb":29284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2903,"num_bytes_recovered":11874304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1298},{"shards":"2","objects":6024}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7578","reported_seq":7648,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.812547+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:29.812547+0000","last_peered":"2024-09-16T10:25:29.812547+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:29.812547+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7578,"log_dups_size":0,"ondisk_log_size":7578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31007686,"num_objects":7572,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7576,"num_write_kb":30284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3138},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7360","reported_seq":7463,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:30.618921+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:30.618921+0000","last_peered":"2024-09-16T10:25:30.618921+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:30.618921+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7360,"log_dups_size":0,"ondisk_log_size":7360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30127046,"num_objects":7357,"num_object_clones":0,"num_object_copies":14714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7358,"num_write_kb":29423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7429","reported_seq":7513,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:30.692574+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:30.692574+0000","last_peered":"2024-09-16T10:25:30.692574+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:30.692574+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30405574,"num_objects":7425,"num_object_clones":0,"num_object_copies":14850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":29695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3233}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7208","reported_seq":7285,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.681312+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:29.681312+0000","last_peered":"2024-09-16T10:25:29.681312+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:29.681312+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7208,"log_dups_size":0,"ondisk_log_size":7208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29516259,"num_objects":7207,"num_object_clones":0,"num_object_copies":14414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7207,"num_write_kb":28825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3048},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7554","reported_seq":7684,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.894964+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:29.894964+0000","last_peered":"2024-09-16T10:25:29.894964+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:29.894964+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30909865,"num_objects":7549,"num_object_clones":0,"num_object_copies":15098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":30188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3212},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7253","reported_seq":7331,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.861016+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:29.861016+0000","last_peered":"2024-09-16T10:25:29.861016+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:29.861016+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7253,"log_dups_size":0,"ondisk_log_size":7253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29692870,"num_objects":7251,"num_object_clones":0,"num_object_copies":14502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7251,"num_write_kb":28998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3084},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7561","reported_seq":7651,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:29.806972+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:29.806972+0000","last_peered":"2024-09-16T10:25:29.806972+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:29.806972+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7561,"log_dups_size":0,"ondisk_log_size":7561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30922636,"num_objects":7553,"num_object_clones":0,"num_object_copies":15106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7558,"num_write_kb":30203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3158},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":485638646,"num_objects":118580,"num_object_clones":0,"num_object_copies":237160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1298,"num_objects_unfound":0,"num_objects_dirty":118580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96209,"num_write_kb":384724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2920,"num_bytes_recovered":11878883,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":979755008,"data_stored":979671909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118644,"ondisk_log_size":118644,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018916,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":612284,"kb_used_data":373940,"kb_used_omap":4,"kb_used_meta":238331,"kb_avail":93759556,"statfs":{"total":96636764160,"available":96009785344,"internally_reserved":0,"allocated":382914560,"data_stored":382671321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":244051632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,0,2,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":5.5650000000000004},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":5.3929999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":8.9659999999999993}]}]},{"osd":1,"up_from":40,"seq":171798691860,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306208,"kb_used_data":158100,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":94065632,"statfs":{"total":96636764160,"available":96323207168,"internally_reserved":0,"allocated":161894400,"data_stored":161578901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":151645866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.69899999999999995},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":2.1890000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.98999999999999999},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":2.0070000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215144,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690528,"kb_used_data":424344,"kb_used_omap":4,"kb_used_meta":266171,"kb_avail":93681312,"statfs":{"total":96636764160,"available":95929663488,"internally_reserved":0,"allocated":434528256,"data_stored":434288631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":272559784},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,2,1,3,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":2.6419999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.339}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":2.0350000000000001},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.6789999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":434278400,"data_stored":434274787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":162545664,"data_stored":162469791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382930944,"data_stored":382927331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:34.545 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 142 20 120680 120660 3.3185 2.00391 0.0547174 0.0235347 2024-09-16T10:25:35.545 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 143 20 121209 121189 3.30975 2.06641 0.0289487 0.0235981 2024-09-16T10:25:36.545 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 144 20 121718 121698 3.30057 1.98828 0.018665 0.0236631 2024-09-16T10:25:36.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:37.254 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:37.256 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:37.279 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-16T10:25:36.605838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":495768073,"num_objects":121055,"num_object_clones":0,"num_object_copies":242110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":121055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98684,"num_write_kb":394618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3107,"num_bytes_recovered":12640758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121120,"ondisk_log_size":121120,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1613248,"kb_used_data":977000,"kb_used_omap":12,"kb_used_meta":636211,"kb_avail":281502272,"statfs":{"total":289910292480,"available":288258326528,"internally_reserved":0,"allocated":1000448000,"data_stored":999649637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":651480066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,5,6,7,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30220288,"num_objects":7378,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-560,"num_objects_unfound":0,"num_objects_dirty":7378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7378,"num_write_kb":29512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":560,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005920"},"pg_stats":[{"pgid":"2.f","version":"42'7573","reported_seq":7639,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.940200+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:34.940200+0000","last_peered":"2024-09-16T10:25:34.940200+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:34.940200+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31019008,"num_objects":7573,"num_object_clones":0,"num_object_copies":15146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7573,"num_write_kb":30292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3381},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7575","reported_seq":7729,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:35.685352+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:35.685352+0000","last_peered":"2024-09-16T10:25:35.685352+0000","last_clean":"2024-09-16T10:25:35.685352+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:35.685352+0000","last_undegraded":"2024-09-16T10:25:35.685352+0000","last_fullsized":"2024-09-16T10:25:35.685352+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7575,"log_dups_size":0,"ondisk_log_size":7575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31002624,"num_objects":7569,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3788,"num_write_kb":15152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7589","reported_seq":7772,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:35.730557+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:35.730557+0000","last_peered":"2024-09-16T10:25:35.730557+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:35.730557+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31068160,"num_objects":7585,"num_object_clones":0,"num_object_copies":15170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3646,"num_write_kb":14584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3250}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7586","reported_seq":7640,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:35.681106+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:35.681106+0000","last_peered":"2024-09-16T10:25:35.681106+0000","last_clean":"2024-09-16T10:25:35.681106+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:35.681106+0000","last_undegraded":"2024-09-16T10:25:35.681106+0000","last_fullsized":"2024-09-16T10:25:35.681106+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31072256,"num_objects":7586,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7586,"num_write_kb":30344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7542","reported_seq":7696,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:34.834922+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:34.834922+0000","last_peered":"2024-09-16T10:25:34.834922+0000","last_clean":"2024-09-16T10:25:34.834922+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:34.834922+0000","last_undegraded":"2024-09-16T10:25:34.834922+0000","last_fullsized":"2024-09-16T10:25:34.834922+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30863360,"num_objects":7535,"num_object_clones":0,"num_object_copies":15070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4184,"num_write_kb":16736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7457","reported_seq":7618,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.882654+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:34.882654+0000","last_peered":"2024-09-16T10:25:34.882654+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:34.882654+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7457,"log_dups_size":0,"ondisk_log_size":7457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30523392,"num_objects":7452,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4082,"num_write_kb":16328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3235},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7790","reported_seq":7961,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.919692+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:34.919692+0000","last_peered":"2024-09-16T10:25:34.919692+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:34.919692+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7790,"log_dups_size":0,"ondisk_log_size":7790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31883264,"num_objects":7784,"num_object_clones":0,"num_object_copies":15568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4032,"num_write_kb":16128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3381},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7543","reported_seq":7715,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:34.925325+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:34.925325+0000","last_peered":"2024-09-16T10:25:34.925325+0000","last_clean":"2024-09-16T10:25:34.925325+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:34.925325+0000","last_undegraded":"2024-09-16T10:25:34.925325+0000","last_fullsized":"2024-09-16T10:25:34.925325+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30887936,"num_objects":7541,"num_object_clones":0,"num_object_copies":15082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":13384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7479","reported_seq":13746,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:35.750668+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:35.750668+0000","last_peered":"2024-09-16T10:25:35.750668+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:35.750668+0000","last_undegraded":"2024-09-16T10:25:35.750668+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30606278,"num_objects":7474,"num_object_clones":0,"num_object_copies":14948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":7474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7477,"num_write_kb":29892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3089,"num_bytes_recovered":12636160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1112},{"shards":"2","objects":6362}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7727","reported_seq":7801,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.950949+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:34.950949+0000","last_peered":"2024-09-16T10:25:34.950949+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:34.950949+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7727,"log_dups_size":0,"ondisk_log_size":7727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31617990,"num_objects":7721,"num_object_clones":0,"num_object_copies":15442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":30880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3287},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7515","reported_seq":7617,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:35.700028+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:35.700028+0000","last_peered":"2024-09-16T10:25:35.700028+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:35.700028+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7515,"log_dups_size":0,"ondisk_log_size":7515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30761926,"num_objects":7512,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7513,"num_write_kb":30043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7552","reported_seq":7636,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:35.718915+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:35.718915+0000","last_peered":"2024-09-16T10:25:35.718915+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:35.718915+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7552,"log_dups_size":0,"ondisk_log_size":7552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30909382,"num_objects":7548,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":30187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3356}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7352","reported_seq":7429,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.927025+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:34.927025+0000","last_peered":"2024-09-16T10:25:34.927025+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:34.927025+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7352,"log_dups_size":0,"ondisk_log_size":7352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30106083,"num_objects":7351,"num_object_clones":0,"num_object_copies":14702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":29401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3192},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7697","reported_seq":7827,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.861597+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:34.861597+0000","last_peered":"2024-09-16T10:25:34.861597+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:34.861597+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31495593,"num_objects":7692,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7694,"num_write_kb":30760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3355},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7401","reported_seq":7479,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.959688+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:34.959688+0000","last_peered":"2024-09-16T10:25:34.959688+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:34.959688+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30299078,"num_objects":7399,"num_object_clones":0,"num_object_copies":14798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":29590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3232},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7739","reported_seq":7826,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.910864+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:34.910864+0000","last_peered":"2024-09-16T10:25:34.910864+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:34.910864+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7739,"log_dups_size":0,"ondisk_log_size":7739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31651724,"num_objects":7731,"num_object_clones":0,"num_object_copies":15462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":30915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3336},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":495768054,"num_objects":121053,"num_object_clones":0,"num_object_copies":242106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":121053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98682,"num_write_kb":394616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3106,"num_bytes_recovered":12640739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1000853504,"data_stored":1000770405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121117,"ondisk_log_size":121117,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018917,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604968,"kb_used_data":378912,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93766872,"statfs":{"total":96636764160,"available":96017276928,"internally_reserved":0,"allocated":388005888,"data_stored":387762649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":231468720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,4,0,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":1.55},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.47099999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691861,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306244,"kb_used_data":166332,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94065596,"statfs":{"total":96636764160,"available":96323170304,"internally_reserved":0,"allocated":170323968,"data_stored":170008469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":143257258},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.69499999999999995},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.66900000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.27700000000000002},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":0.71199999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215145,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702036,"kb_used_data":431756,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93669804,"statfs":{"total":96636764160,"available":95917879296,"internally_reserved":0,"allocated":442118144,"data_stored":441878519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":276754088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.74199999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.78100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":441880576,"data_stored":441876963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171008000,"data_stored":170932127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387964928,"data_stored":387961315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:37.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:37.545 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 145 20 122180 122160 3.29025 1.80469 0.0313704 0.0237386 2024-09-16T10:25:37.828 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:37.829 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:37.853 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-16T10:25:36.605838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":495768073,"num_objects":121055,"num_object_clones":0,"num_object_copies":242110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":121055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98684,"num_write_kb":394618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3107,"num_bytes_recovered":12640758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121120,"ondisk_log_size":121120,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1613248,"kb_used_data":977000,"kb_used_omap":12,"kb_used_meta":636211,"kb_avail":281502272,"statfs":{"total":289910292480,"available":288258326528,"internally_reserved":0,"allocated":1000448000,"data_stored":999649637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":651480066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,5,6,7,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30220288,"num_objects":7378,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-560,"num_objects_unfound":0,"num_objects_dirty":7378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7378,"num_write_kb":29512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":560,"num_bytes_recovered":2293760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005920"},"pg_stats":[{"pgid":"2.f","version":"42'7573","reported_seq":7639,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.940200+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:34.940200+0000","last_peered":"2024-09-16T10:25:34.940200+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:34.940200+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31019008,"num_objects":7573,"num_object_clones":0,"num_object_copies":15146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7573,"num_write_kb":30292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3381},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7575","reported_seq":7729,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:35.685352+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:35.685352+0000","last_peered":"2024-09-16T10:25:35.685352+0000","last_clean":"2024-09-16T10:25:35.685352+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:35.685352+0000","last_undegraded":"2024-09-16T10:25:35.685352+0000","last_fullsized":"2024-09-16T10:25:35.685352+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7575,"log_dups_size":0,"ondisk_log_size":7575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31002624,"num_objects":7569,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3788,"num_write_kb":15152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7589","reported_seq":7772,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:35.730557+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:35.730557+0000","last_peered":"2024-09-16T10:25:35.730557+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:35.730557+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31068160,"num_objects":7585,"num_object_clones":0,"num_object_copies":15170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3646,"num_write_kb":14584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3250}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7586","reported_seq":7640,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:35.681106+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:35.681106+0000","last_peered":"2024-09-16T10:25:35.681106+0000","last_clean":"2024-09-16T10:25:35.681106+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:35.681106+0000","last_undegraded":"2024-09-16T10:25:35.681106+0000","last_fullsized":"2024-09-16T10:25:35.681106+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31072256,"num_objects":7586,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7586,"num_write_kb":30344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7542","reported_seq":7696,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:34.834922+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:34.834922+0000","last_peered":"2024-09-16T10:25:34.834922+0000","last_clean":"2024-09-16T10:25:34.834922+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:34.834922+0000","last_undegraded":"2024-09-16T10:25:34.834922+0000","last_fullsized":"2024-09-16T10:25:34.834922+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30863360,"num_objects":7535,"num_object_clones":0,"num_object_copies":15070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4184,"num_write_kb":16736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7457","reported_seq":7618,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.882654+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:34.882654+0000","last_peered":"2024-09-16T10:25:34.882654+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:34.882654+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7457,"log_dups_size":0,"ondisk_log_size":7457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30523392,"num_objects":7452,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4082,"num_write_kb":16328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3235},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7790","reported_seq":7961,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.919692+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:34.919692+0000","last_peered":"2024-09-16T10:25:34.919692+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:34.919692+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7790,"log_dups_size":0,"ondisk_log_size":7790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31883264,"num_objects":7784,"num_object_clones":0,"num_object_copies":15568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4032,"num_write_kb":16128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3381},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7543","reported_seq":7715,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:34.925325+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:34.925325+0000","last_peered":"2024-09-16T10:25:34.925325+0000","last_clean":"2024-09-16T10:25:34.925325+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:34.925325+0000","last_undegraded":"2024-09-16T10:25:34.925325+0000","last_fullsized":"2024-09-16T10:25:34.925325+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30887936,"num_objects":7541,"num_object_clones":0,"num_object_copies":15082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":13384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7479","reported_seq":13746,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:35.750668+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:35.750668+0000","last_peered":"2024-09-16T10:25:35.750668+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:35.750668+0000","last_undegraded":"2024-09-16T10:25:35.750668+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30606278,"num_objects":7474,"num_object_clones":0,"num_object_copies":14948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":7474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7477,"num_write_kb":29892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3089,"num_bytes_recovered":12636160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1112},{"shards":"2","objects":6362}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7727","reported_seq":7801,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.950949+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:34.950949+0000","last_peered":"2024-09-16T10:25:34.950949+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:34.950949+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7727,"log_dups_size":0,"ondisk_log_size":7727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31617990,"num_objects":7721,"num_object_clones":0,"num_object_copies":15442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":30880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3287},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7515","reported_seq":7617,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:35.700028+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:35.700028+0000","last_peered":"2024-09-16T10:25:35.700028+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:35.700028+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7515,"log_dups_size":0,"ondisk_log_size":7515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30761926,"num_objects":7512,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7513,"num_write_kb":30043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7552","reported_seq":7636,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:35.718915+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:35.718915+0000","last_peered":"2024-09-16T10:25:35.718915+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:35.718915+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7552,"log_dups_size":0,"ondisk_log_size":7552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30909382,"num_objects":7548,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":30187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3356}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7352","reported_seq":7429,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.927025+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:34.927025+0000","last_peered":"2024-09-16T10:25:34.927025+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:34.927025+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7352,"log_dups_size":0,"ondisk_log_size":7352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30106083,"num_objects":7351,"num_object_clones":0,"num_object_copies":14702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":29401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3192},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7697","reported_seq":7827,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.861597+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:34.861597+0000","last_peered":"2024-09-16T10:25:34.861597+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:34.861597+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31495593,"num_objects":7692,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7694,"num_write_kb":30760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3355},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7401","reported_seq":7479,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.959688+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:34.959688+0000","last_peered":"2024-09-16T10:25:34.959688+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:34.959688+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30299078,"num_objects":7399,"num_object_clones":0,"num_object_copies":14798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":29590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3232},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7739","reported_seq":7826,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.910864+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:34.910864+0000","last_peered":"2024-09-16T10:25:34.910864+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:34.910864+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7739,"log_dups_size":0,"ondisk_log_size":7739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31651724,"num_objects":7731,"num_object_clones":0,"num_object_copies":15462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":30915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3336},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":495768054,"num_objects":121053,"num_object_clones":0,"num_object_copies":242106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":121053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98682,"num_write_kb":394616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3106,"num_bytes_recovered":12640739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1000853504,"data_stored":1000770405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121117,"ondisk_log_size":121117,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018917,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604968,"kb_used_data":378912,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93766872,"statfs":{"total":96636764160,"available":96017276928,"internally_reserved":0,"allocated":388005888,"data_stored":387762649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":231468720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,4,0,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":1.55},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.47099999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691861,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306244,"kb_used_data":166332,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94065596,"statfs":{"total":96636764160,"available":96323170304,"internally_reserved":0,"allocated":170323968,"data_stored":170008469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":143257258},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.69499999999999995},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.66900000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.27700000000000002},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":0.71199999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215145,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702036,"kb_used_data":431756,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93669804,"statfs":{"total":96636764160,"available":95917879296,"internally_reserved":0,"allocated":442118144,"data_stored":441878519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":276754088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.74199999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.78100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":441880576,"data_stored":441876963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171008000,"data_stored":170932127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387964928,"data_stored":387961315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:37.854 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:37.854 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'7573", 'reported_seq': 7639, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:34.940200+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:25:34.940200+0000', 'last_peered': '2024-09-16T10:25:34.940200+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:25:34.940200+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7573, 'log_dups_size': 0, 'ondisk_log_size': 7573, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31019008, 'num_objects': 7573, 'num_object_clones': 0, 'num_object_copies': 15146, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7573, 'num_write_kb': 30292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3381}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:37.854 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:37.855 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'7589", 'reported_seq': 7772, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:35.730557+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:35.730557+0000', 'last_peered': '2024-09-16T10:25:35.730557+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:35.730557+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 7589, 'log_dups_size': 0, 'ondisk_log_size': 7589, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31068160, 'num_objects': 7585, 'num_object_clones': 0, 'num_object_copies': 15170, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3646, 'num_write_kb': 14584, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 3250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:37.855 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:37.855 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'7457", 'reported_seq': 7618, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:34.882654+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:25:34.882654+0000', 'last_peered': '2024-09-16T10:25:34.882654+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:25:34.882654+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 7457, 'log_dups_size': 0, 'ondisk_log_size': 7457, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30523392, 'num_objects': 7452, 'num_object_clones': 0, 'num_object_copies': 14904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4082, 'num_write_kb': 16328, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3235}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:37.855 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:37.855 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'7790", 'reported_seq': 7961, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:34.919692+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:25:34.919692+0000', 'last_peered': '2024-09-16T10:25:34.919692+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:25:34.919692+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 7790, 'log_dups_size': 0, 'ondisk_log_size': 7790, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31883264, 'num_objects': 7784, 'num_object_clones': 0, 'num_object_copies': 15568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4032, 'num_write_kb': 16128, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3381}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:37.855 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:37.855 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'7479", 'reported_seq': 13746, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:35.750668+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:35.750668+0000', 'last_peered': '2024-09-16T10:25:35.750668+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:35.750668+0000', 'last_undegraded': '2024-09-16T10:25:35.750668+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7479, 'log_dups_size': 0, 'ondisk_log_size': 7479, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30606278, 'num_objects': 7474, 'num_object_clones': 0, 'num_object_copies': 14948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1112, 'num_objects_unfound': 0, 'num_objects_dirty': 7474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7477, 'num_write_kb': 29892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3089, 'num_bytes_recovered': 12636160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1112}, {'shards': '2', 'objects': 6362}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:37.855 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:37.855 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'7727", 'reported_seq': 7801, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:34.950949+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:25:34.950949+0000', 'last_peered': '2024-09-16T10:25:34.950949+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:25:34.950949+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7727, 'log_dups_size': 0, 'ondisk_log_size': 7727, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31617990, 'num_objects': 7721, 'num_object_clones': 0, 'num_object_copies': 15442, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7725, 'num_write_kb': 30880, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3287}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:37.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:37.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'7515", 'reported_seq': 7617, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:35.700028+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:35.700028+0000', 'last_peered': '2024-09-16T10:25:35.700028+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:35.700028+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 7515, 'log_dups_size': 0, 'ondisk_log_size': 7515, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30761926, 'num_objects': 7512, 'num_object_clones': 0, 'num_object_copies': 15024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7513, 'num_write_kb': 30043, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 3235}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:37.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:37.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'7552", 'reported_seq': 7636, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:35.718915+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:35.718915+0000', 'last_peered': '2024-09-16T10:25:35.718915+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:35.718915+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7552, 'log_dups_size': 0, 'ondisk_log_size': 7552, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30909382, 'num_objects': 7548, 'num_object_clones': 0, 'num_object_copies': 15096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7550, 'num_write_kb': 30187, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 3356}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:37.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:37.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'7352", 'reported_seq': 7429, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:34.927025+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:25:34.927025+0000', 'last_peered': '2024-09-16T10:25:34.927025+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:25:34.927025+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7352, 'log_dups_size': 0, 'ondisk_log_size': 7352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30106083, 'num_objects': 7351, 'num_object_clones': 0, 'num_object_copies': 14702, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7351, 'num_write_kb': 29401, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3192}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:37.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:37.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'7697", 'reported_seq': 7827, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:34.861597+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:34.861597+0000', 'last_peered': '2024-09-16T10:25:34.861597+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:34.861597+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 7697, 'log_dups_size': 0, 'ondisk_log_size': 7697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31495593, 'num_objects': 7692, 'num_object_clones': 0, 'num_object_copies': 15384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7692, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7694, 'num_write_kb': 30760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3355}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:37.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:37.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'7401", 'reported_seq': 7479, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:34.959688+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:34.959688+0000', 'last_peered': '2024-09-16T10:25:34.959688+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:34.959688+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7401, 'log_dups_size': 0, 'ondisk_log_size': 7401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30299078, 'num_objects': 7399, 'num_object_clones': 0, 'num_object_copies': 14798, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7399, 'num_write_kb': 29590, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3232}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:37.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:37.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'7739", 'reported_seq': 7826, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:34.910864+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:25:34.910864+0000', 'last_peered': '2024-09-16T10:25:34.910864+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:25:34.910864+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7739, 'log_dups_size': 0, 'ondisk_log_size': 7739, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31651724, 'num_objects': 7731, 'num_object_clones': 0, 'num_object_copies': 15462, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7736, 'num_write_kb': 30915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3336}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:37.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:38.519 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:38.520 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/242110 objects degraded (19.466%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":227,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":121055,"data_bytes":495768073,"bytes_used":1651965952,"bytes_avail":288258326528,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":242110,"degraded_ratio":0.19465532196109206,"misplaced_objects":1112,"misplaced_total":242110,"misplaced_ratio":0.0045929536161249023,"recovering_objects_per_sec":46,"recovering_bytes_per_sec":191052,"recovering_keys_per_sec":0,"num_objects_recovered":560,"num_bytes_recovered":2293760,"num_keys_recovered":0,"write_bytes_sec":2517115,"read_op_per_sec":0,"write_op_per_sec":614},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:38.543 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:38.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:38.548 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 146 20 122620 122600 3.27945 1.71875 0.0183017 0.0238126 2024-09-16T10:25:39.096 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:39.122 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-16T10:25:38.606388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":495768073,"num_objects":121055,"num_object_clones":0,"num_object_copies":242110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":121055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98684,"num_write_kb":394618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3107,"num_bytes_recovered":12640758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121120,"ondisk_log_size":121120,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1625160,"kb_used_data":984816,"kb_used_omap":12,"kb_used_meta":640307,"kb_avail":281490360,"statfs":{"total":289910292480,"available":288246128640,"internally_reserved":0,"allocated":1008451584,"data_stored":1007653221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":655674370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,0,6,10,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20131840,"num_objects":4915,"num_object_clones":0,"num_object_copies":9830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-371,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4915,"num_write_kb":19660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":1519616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005991"},"pg_stats":[{"pgid":"2.f","version":"42'7573","reported_seq":7639,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.940200+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:34.940200+0000","last_peered":"2024-09-16T10:25:34.940200+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:34.940200+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7573,"log_dups_size":0,"ondisk_log_size":7573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31019008,"num_objects":7573,"num_object_clones":0,"num_object_copies":15146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7573,"num_write_kb":30292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3381},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7575","reported_seq":7729,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:35.685352+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:35.685352+0000","last_peered":"2024-09-16T10:25:35.685352+0000","last_clean":"2024-09-16T10:25:35.685352+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:35.685352+0000","last_undegraded":"2024-09-16T10:25:35.685352+0000","last_fullsized":"2024-09-16T10:25:35.685352+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7575,"log_dups_size":0,"ondisk_log_size":7575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31002624,"num_objects":7569,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3788,"num_write_kb":15152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7589","reported_seq":7772,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:35.730557+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:35.730557+0000","last_peered":"2024-09-16T10:25:35.730557+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:35.730557+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7589,"log_dups_size":0,"ondisk_log_size":7589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31068160,"num_objects":7585,"num_object_clones":0,"num_object_copies":15170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3646,"num_write_kb":14584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3250}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7586","reported_seq":7640,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:35.681106+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:35.681106+0000","last_peered":"2024-09-16T10:25:35.681106+0000","last_clean":"2024-09-16T10:25:35.681106+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:35.681106+0000","last_undegraded":"2024-09-16T10:25:35.681106+0000","last_fullsized":"2024-09-16T10:25:35.681106+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7586,"log_dups_size":0,"ondisk_log_size":7586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31072256,"num_objects":7586,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7586,"num_write_kb":30344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7542","reported_seq":7696,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:34.834922+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:34.834922+0000","last_peered":"2024-09-16T10:25:34.834922+0000","last_clean":"2024-09-16T10:25:34.834922+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:34.834922+0000","last_undegraded":"2024-09-16T10:25:34.834922+0000","last_fullsized":"2024-09-16T10:25:34.834922+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30863360,"num_objects":7535,"num_object_clones":0,"num_object_copies":15070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4184,"num_write_kb":16736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7457","reported_seq":7618,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.882654+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:34.882654+0000","last_peered":"2024-09-16T10:25:34.882654+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:34.882654+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7457,"log_dups_size":0,"ondisk_log_size":7457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30523392,"num_objects":7452,"num_object_clones":0,"num_object_copies":14904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4082,"num_write_kb":16328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3235},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7790","reported_seq":7961,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.919692+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:34.919692+0000","last_peered":"2024-09-16T10:25:34.919692+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:34.919692+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7790,"log_dups_size":0,"ondisk_log_size":7790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31883264,"num_objects":7784,"num_object_clones":0,"num_object_copies":15568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4032,"num_write_kb":16128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3381},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7543","reported_seq":7715,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:34.925325+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:34.925325+0000","last_peered":"2024-09-16T10:25:34.925325+0000","last_clean":"2024-09-16T10:25:34.925325+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:34.925325+0000","last_undegraded":"2024-09-16T10:25:34.925325+0000","last_fullsized":"2024-09-16T10:25:34.925325+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7543,"log_dups_size":0,"ondisk_log_size":7543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30887936,"num_objects":7541,"num_object_clones":0,"num_object_copies":15082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3346,"num_write_kb":13384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7479","reported_seq":13746,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:35.750668+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:35.750668+0000","last_peered":"2024-09-16T10:25:35.750668+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:35.750668+0000","last_undegraded":"2024-09-16T10:25:35.750668+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7479,"log_dups_size":0,"ondisk_log_size":7479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30606278,"num_objects":7474,"num_object_clones":0,"num_object_copies":14948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":7474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7477,"num_write_kb":29892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3089,"num_bytes_recovered":12636160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1112},{"shards":"2","objects":6362}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7727","reported_seq":7801,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.950949+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:34.950949+0000","last_peered":"2024-09-16T10:25:34.950949+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:34.950949+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7727,"log_dups_size":0,"ondisk_log_size":7727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31617990,"num_objects":7721,"num_object_clones":0,"num_object_copies":15442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":30880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3287},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7515","reported_seq":7617,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:35.700028+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:35.700028+0000","last_peered":"2024-09-16T10:25:35.700028+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:35.700028+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7515,"log_dups_size":0,"ondisk_log_size":7515,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30761926,"num_objects":7512,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7513,"num_write_kb":30043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7552","reported_seq":7636,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:35.718915+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:35.718915+0000","last_peered":"2024-09-16T10:25:35.718915+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:35.718915+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7552,"log_dups_size":0,"ondisk_log_size":7552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30909382,"num_objects":7548,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7550,"num_write_kb":30187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3356}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7352","reported_seq":7429,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.927025+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:34.927025+0000","last_peered":"2024-09-16T10:25:34.927025+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:34.927025+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7352,"log_dups_size":0,"ondisk_log_size":7352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30106083,"num_objects":7351,"num_object_clones":0,"num_object_copies":14702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7351,"num_write_kb":29401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3192},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7697","reported_seq":7827,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.861597+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:34.861597+0000","last_peered":"2024-09-16T10:25:34.861597+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:34.861597+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31495593,"num_objects":7692,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7694,"num_write_kb":30760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3355},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7401","reported_seq":7479,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.959688+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:34.959688+0000","last_peered":"2024-09-16T10:25:34.959688+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:34.959688+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30299078,"num_objects":7399,"num_object_clones":0,"num_object_copies":14798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":29590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3232},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7739","reported_seq":7826,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:34.910864+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:34.910864+0000","last_peered":"2024-09-16T10:25:34.910864+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:34.910864+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7739,"log_dups_size":0,"ondisk_log_size":7739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31651724,"num_objects":7731,"num_object_clones":0,"num_object_copies":15462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7736,"num_write_kb":30915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3336},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":495768054,"num_objects":121053,"num_object_clones":0,"num_object_copies":242106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":1112,"num_objects_unfound":0,"num_objects_dirty":121053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98682,"num_write_kb":394616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3106,"num_bytes_recovered":12640739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1009192960,"data_stored":1009109861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121117,"ondisk_log_size":121117,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018917,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604968,"kb_used_data":378912,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93766872,"statfs":{"total":96636764160,"available":96017276928,"internally_reserved":0,"allocated":388005888,"data_stored":387762649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":231468720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,4,0,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":1.55},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.47099999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318156,"kb_used_data":174148,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94053684,"statfs":{"total":96636764160,"available":96310972416,"internally_reserved":0,"allocated":178327552,"data_stored":178012053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":147451562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.17799999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":5.673}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":4.4470000000000001},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":6.2910000000000004}]}]},{"osd":0,"up_from":24,"seq":103079215145,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702036,"kb_used_data":431756,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93669804,"statfs":{"total":96636764160,"available":95917879296,"internally_reserved":0,"allocated":442118144,"data_stored":441878519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":276754088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,5,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.74199999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.78100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":441880576,"data_stored":441876963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179347456,"data_stored":179271583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387964928,"data_stored":387961315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:39.548 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 147 20 123103 123083 3.26998 1.88672 0.0372816 0.0238843 2024-09-16T10:25:40.547 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 148 20 123619 123599 3.2615 2.01562 0.0311425 0.0239477 2024-09-16T10:25:41.547 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 149 20 124064 124044 3.25128 1.73828 0.0160708 0.0240011 2024-09-16T10:25:42.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:42.547 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 150 20 124517 124497 3.2414 1.76953 0.0706605 0.0240949 2024-09-16T10:25:42.671 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:42.671 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:42.699 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-16T10:25:42.607363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":505733641,"num_objects":123488,"num_object_clones":0,"num_object_copies":246976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":904,"num_objects_unfound":0,"num_objects_dirty":123488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101117,"num_write_kb":404350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3315,"num_bytes_recovered":13492726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123553,"ondisk_log_size":123553,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1645352,"kb_used_data":996816,"kb_used_omap":12,"kb_used_meta":648499,"kb_avail":281470168,"statfs":{"total":289910292480,"available":288225452032,"internally_reserved":0,"allocated":1020739584,"data_stored":1019941221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":664062978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,5,9,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23875584,"num_objects":5829,"num_object_clones":0,"num_object_copies":11658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-579,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5829,"num_write_kb":23316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":2371584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005789"},"pg_stats":[{"pgid":"2.f","version":"42'7741","reported_seq":7807,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.968386+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:39.968386+0000","last_peered":"2024-09-16T10:25:39.968386+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:39.968386+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31707136,"num_objects":7741,"num_object_clones":0,"num_object_copies":15482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7741,"num_write_kb":30964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3549},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7713","reported_seq":7867,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:40.775723+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:40.775723+0000","last_peered":"2024-09-16T10:25:40.775723+0000","last_clean":"2024-09-16T10:25:40.775723+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:40.775723+0000","last_undegraded":"2024-09-16T10:25:40.775723+0000","last_fullsized":"2024-09-16T10:25:40.775723+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31567872,"num_objects":7707,"num_object_clones":0,"num_object_copies":15414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3926,"num_write_kb":15704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7736","reported_seq":7919,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:40.777392+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:40.777392+0000","last_peered":"2024-09-16T10:25:40.777392+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:40.777392+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7736,"log_dups_size":0,"ondisk_log_size":7736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31670272,"num_objects":7732,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":15172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3397}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7744","reported_seq":7798,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:40.716893+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:40.716893+0000","last_peered":"2024-09-16T10:25:40.716893+0000","last_clean":"2024-09-16T10:25:40.716893+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:40.716893+0000","last_undegraded":"2024-09-16T10:25:40.716893+0000","last_fullsized":"2024-09-16T10:25:40.716893+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7744,"log_dups_size":0,"ondisk_log_size":7744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7744,"num_write_kb":30976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7698","reported_seq":7852,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:39.937594+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:39.937594+0000","last_peered":"2024-09-16T10:25:39.937594+0000","last_clean":"2024-09-16T10:25:39.937594+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:39.937594+0000","last_undegraded":"2024-09-16T10:25:39.937594+0000","last_fullsized":"2024-09-16T10:25:39.937594+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31502336,"num_objects":7691,"num_object_clones":0,"num_object_copies":15382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4340,"num_write_kb":17360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7602","reported_seq":7761,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.851911+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:39.851911+0000","last_peered":"2024-09-16T10:25:39.851911+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:39.851911+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31117312,"num_objects":7597,"num_object_clones":0,"num_object_copies":15194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4227,"num_write_kb":16908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3380},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7932","reported_seq":8099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.912531+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:39.912531+0000","last_peered":"2024-09-16T10:25:39.912531+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:39.912531+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32464896,"num_objects":7926,"num_object_clones":0,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4174,"num_write_kb":16696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3523},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7715","reported_seq":7887,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:39.980129+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:39.980129+0000","last_peered":"2024-09-16T10:25:39.980129+0000","last_clean":"2024-09-16T10:25:39.980129+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:39.980129+0000","last_undegraded":"2024-09-16T10:25:39.980129+0000","last_fullsized":"2024-09-16T10:25:39.980129+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7715,"log_dups_size":0,"ondisk_log_size":7715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31592448,"num_objects":7713,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3518,"num_write_kb":14072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7628","reported_seq":14312,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:40.770655+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:40.770655+0000","last_peered":"2024-09-16T10:25:40.770655+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:40.770655+0000","last_undegraded":"2024-09-16T10:25:40.770655+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7628,"log_dups_size":0,"ondisk_log_size":7628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31216582,"num_objects":7623,"num_object_clones":0,"num_object_copies":15246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":904,"num_objects_unfound":0,"num_objects_dirty":7623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7626,"num_write_kb":30488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3297,"num_bytes_recovered":13488128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":904},{"shards":"2","objects":6719}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7885","reported_seq":7960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.945056+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:39.945056+0000","last_peered":"2024-09-16T10:25:39.945056+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:39.945056+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7885,"log_dups_size":0,"ondisk_log_size":7885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32265158,"num_objects":7879,"num_object_clones":0,"num_object_copies":15758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":31512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3445},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7657","reported_seq":7759,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:40.758728+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:40.758728+0000","last_peered":"2024-09-16T10:25:40.758728+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:40.758728+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7657,"log_dups_size":0,"ondisk_log_size":7657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31343558,"num_objects":7654,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7655,"num_write_kb":30611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7712","reported_seq":7796,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:40.699418+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:40.699418+0000","last_peered":"2024-09-16T10:25:40.699418+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:40.699418+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7712,"log_dups_size":0,"ondisk_log_size":7712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31564742,"num_objects":7708,"num_object_clones":0,"num_object_copies":15416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7710,"num_write_kb":30827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7502","reported_seq":7579,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.937725+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:39.937725+0000","last_peered":"2024-09-16T10:25:39.937725+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:39.937725+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7502,"log_dups_size":0,"ondisk_log_size":7502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30720483,"num_objects":7501,"num_object_clones":0,"num_object_copies":15002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7501,"num_write_kb":30001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3342},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7852","reported_seq":7980,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.960274+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:39.960274+0000","last_peered":"2024-09-16T10:25:39.960274+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:39.960274+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32130473,"num_objects":7847,"num_object_clones":0,"num_object_copies":15694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":31380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3510},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7542","reported_seq":7620,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.963094+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:39.963094+0000","last_peered":"2024-09-16T10:25:39.963094+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:39.963094+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30876614,"num_objects":7540,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":30154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3373},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7891","reported_seq":7981,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.918561+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:39.918561+0000","last_peered":"2024-09-16T10:25:39.918561+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:39.918561+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7891,"log_dups_size":0,"ondisk_log_size":7891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32274316,"num_objects":7883,"num_object_clones":0,"num_object_copies":15766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":31523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3488},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":505733622,"num_objects":123486,"num_object_clones":0,"num_object_copies":246972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":904,"num_objects_unfound":0,"num_objects_dirty":123486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101115,"num_write_kb":404348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3314,"num_bytes_recovered":13492707,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1021685760,"data_stored":1021602661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123550,"ondisk_log_size":123550,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018918,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613832,"kb_used_data":383680,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93758008,"statfs":{"total":96636764160,"available":96008200192,"internally_reserved":0,"allocated":392888320,"data_stored":392645081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":235663024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.317},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":5.907}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":1.6819999999999999},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.33800000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318156,"kb_used_data":174148,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94053684,"statfs":{"total":96636764160,"available":96310972416,"internally_reserved":0,"allocated":178327552,"data_stored":178012053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":147451562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.17799999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":5.673}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":4.4470000000000001},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":6.2910000000000004}]}]},{"osd":0,"up_from":24,"seq":103079215146,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713364,"kb_used_data":438988,"kb_used_omap":4,"kb_used_meta":274363,"kb_avail":93658476,"statfs":{"total":96636764160,"available":95906279424,"internally_reserved":0,"allocated":449523712,"data_stored":449284087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":280948392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.20100000000000001},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.2170000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.90100000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":449425408,"data_stored":449421795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179347456,"data_stored":179271583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392912896,"data_stored":392909283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:42.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:43.252 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:43.253 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:43.277 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-16T10:25:42.607363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":505733641,"num_objects":123488,"num_object_clones":0,"num_object_copies":246976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":904,"num_objects_unfound":0,"num_objects_dirty":123488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101117,"num_write_kb":404350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3315,"num_bytes_recovered":13492726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123553,"ondisk_log_size":123553,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1645352,"kb_used_data":996816,"kb_used_omap":12,"kb_used_meta":648499,"kb_avail":281470168,"statfs":{"total":289910292480,"available":288225452032,"internally_reserved":0,"allocated":1020739584,"data_stored":1019941221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":664062978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,5,9,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23875584,"num_objects":5829,"num_object_clones":0,"num_object_copies":11658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-579,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5829,"num_write_kb":23316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":2371584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005789"},"pg_stats":[{"pgid":"2.f","version":"42'7741","reported_seq":7807,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.968386+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:39.968386+0000","last_peered":"2024-09-16T10:25:39.968386+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:39.968386+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31707136,"num_objects":7741,"num_object_clones":0,"num_object_copies":15482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7741,"num_write_kb":30964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3549},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7713","reported_seq":7867,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:40.775723+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:40.775723+0000","last_peered":"2024-09-16T10:25:40.775723+0000","last_clean":"2024-09-16T10:25:40.775723+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:40.775723+0000","last_undegraded":"2024-09-16T10:25:40.775723+0000","last_fullsized":"2024-09-16T10:25:40.775723+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31567872,"num_objects":7707,"num_object_clones":0,"num_object_copies":15414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3926,"num_write_kb":15704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7736","reported_seq":7919,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:40.777392+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:40.777392+0000","last_peered":"2024-09-16T10:25:40.777392+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:40.777392+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7736,"log_dups_size":0,"ondisk_log_size":7736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31670272,"num_objects":7732,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":15172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3397}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7744","reported_seq":7798,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:40.716893+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:40.716893+0000","last_peered":"2024-09-16T10:25:40.716893+0000","last_clean":"2024-09-16T10:25:40.716893+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:40.716893+0000","last_undegraded":"2024-09-16T10:25:40.716893+0000","last_fullsized":"2024-09-16T10:25:40.716893+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7744,"log_dups_size":0,"ondisk_log_size":7744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7744,"num_write_kb":30976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7698","reported_seq":7852,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:39.937594+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:39.937594+0000","last_peered":"2024-09-16T10:25:39.937594+0000","last_clean":"2024-09-16T10:25:39.937594+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:39.937594+0000","last_undegraded":"2024-09-16T10:25:39.937594+0000","last_fullsized":"2024-09-16T10:25:39.937594+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31502336,"num_objects":7691,"num_object_clones":0,"num_object_copies":15382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4340,"num_write_kb":17360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7602","reported_seq":7761,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.851911+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:39.851911+0000","last_peered":"2024-09-16T10:25:39.851911+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:39.851911+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31117312,"num_objects":7597,"num_object_clones":0,"num_object_copies":15194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4227,"num_write_kb":16908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3380},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7932","reported_seq":8099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.912531+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:39.912531+0000","last_peered":"2024-09-16T10:25:39.912531+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:39.912531+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32464896,"num_objects":7926,"num_object_clones":0,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4174,"num_write_kb":16696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3523},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7715","reported_seq":7887,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:39.980129+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:39.980129+0000","last_peered":"2024-09-16T10:25:39.980129+0000","last_clean":"2024-09-16T10:25:39.980129+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:39.980129+0000","last_undegraded":"2024-09-16T10:25:39.980129+0000","last_fullsized":"2024-09-16T10:25:39.980129+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7715,"log_dups_size":0,"ondisk_log_size":7715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31592448,"num_objects":7713,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3518,"num_write_kb":14072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7628","reported_seq":14312,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:40.770655+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:40.770655+0000","last_peered":"2024-09-16T10:25:40.770655+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:40.770655+0000","last_undegraded":"2024-09-16T10:25:40.770655+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7628,"log_dups_size":0,"ondisk_log_size":7628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31216582,"num_objects":7623,"num_object_clones":0,"num_object_copies":15246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":904,"num_objects_unfound":0,"num_objects_dirty":7623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7626,"num_write_kb":30488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3297,"num_bytes_recovered":13488128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":904},{"shards":"2","objects":6719}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7885","reported_seq":7960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.945056+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:39.945056+0000","last_peered":"2024-09-16T10:25:39.945056+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:39.945056+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7885,"log_dups_size":0,"ondisk_log_size":7885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32265158,"num_objects":7879,"num_object_clones":0,"num_object_copies":15758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":31512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3445},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7657","reported_seq":7759,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:40.758728+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:40.758728+0000","last_peered":"2024-09-16T10:25:40.758728+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:40.758728+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7657,"log_dups_size":0,"ondisk_log_size":7657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31343558,"num_objects":7654,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7655,"num_write_kb":30611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7712","reported_seq":7796,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:40.699418+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:40.699418+0000","last_peered":"2024-09-16T10:25:40.699418+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:40.699418+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7712,"log_dups_size":0,"ondisk_log_size":7712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31564742,"num_objects":7708,"num_object_clones":0,"num_object_copies":15416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7710,"num_write_kb":30827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7502","reported_seq":7579,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.937725+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:39.937725+0000","last_peered":"2024-09-16T10:25:39.937725+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:39.937725+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7502,"log_dups_size":0,"ondisk_log_size":7502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30720483,"num_objects":7501,"num_object_clones":0,"num_object_copies":15002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7501,"num_write_kb":30001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3342},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7852","reported_seq":7980,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.960274+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:39.960274+0000","last_peered":"2024-09-16T10:25:39.960274+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:39.960274+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32130473,"num_objects":7847,"num_object_clones":0,"num_object_copies":15694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":31380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3510},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7542","reported_seq":7620,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.963094+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:39.963094+0000","last_peered":"2024-09-16T10:25:39.963094+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:39.963094+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30876614,"num_objects":7540,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":30154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3373},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7891","reported_seq":7981,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.918561+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:39.918561+0000","last_peered":"2024-09-16T10:25:39.918561+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:39.918561+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7891,"log_dups_size":0,"ondisk_log_size":7891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32274316,"num_objects":7883,"num_object_clones":0,"num_object_copies":15766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":31523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3488},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":505733622,"num_objects":123486,"num_object_clones":0,"num_object_copies":246972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":904,"num_objects_unfound":0,"num_objects_dirty":123486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101115,"num_write_kb":404348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3314,"num_bytes_recovered":13492707,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1021685760,"data_stored":1021602661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123550,"ondisk_log_size":123550,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018918,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613832,"kb_used_data":383680,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93758008,"statfs":{"total":96636764160,"available":96008200192,"internally_reserved":0,"allocated":392888320,"data_stored":392645081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":235663024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.317},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":5.907}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":1.6819999999999999},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.33800000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318156,"kb_used_data":174148,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94053684,"statfs":{"total":96636764160,"available":96310972416,"internally_reserved":0,"allocated":178327552,"data_stored":178012053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":147451562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.17799999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":5.673}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":4.4470000000000001},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":6.2910000000000004}]}]},{"osd":0,"up_from":24,"seq":103079215146,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713364,"kb_used_data":438988,"kb_used_omap":4,"kb_used_meta":274363,"kb_avail":93658476,"statfs":{"total":96636764160,"available":95906279424,"internally_reserved":0,"allocated":449523712,"data_stored":449284087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":280948392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.20100000000000001},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.2170000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.90100000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":449425408,"data_stored":449421795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179347456,"data_stored":179271583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392912896,"data_stored":392909283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:43.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:43.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'7741", 'reported_seq': 7807, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:39.968386+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:25:39.968386+0000', 'last_peered': '2024-09-16T10:25:39.968386+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:25:39.968386+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7741, 'log_dups_size': 0, 'ondisk_log_size': 7741, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31707136, 'num_objects': 7741, 'num_object_clones': 0, 'num_object_copies': 15482, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7741, 'num_write_kb': 30964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3549}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:43.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:43.278 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'7736", 'reported_seq': 7919, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:40.777392+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:40.777392+0000', 'last_peered': '2024-09-16T10:25:40.777392+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:40.777392+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 7736, 'log_dups_size': 0, 'ondisk_log_size': 7736, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31670272, 'num_objects': 7732, 'num_object_clones': 0, 'num_object_copies': 15464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3793, 'num_write_kb': 15172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 3397}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:43.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:43.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'7602", 'reported_seq': 7761, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:39.851911+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:25:39.851911+0000', 'last_peered': '2024-09-16T10:25:39.851911+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:25:39.851911+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 7602, 'log_dups_size': 0, 'ondisk_log_size': 7602, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31117312, 'num_objects': 7597, 'num_object_clones': 0, 'num_object_copies': 15194, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4227, 'num_write_kb': 16908, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3380}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:43.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:43.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'7932", 'reported_seq': 8099, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:39.912531+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:25:39.912531+0000', 'last_peered': '2024-09-16T10:25:39.912531+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:25:39.912531+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 7932, 'log_dups_size': 0, 'ondisk_log_size': 7932, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32464896, 'num_objects': 7926, 'num_object_clones': 0, 'num_object_copies': 15852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4174, 'num_write_kb': 16696, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3523}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:43.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:43.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'7628", 'reported_seq': 14312, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:40.770655+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:40.770655+0000', 'last_peered': '2024-09-16T10:25:40.770655+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:40.770655+0000', 'last_undegraded': '2024-09-16T10:25:40.770655+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7628, 'log_dups_size': 0, 'ondisk_log_size': 7628, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31216582, 'num_objects': 7623, 'num_object_clones': 0, 'num_object_copies': 15246, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 904, 'num_objects_unfound': 0, 'num_objects_dirty': 7623, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7626, 'num_write_kb': 30488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3297, 'num_bytes_recovered': 13488128, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 904}, {'shards': '2', 'objects': 6719}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:43.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:43.279 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'7885", 'reported_seq': 7960, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:39.945056+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:25:39.945056+0000', 'last_peered': '2024-09-16T10:25:39.945056+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:25:39.945056+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7885, 'log_dups_size': 0, 'ondisk_log_size': 7885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32265158, 'num_objects': 7879, 'num_object_clones': 0, 'num_object_copies': 15758, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7883, 'num_write_kb': 31512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3445}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'7657", 'reported_seq': 7759, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:40.758728+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:40.758728+0000', 'last_peered': '2024-09-16T10:25:40.758728+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:40.758728+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 7657, 'log_dups_size': 0, 'ondisk_log_size': 7657, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31343558, 'num_objects': 7654, 'num_object_clones': 0, 'num_object_copies': 15308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7654, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7655, 'num_write_kb': 30611, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 3377}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'7712", 'reported_seq': 7796, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:40.699418+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:40.699418+0000', 'last_peered': '2024-09-16T10:25:40.699418+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:40.699418+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7712, 'log_dups_size': 0, 'ondisk_log_size': 7712, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31564742, 'num_objects': 7708, 'num_object_clones': 0, 'num_object_copies': 15416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7708, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7710, 'num_write_kb': 30827, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 3516}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'7502", 'reported_seq': 7579, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:39.937725+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:25:39.937725+0000', 'last_peered': '2024-09-16T10:25:39.937725+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:25:39.937725+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7502, 'log_dups_size': 0, 'ondisk_log_size': 7502, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30720483, 'num_objects': 7501, 'num_object_clones': 0, 'num_object_copies': 15002, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7501, 'num_write_kb': 30001, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3342}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'7852", 'reported_seq': 7980, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:39.960274+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:39.960274+0000', 'last_peered': '2024-09-16T10:25:39.960274+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:39.960274+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 7852, 'log_dups_size': 0, 'ondisk_log_size': 7852, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32130473, 'num_objects': 7847, 'num_object_clones': 0, 'num_object_copies': 15694, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7849, 'num_write_kb': 31380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3510}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'7542", 'reported_seq': 7620, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:39.963094+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:39.963094+0000', 'last_peered': '2024-09-16T10:25:39.963094+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:39.963094+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7542, 'log_dups_size': 0, 'ondisk_log_size': 7542, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30876614, 'num_objects': 7540, 'num_object_clones': 0, 'num_object_copies': 15080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7540, 'num_write_kb': 30154, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3373}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:43.280 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:43.281 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'7891", 'reported_seq': 7981, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:39.918561+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:25:39.918561+0000', 'last_peered': '2024-09-16T10:25:39.918561+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:25:39.918561+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7891, 'log_dups_size': 0, 'ondisk_log_size': 7891, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32274316, 'num_objects': 7883, 'num_object_clones': 0, 'num_object_copies': 15766, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7888, 'num_write_kb': 31523, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3488}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:43.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:43.547 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 151 20 124965 124945 3.23152 1.75 0.0351439 0.0241683 2024-09-16T10:25:43.934 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:43.934 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/246976 objects degraded (19.082%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":232,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":123488,"data_bytes":505733641,"bytes_used":1684840448,"bytes_avail":288225452032,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":246976,"degraded_ratio":0.19082016066338428,"misplaced_objects":904,"misplaced_total":246976,"misplaced_ratio":0.0036602746825602482,"recovering_objects_per_sec":48,"recovering_bytes_per_sec":197536,"recovering_keys_per_sec":0,"num_objects_recovered":579,"num_bytes_recovered":2371584,"num_keys_recovered":0,"write_bytes_sec":1988672,"read_op_per_sec":0,"write_op_per_sec":485},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:43.965 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:43.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:44.545 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:44.545 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:44.547 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 152 20 125409 125389 3.22168 1.73438 0.0818324 0.024242 2024-09-16T10:25:44.565 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-09-16T10:25:42.607363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":505733641,"num_objects":123488,"num_object_clones":0,"num_object_copies":246976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":904,"num_objects_unfound":0,"num_objects_dirty":123488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101117,"num_write_kb":404350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3315,"num_bytes_recovered":13492726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123553,"ondisk_log_size":123553,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1645352,"kb_used_data":996816,"kb_used_omap":12,"kb_used_meta":648499,"kb_avail":281470168,"statfs":{"total":289910292480,"available":288225452032,"internally_reserved":0,"allocated":1020739584,"data_stored":1019941221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":664062978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,5,9,5,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23875584,"num_objects":5829,"num_object_clones":0,"num_object_copies":11658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-579,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5829,"num_write_kb":23316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":2371584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005789"},"pg_stats":[{"pgid":"2.f","version":"42'7741","reported_seq":7807,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.968386+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:39.968386+0000","last_peered":"2024-09-16T10:25:39.968386+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:39.968386+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7741,"log_dups_size":0,"ondisk_log_size":7741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31707136,"num_objects":7741,"num_object_clones":0,"num_object_copies":15482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7741,"num_write_kb":30964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3549},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7713","reported_seq":7867,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:40.775723+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:40.775723+0000","last_peered":"2024-09-16T10:25:40.775723+0000","last_clean":"2024-09-16T10:25:40.775723+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:40.775723+0000","last_undegraded":"2024-09-16T10:25:40.775723+0000","last_fullsized":"2024-09-16T10:25:40.775723+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31567872,"num_objects":7707,"num_object_clones":0,"num_object_copies":15414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3926,"num_write_kb":15704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7736","reported_seq":7919,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:40.777392+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:40.777392+0000","last_peered":"2024-09-16T10:25:40.777392+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:40.777392+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7736,"log_dups_size":0,"ondisk_log_size":7736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31670272,"num_objects":7732,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":15172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3397}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7744","reported_seq":7798,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:40.716893+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:40.716893+0000","last_peered":"2024-09-16T10:25:40.716893+0000","last_clean":"2024-09-16T10:25:40.716893+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:40.716893+0000","last_undegraded":"2024-09-16T10:25:40.716893+0000","last_fullsized":"2024-09-16T10:25:40.716893+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7744,"log_dups_size":0,"ondisk_log_size":7744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7744,"num_write_kb":30976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7698","reported_seq":7852,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:39.937594+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:39.937594+0000","last_peered":"2024-09-16T10:25:39.937594+0000","last_clean":"2024-09-16T10:25:39.937594+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:39.937594+0000","last_undegraded":"2024-09-16T10:25:39.937594+0000","last_fullsized":"2024-09-16T10:25:39.937594+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31502336,"num_objects":7691,"num_object_clones":0,"num_object_copies":15382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4340,"num_write_kb":17360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7602","reported_seq":7761,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.851911+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:39.851911+0000","last_peered":"2024-09-16T10:25:39.851911+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:39.851911+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31117312,"num_objects":7597,"num_object_clones":0,"num_object_copies":15194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4227,"num_write_kb":16908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3380},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'7932","reported_seq":8099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.912531+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:39.912531+0000","last_peered":"2024-09-16T10:25:39.912531+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:39.912531+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":7932,"log_dups_size":0,"ondisk_log_size":7932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32464896,"num_objects":7926,"num_object_clones":0,"num_object_copies":15852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4174,"num_write_kb":16696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3523},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7715","reported_seq":7887,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:39.980129+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:39.980129+0000","last_peered":"2024-09-16T10:25:39.980129+0000","last_clean":"2024-09-16T10:25:39.980129+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:39.980129+0000","last_undegraded":"2024-09-16T10:25:39.980129+0000","last_fullsized":"2024-09-16T10:25:39.980129+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7715,"log_dups_size":0,"ondisk_log_size":7715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31592448,"num_objects":7713,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3518,"num_write_kb":14072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7628","reported_seq":14312,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:40.770655+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:40.770655+0000","last_peered":"2024-09-16T10:25:40.770655+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:40.770655+0000","last_undegraded":"2024-09-16T10:25:40.770655+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7628,"log_dups_size":0,"ondisk_log_size":7628,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31216582,"num_objects":7623,"num_object_clones":0,"num_object_copies":15246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":904,"num_objects_unfound":0,"num_objects_dirty":7623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7626,"num_write_kb":30488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3297,"num_bytes_recovered":13488128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":904},{"shards":"2","objects":6719}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'7885","reported_seq":7960,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.945056+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:39.945056+0000","last_peered":"2024-09-16T10:25:39.945056+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:39.945056+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7885,"log_dups_size":0,"ondisk_log_size":7885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32265158,"num_objects":7879,"num_object_clones":0,"num_object_copies":15758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7883,"num_write_kb":31512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3445},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7657","reported_seq":7759,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:40.758728+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:40.758728+0000","last_peered":"2024-09-16T10:25:40.758728+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:40.758728+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7657,"log_dups_size":0,"ondisk_log_size":7657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31343558,"num_objects":7654,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7655,"num_write_kb":30611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7712","reported_seq":7796,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:40.699418+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:40.699418+0000","last_peered":"2024-09-16T10:25:40.699418+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:40.699418+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7712,"log_dups_size":0,"ondisk_log_size":7712,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31564742,"num_objects":7708,"num_object_clones":0,"num_object_copies":15416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7710,"num_write_kb":30827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7502","reported_seq":7579,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.937725+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:39.937725+0000","last_peered":"2024-09-16T10:25:39.937725+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:39.937725+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7502,"log_dups_size":0,"ondisk_log_size":7502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30720483,"num_objects":7501,"num_object_clones":0,"num_object_copies":15002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7501,"num_write_kb":30001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3342},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7852","reported_seq":7980,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.960274+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:39.960274+0000","last_peered":"2024-09-16T10:25:39.960274+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:39.960274+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7852,"log_dups_size":0,"ondisk_log_size":7852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32130473,"num_objects":7847,"num_object_clones":0,"num_object_copies":15694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":31380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3510},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7542","reported_seq":7620,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.963094+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:39.963094+0000","last_peered":"2024-09-16T10:25:39.963094+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:39.963094+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30876614,"num_objects":7540,"num_object_clones":0,"num_object_copies":15080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":30154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3373},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'7891","reported_seq":7981,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:39.918561+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:39.918561+0000","last_peered":"2024-09-16T10:25:39.918561+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:39.918561+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7891,"log_dups_size":0,"ondisk_log_size":7891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32274316,"num_objects":7883,"num_object_clones":0,"num_object_copies":15766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":31523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3488},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":505733622,"num_objects":123486,"num_object_clones":0,"num_object_copies":246972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":904,"num_objects_unfound":0,"num_objects_dirty":123486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101115,"num_write_kb":404348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3314,"num_bytes_recovered":13492707,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1021685760,"data_stored":1021602661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123550,"ondisk_log_size":123550,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018918,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613832,"kb_used_data":383680,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93758008,"statfs":{"total":96636764160,"available":96008200192,"internally_reserved":0,"allocated":392888320,"data_stored":392645081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":235663024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.317},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":5.907}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":1.6819999999999999},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.33800000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":318156,"kb_used_data":174148,"kb_used_omap":4,"kb_used_meta":143995,"kb_avail":94053684,"statfs":{"total":96636764160,"available":96310972416,"internally_reserved":0,"allocated":178327552,"data_stored":178012053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":147451562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":0.17799999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":5.673}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":4.4470000000000001},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":6.2910000000000004}]}]},{"osd":0,"up_from":24,"seq":103079215146,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713364,"kb_used_data":438988,"kb_used_omap":4,"kb_used_meta":274363,"kb_avail":93658476,"statfs":{"total":96636764160,"available":95906279424,"internally_reserved":0,"allocated":449523712,"data_stored":449284087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":280948392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.20100000000000001},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.2170000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.90100000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":449425408,"data_stored":449421795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":179347456,"data_stored":179271583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392912896,"data_stored":392909283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:45.547 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 153 20 125911 125891 3.21343 1.96094 0.0519218 0.0243018 2024-09-16T10:25:46.549 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 154 20 126427 126407 3.20562 2.01562 0.0683095 0.0243627 2024-09-16T10:25:47.549 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 155 20 126926 126906 3.19752 1.94922 0.0309774 0.0244246 2024-09-16T10:25:47.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:48.138 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:48.140 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:48.165 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-16T10:25:46.608312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":515240457,"num_objects":125809,"num_object_clones":0,"num_object_copies":251618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":661,"num_objects_unfound":0,"num_objects_dirty":125809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103438,"num_write_kb":413634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3558,"num_bytes_recovered":14488054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125874,"ondisk_log_size":125874,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1660512,"kb_used_data":1016072,"kb_used_omap":12,"kb_used_meta":644403,"kb_avail":281455008,"statfs":{"total":289910292480,"available":288209928192,"internally_reserved":0,"allocated":1040457728,"data_stored":1039659365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":659868674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,0,9,5,9,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29601792,"num_objects":7227,"num_object_clones":0,"num_object_copies":14454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-637,"num_objects_unfound":0,"num_objects_dirty":7227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7227,"num_write_kb":28908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":2609152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005748"},"pg_stats":[{"pgid":"2.f","version":"42'7872","reported_seq":7938,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.954392+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:44.954392+0000","last_peered":"2024-09-16T10:25:44.954392+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:44.954392+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32243712,"num_objects":7872,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7872,"num_write_kb":31488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3680},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7862","reported_seq":8015,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:45.741011+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:45.741011+0000","last_peered":"2024-09-16T10:25:45.741011+0000","last_clean":"2024-09-16T10:25:45.741011+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:45.741011+0000","last_undegraded":"2024-09-16T10:25:45.741011+0000","last_fullsized":"2024-09-16T10:25:45.741011+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":7856,"num_object_clones":0,"num_object_copies":15712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4075,"num_write_kb":16300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7859","reported_seq":8042,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.756878+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:45.756878+0000","last_peered":"2024-09-16T10:25:45.756878+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:45.756878+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7859,"log_dups_size":0,"ondisk_log_size":7859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32174080,"num_objects":7855,"num_object_clones":0,"num_object_copies":15710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3916,"num_write_kb":15664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7868","reported_seq":7922,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:45.681467+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:45.681467+0000","last_peered":"2024-09-16T10:25:45.681467+0000","last_clean":"2024-09-16T10:25:45.681467+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:45.681467+0000","last_undegraded":"2024-09-16T10:25:45.681467+0000","last_fullsized":"2024-09-16T10:25:45.681467+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7868,"log_dups_size":0,"ondisk_log_size":7868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32227328,"num_objects":7868,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7868,"num_write_kb":31472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7859","reported_seq":8013,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:45.018107+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:45.018107+0000","last_peered":"2024-09-16T10:25:45.018107+0000","last_clean":"2024-09-16T10:25:45.018107+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:45.018107+0000","last_undegraded":"2024-09-16T10:25:45.018107+0000","last_fullsized":"2024-09-16T10:25:45.018107+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7859,"log_dups_size":0,"ondisk_log_size":7859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32161792,"num_objects":7852,"num_object_clones":0,"num_object_copies":15704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4501,"num_write_kb":18004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7749","reported_seq":7910,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.898414+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:44.898414+0000","last_peered":"2024-09-16T10:25:44.898414+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:44.898414+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7749,"log_dups_size":0,"ondisk_log_size":7749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":17496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3527},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8071","reported_seq":8242,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.027910+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:45.027910+0000","last_peered":"2024-09-16T10:25:45.027910+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:45.027910+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8071,"log_dups_size":0,"ondisk_log_size":8071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33034240,"num_objects":8065,"num_object_clones":0,"num_object_copies":16130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4313,"num_write_kb":17252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3662},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7861","reported_seq":8033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:44.980924+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:44.980924+0000","last_peered":"2024-09-16T10:25:44.980924+0000","last_clean":"2024-09-16T10:25:44.980924+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:44.980924+0000","last_undegraded":"2024-09-16T10:25:44.980924+0000","last_fullsized":"2024-09-16T10:25:44.980924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7861,"log_dups_size":0,"ondisk_log_size":7861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32190464,"num_objects":7859,"num_object_clones":0,"num_object_copies":15718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":14656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7783","reported_seq":14953,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:45.787848+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:45.787848+0000","last_peered":"2024-09-16T10:25:45.787848+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:45.787848+0000","last_undegraded":"2024-09-16T10:25:45.787848+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31851462,"num_objects":7778,"num_object_clones":0,"num_object_copies":15556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":661,"num_objects_unfound":0,"num_objects_dirty":7778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7781,"num_write_kb":31108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3540,"num_bytes_recovered":14483456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":661},{"shards":"2","objects":7117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8054","reported_seq":8130,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.044621+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:45.044621+0000","last_peered":"2024-09-16T10:25:45.044621+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:45.044621+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8054,"log_dups_size":0,"ondisk_log_size":8054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32957382,"num_objects":8048,"num_object_clones":0,"num_object_copies":16096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8052,"num_write_kb":32188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3614},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7796","reported_seq":7899,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.668114+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:45.668114+0000","last_peered":"2024-09-16T10:25:45.668114+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:45.668114+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7796,"log_dups_size":0,"ondisk_log_size":7796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31912902,"num_objects":7793,"num_object_clones":0,"num_object_copies":15586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7794,"num_write_kb":31167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7866","reported_seq":7950,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.760756+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:45.760756+0000","last_peered":"2024-09-16T10:25:45.760756+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:45.760756+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32195526,"num_objects":7862,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7864,"num_write_kb":31443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7666","reported_seq":7740,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.052467+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:45.052467+0000","last_peered":"2024-09-16T10:25:45.052467+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:45.052467+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7666,"log_dups_size":0,"ondisk_log_size":7666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31392227,"num_objects":7665,"num_object_clones":0,"num_object_copies":15330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":30657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3506},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7975","reported_seq":8105,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.964861+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:44.964861+0000","last_peered":"2024-09-16T10:25:44.964861+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:44.964861+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32634281,"num_objects":7970,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7972,"num_write_kb":31872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3633},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7700","reported_seq":7778,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.933317+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:44.933317+0000","last_peered":"2024-09-16T10:25:44.933317+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:44.933317+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31523782,"num_objects":7698,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7698,"num_write_kb":30786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3531},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8030","reported_seq":8120,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.922574+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:44.922574+0000","last_peered":"2024-09-16T10:25:44.922574+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:44.922574+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32843660,"num_objects":8022,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":32079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3627},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":515240438,"num_objects":125807,"num_object_clones":0,"num_object_copies":251614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":661,"num_objects_unfound":0,"num_objects_dirty":125807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103436,"num_write_kb":413632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3557,"num_bytes_recovered":14488035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1041846272,"data_stored":1041763173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125871,"ondisk_log_size":125871,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018919,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622296,"kb_used_data":388048,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93749544,"statfs":{"total":96636764160,"available":95999533056,"internally_reserved":0,"allocated":397361152,"data_stored":397117913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":239857328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,1,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":3.0899999999999999},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":6.694}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":3.1349999999999998},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.8680000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691863,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330180,"kb_used_data":182076,"kb_used_omap":4,"kb_used_meta":148091,"kb_avail":94041660,"statfs":{"total":96636764160,"available":96298659840,"internally_reserved":0,"allocated":186445824,"data_stored":186130325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":151645866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":2.7509999999999999},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.66900000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.35299999999999998},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":0.32000000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215147,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708036,"kb_used_data":445948,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93663804,"statfs":{"total":96636764160,"available":95911735296,"internally_reserved":0,"allocated":456650752,"data_stored":456411127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":268365480},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,0,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":2.3479999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.6660000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":456638464,"data_stored":456634851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187555840,"data_stored":187479967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397651968,"data_stored":397648355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:48.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:48.549 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 156 20 127357 127337 3.18781 1.68359 0.110541 0.0244986 2024-09-16T10:25:48.716 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:48.718 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:48.749 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-16T10:25:48.608838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":515240457,"num_objects":125809,"num_object_clones":0,"num_object_copies":251618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":661,"num_objects_unfound":0,"num_objects_dirty":125809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103438,"num_write_kb":413634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3558,"num_bytes_recovered":14488054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125874,"ondisk_log_size":125874,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1679828,"kb_used_data":1025532,"kb_used_omap":12,"kb_used_meta":654259,"kb_avail":281435692,"statfs":{"total":289910292480,"available":288190148608,"internally_reserved":0,"allocated":1050144768,"data_stored":1049346405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":669961218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,0,6,2,5,17],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19472384,"num_objects":4754,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-451,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4754,"num_write_kb":19016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":1847296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003000"},"pg_stats":[{"pgid":"2.f","version":"42'7872","reported_seq":7938,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.954392+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:44.954392+0000","last_peered":"2024-09-16T10:25:44.954392+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:44.954392+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32243712,"num_objects":7872,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7872,"num_write_kb":31488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3680},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7862","reported_seq":8015,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:45.741011+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:45.741011+0000","last_peered":"2024-09-16T10:25:45.741011+0000","last_clean":"2024-09-16T10:25:45.741011+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:45.741011+0000","last_undegraded":"2024-09-16T10:25:45.741011+0000","last_fullsized":"2024-09-16T10:25:45.741011+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":7856,"num_object_clones":0,"num_object_copies":15712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4075,"num_write_kb":16300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7859","reported_seq":8042,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.756878+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:45.756878+0000","last_peered":"2024-09-16T10:25:45.756878+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:45.756878+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7859,"log_dups_size":0,"ondisk_log_size":7859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32174080,"num_objects":7855,"num_object_clones":0,"num_object_copies":15710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3916,"num_write_kb":15664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7868","reported_seq":7922,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:45.681467+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:45.681467+0000","last_peered":"2024-09-16T10:25:45.681467+0000","last_clean":"2024-09-16T10:25:45.681467+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:45.681467+0000","last_undegraded":"2024-09-16T10:25:45.681467+0000","last_fullsized":"2024-09-16T10:25:45.681467+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7868,"log_dups_size":0,"ondisk_log_size":7868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32227328,"num_objects":7868,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7868,"num_write_kb":31472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7859","reported_seq":8013,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:45.018107+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:45.018107+0000","last_peered":"2024-09-16T10:25:45.018107+0000","last_clean":"2024-09-16T10:25:45.018107+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:45.018107+0000","last_undegraded":"2024-09-16T10:25:45.018107+0000","last_fullsized":"2024-09-16T10:25:45.018107+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7859,"log_dups_size":0,"ondisk_log_size":7859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32161792,"num_objects":7852,"num_object_clones":0,"num_object_copies":15704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4501,"num_write_kb":18004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7749","reported_seq":7910,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.898414+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:44.898414+0000","last_peered":"2024-09-16T10:25:44.898414+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:44.898414+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7749,"log_dups_size":0,"ondisk_log_size":7749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":17496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3527},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8071","reported_seq":8242,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.027910+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:45.027910+0000","last_peered":"2024-09-16T10:25:45.027910+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:45.027910+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8071,"log_dups_size":0,"ondisk_log_size":8071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33034240,"num_objects":8065,"num_object_clones":0,"num_object_copies":16130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4313,"num_write_kb":17252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3662},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7861","reported_seq":8033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:44.980924+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:44.980924+0000","last_peered":"2024-09-16T10:25:44.980924+0000","last_clean":"2024-09-16T10:25:44.980924+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:44.980924+0000","last_undegraded":"2024-09-16T10:25:44.980924+0000","last_fullsized":"2024-09-16T10:25:44.980924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7861,"log_dups_size":0,"ondisk_log_size":7861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32190464,"num_objects":7859,"num_object_clones":0,"num_object_copies":15718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":14656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7783","reported_seq":14953,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:45.787848+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:45.787848+0000","last_peered":"2024-09-16T10:25:45.787848+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:45.787848+0000","last_undegraded":"2024-09-16T10:25:45.787848+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31851462,"num_objects":7778,"num_object_clones":0,"num_object_copies":15556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":661,"num_objects_unfound":0,"num_objects_dirty":7778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7781,"num_write_kb":31108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3540,"num_bytes_recovered":14483456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":661},{"shards":"2","objects":7117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8054","reported_seq":8130,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.044621+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:45.044621+0000","last_peered":"2024-09-16T10:25:45.044621+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:45.044621+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8054,"log_dups_size":0,"ondisk_log_size":8054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32957382,"num_objects":8048,"num_object_clones":0,"num_object_copies":16096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8052,"num_write_kb":32188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3614},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7796","reported_seq":7899,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.668114+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:45.668114+0000","last_peered":"2024-09-16T10:25:45.668114+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:45.668114+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7796,"log_dups_size":0,"ondisk_log_size":7796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31912902,"num_objects":7793,"num_object_clones":0,"num_object_copies":15586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7794,"num_write_kb":31167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7866","reported_seq":7950,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.760756+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:45.760756+0000","last_peered":"2024-09-16T10:25:45.760756+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:45.760756+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32195526,"num_objects":7862,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7864,"num_write_kb":31443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7666","reported_seq":7740,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.052467+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:45.052467+0000","last_peered":"2024-09-16T10:25:45.052467+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:45.052467+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7666,"log_dups_size":0,"ondisk_log_size":7666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31392227,"num_objects":7665,"num_object_clones":0,"num_object_copies":15330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":30657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3506},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7975","reported_seq":8105,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.964861+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:44.964861+0000","last_peered":"2024-09-16T10:25:44.964861+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:44.964861+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32634281,"num_objects":7970,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7972,"num_write_kb":31872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3633},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7700","reported_seq":7778,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.933317+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:44.933317+0000","last_peered":"2024-09-16T10:25:44.933317+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:44.933317+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31523782,"num_objects":7698,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7698,"num_write_kb":30786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3531},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8030","reported_seq":8120,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.922574+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:44.922574+0000","last_peered":"2024-09-16T10:25:44.922574+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:44.922574+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32843660,"num_objects":8022,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":32079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3627},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":515240438,"num_objects":125807,"num_object_clones":0,"num_object_copies":251614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":661,"num_objects_unfound":0,"num_objects_dirty":125807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103436,"num_write_kb":413632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3557,"num_bytes_recovered":14488035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1050222592,"data_stored":1050139493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125871,"ondisk_log_size":125871,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018919,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622296,"kb_used_data":388048,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93749544,"statfs":{"total":96636764160,"available":95999533056,"internally_reserved":0,"allocated":397361152,"data_stored":397117913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":239857328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,1,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":3.0899999999999999},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":6.694}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":3.1349999999999998},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.8680000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349496,"kb_used_data":191536,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94022344,"statfs":{"total":96636764160,"available":96278880256,"internally_reserved":0,"allocated":196132864,"data_stored":195817365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":161738410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":3.4089999999999998},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":5.1340000000000003},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":5.1959999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215147,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708036,"kb_used_data":445948,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93663804,"statfs":{"total":96636764160,"available":95911735296,"internally_reserved":0,"allocated":456650752,"data_stored":456411127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":268365480},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,0,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":2.3479999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.6660000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":456638464,"data_stored":456634851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195932160,"data_stored":195856287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397651968,"data_stored":397648355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:48.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:48.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'7872", 'reported_seq': 7938, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:44.954392+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:25:44.954392+0000', 'last_peered': '2024-09-16T10:25:44.954392+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:25:44.954392+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7872, 'log_dups_size': 0, 'ondisk_log_size': 7872, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32243712, 'num_objects': 7872, 'num_object_clones': 0, 'num_object_copies': 15744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7872, 'num_write_kb': 31488, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3680}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'7859", 'reported_seq': 8042, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:45.756878+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:45.756878+0000', 'last_peered': '2024-09-16T10:25:45.756878+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:45.756878+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 7859, 'log_dups_size': 0, 'ondisk_log_size': 7859, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32174080, 'num_objects': 7855, 'num_object_clones': 0, 'num_object_copies': 15710, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7855, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3916, 'num_write_kb': 15664, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 3520}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'7749", 'reported_seq': 7910, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:44.898414+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:25:44.898414+0000', 'last_peered': '2024-09-16T10:25:44.898414+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:25:44.898414+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 7749, 'log_dups_size': 0, 'ondisk_log_size': 7749, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31719424, 'num_objects': 7744, 'num_object_clones': 0, 'num_object_copies': 15488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4374, 'num_write_kb': 17496, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3527}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'8071", 'reported_seq': 8242, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:45.027910+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:25:45.027910+0000', 'last_peered': '2024-09-16T10:25:45.027910+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:25:45.027910+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 8071, 'log_dups_size': 0, 'ondisk_log_size': 8071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33034240, 'num_objects': 8065, 'num_object_clones': 0, 'num_object_copies': 16130, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4313, 'num_write_kb': 17252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3662}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'7783", 'reported_seq': 14953, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:45.787848+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:45.787848+0000', 'last_peered': '2024-09-16T10:25:45.787848+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:45.787848+0000', 'last_undegraded': '2024-09-16T10:25:45.787848+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7783, 'log_dups_size': 0, 'ondisk_log_size': 7783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31851462, 'num_objects': 7778, 'num_object_clones': 0, 'num_object_copies': 15556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 661, 'num_objects_unfound': 0, 'num_objects_dirty': 7778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7781, 'num_write_kb': 31108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3540, 'num_bytes_recovered': 14483456, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 661}, {'shards': '2', 'objects': 7117}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'8054", 'reported_seq': 8130, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:45.044621+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:25:45.044621+0000', 'last_peered': '2024-09-16T10:25:45.044621+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:25:45.044621+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8054, 'log_dups_size': 0, 'ondisk_log_size': 8054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32957382, 'num_objects': 8048, 'num_object_clones': 0, 'num_object_copies': 16096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8048, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8052, 'num_write_kb': 32188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3614}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:48.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:48.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'7796", 'reported_seq': 7899, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:45.668114+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:45.668114+0000', 'last_peered': '2024-09-16T10:25:45.668114+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:45.668114+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 7796, 'log_dups_size': 0, 'ondisk_log_size': 7796, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31912902, 'num_objects': 7793, 'num_object_clones': 0, 'num_object_copies': 15586, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7794, 'num_write_kb': 31167, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 3516}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:48.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:48.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'7866", 'reported_seq': 7950, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:45.760756+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:45.760756+0000', 'last_peered': '2024-09-16T10:25:45.760756+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:45.760756+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7866, 'log_dups_size': 0, 'ondisk_log_size': 7866, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32195526, 'num_objects': 7862, 'num_object_clones': 0, 'num_object_copies': 15724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7862, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7864, 'num_write_kb': 31443, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 3670}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:48.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:48.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'7666", 'reported_seq': 7740, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:45.052467+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:25:45.052467+0000', 'last_peered': '2024-09-16T10:25:45.052467+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:25:45.052467+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7666, 'log_dups_size': 0, 'ondisk_log_size': 7666, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31392227, 'num_objects': 7665, 'num_object_clones': 0, 'num_object_copies': 15330, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7665, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7665, 'num_write_kb': 30657, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3506}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:48.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:48.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'7975", 'reported_seq': 8105, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:44.964861+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:44.964861+0000', 'last_peered': '2024-09-16T10:25:44.964861+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:44.964861+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 7975, 'log_dups_size': 0, 'ondisk_log_size': 7975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32634281, 'num_objects': 7970, 'num_object_clones': 0, 'num_object_copies': 15940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7972, 'num_write_kb': 31872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3633}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:48.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:48.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'7700", 'reported_seq': 7778, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:44.933317+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:44.933317+0000', 'last_peered': '2024-09-16T10:25:44.933317+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:44.933317+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7700, 'log_dups_size': 0, 'ondisk_log_size': 7700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31523782, 'num_objects': 7698, 'num_object_clones': 0, 'num_object_copies': 15396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7698, 'num_write_kb': 30786, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3531}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:48.752 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:48.753 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'8030", 'reported_seq': 8120, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:44.922574+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:25:44.922574+0000', 'last_peered': '2024-09-16T10:25:44.922574+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:25:44.922574+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8030, 'log_dups_size': 0, 'ondisk_log_size': 8030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32843660, 'num_objects': 8022, 'num_object_clones': 0, 'num_object_copies': 16044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8022, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8027, 'num_write_kb': 32079, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3627}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:48.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:49.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:49.443 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/251618 objects degraded (18.730%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":238,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":125809,"data_bytes":515240457,"bytes_used":1720143872,"bytes_avail":288190148608,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":251618,"degraded_ratio":0.18729979572208666,"misplaced_objects":661,"misplaced_total":251618,"misplaced_ratio":0.0026269980685006637,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":153902,"recovering_keys_per_sec":0,"num_objects_recovered":451,"num_bytes_recovered":1847296,"num_keys_recovered":0,"write_bytes_sec":1622293,"read_op_per_sec":0,"write_op_per_sec":396},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:49.471 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:49.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:49.549 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 157 20 127803 127783 3.17861 1.74219 0.0500596 0.0245695 2024-09-16T10:25:50.033 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:50.034 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:50.060 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-16T10:25:48.608838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":515240457,"num_objects":125809,"num_object_clones":0,"num_object_copies":251618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":661,"num_objects_unfound":0,"num_objects_dirty":125809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103438,"num_write_kb":413634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3558,"num_bytes_recovered":14488054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125874,"ondisk_log_size":125874,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1679828,"kb_used_data":1025532,"kb_used_omap":12,"kb_used_meta":654259,"kb_avail":281435692,"statfs":{"total":289910292480,"available":288190148608,"internally_reserved":0,"allocated":1050144768,"data_stored":1049346405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":669961218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,0,6,2,5,17],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19472384,"num_objects":4754,"num_object_clones":0,"num_object_copies":9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-451,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4754,"num_write_kb":19016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":1847296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003000"},"pg_stats":[{"pgid":"2.f","version":"42'7872","reported_seq":7938,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.954392+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:44.954392+0000","last_peered":"2024-09-16T10:25:44.954392+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:44.954392+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32243712,"num_objects":7872,"num_object_clones":0,"num_object_copies":15744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7872,"num_write_kb":31488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3680},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7862","reported_seq":8015,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:45.741011+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:45.741011+0000","last_peered":"2024-09-16T10:25:45.741011+0000","last_clean":"2024-09-16T10:25:45.741011+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:45.741011+0000","last_undegraded":"2024-09-16T10:25:45.741011+0000","last_fullsized":"2024-09-16T10:25:45.741011+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7862,"log_dups_size":0,"ondisk_log_size":7862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":7856,"num_object_clones":0,"num_object_copies":15712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4075,"num_write_kb":16300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'7859","reported_seq":8042,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.756878+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:45.756878+0000","last_peered":"2024-09-16T10:25:45.756878+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:45.756878+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":7859,"log_dups_size":0,"ondisk_log_size":7859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32174080,"num_objects":7855,"num_object_clones":0,"num_object_copies":15710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3916,"num_write_kb":15664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'7868","reported_seq":7922,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:45.681467+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:45.681467+0000","last_peered":"2024-09-16T10:25:45.681467+0000","last_clean":"2024-09-16T10:25:45.681467+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:45.681467+0000","last_undegraded":"2024-09-16T10:25:45.681467+0000","last_fullsized":"2024-09-16T10:25:45.681467+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7868,"log_dups_size":0,"ondisk_log_size":7868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32227328,"num_objects":7868,"num_object_clones":0,"num_object_copies":15736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7868,"num_write_kb":31472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'7859","reported_seq":8013,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:45.018107+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:45.018107+0000","last_peered":"2024-09-16T10:25:45.018107+0000","last_clean":"2024-09-16T10:25:45.018107+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:45.018107+0000","last_undegraded":"2024-09-16T10:25:45.018107+0000","last_fullsized":"2024-09-16T10:25:45.018107+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":7859,"log_dups_size":0,"ondisk_log_size":7859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32161792,"num_objects":7852,"num_object_clones":0,"num_object_copies":15704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4501,"num_write_kb":18004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7749","reported_seq":7910,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.898414+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:44.898414+0000","last_peered":"2024-09-16T10:25:44.898414+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:44.898414+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7749,"log_dups_size":0,"ondisk_log_size":7749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4374,"num_write_kb":17496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3527},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8071","reported_seq":8242,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.027910+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:45.027910+0000","last_peered":"2024-09-16T10:25:45.027910+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:45.027910+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8071,"log_dups_size":0,"ondisk_log_size":8071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33034240,"num_objects":8065,"num_object_clones":0,"num_object_copies":16130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4313,"num_write_kb":17252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3662},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'7861","reported_seq":8033,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:44.980924+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:44.980924+0000","last_peered":"2024-09-16T10:25:44.980924+0000","last_clean":"2024-09-16T10:25:44.980924+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:44.980924+0000","last_undegraded":"2024-09-16T10:25:44.980924+0000","last_fullsized":"2024-09-16T10:25:44.980924+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":7861,"log_dups_size":0,"ondisk_log_size":7861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32190464,"num_objects":7859,"num_object_clones":0,"num_object_copies":15718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":14656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7783","reported_seq":14953,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:45.787848+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:45.787848+0000","last_peered":"2024-09-16T10:25:45.787848+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:45.787848+0000","last_undegraded":"2024-09-16T10:25:45.787848+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31851462,"num_objects":7778,"num_object_clones":0,"num_object_copies":15556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":661,"num_objects_unfound":0,"num_objects_dirty":7778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7781,"num_write_kb":31108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3540,"num_bytes_recovered":14483456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":661},{"shards":"2","objects":7117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8054","reported_seq":8130,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.044621+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:45.044621+0000","last_peered":"2024-09-16T10:25:45.044621+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:45.044621+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8054,"log_dups_size":0,"ondisk_log_size":8054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32957382,"num_objects":8048,"num_object_clones":0,"num_object_copies":16096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8052,"num_write_kb":32188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3614},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7796","reported_seq":7899,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.668114+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:45.668114+0000","last_peered":"2024-09-16T10:25:45.668114+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:45.668114+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7796,"log_dups_size":0,"ondisk_log_size":7796,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31912902,"num_objects":7793,"num_object_clones":0,"num_object_copies":15586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7794,"num_write_kb":31167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3516}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'7866","reported_seq":7950,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.760756+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:45.760756+0000","last_peered":"2024-09-16T10:25:45.760756+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:45.760756+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32195526,"num_objects":7862,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7864,"num_write_kb":31443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7666","reported_seq":7740,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:45.052467+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:45.052467+0000","last_peered":"2024-09-16T10:25:45.052467+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:45.052467+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7666,"log_dups_size":0,"ondisk_log_size":7666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31392227,"num_objects":7665,"num_object_clones":0,"num_object_copies":15330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7665,"num_write_kb":30657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3506},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'7975","reported_seq":8105,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.964861+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:44.964861+0000","last_peered":"2024-09-16T10:25:44.964861+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:44.964861+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32634281,"num_objects":7970,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7972,"num_write_kb":31872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3633},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7700","reported_seq":7778,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.933317+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:44.933317+0000","last_peered":"2024-09-16T10:25:44.933317+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:44.933317+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31523782,"num_objects":7698,"num_object_clones":0,"num_object_copies":15396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7698,"num_write_kb":30786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3531},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8030","reported_seq":8120,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:44.922574+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:44.922574+0000","last_peered":"2024-09-16T10:25:44.922574+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:44.922574+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8030,"log_dups_size":0,"ondisk_log_size":8030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32843660,"num_objects":8022,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8027,"num_write_kb":32079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3627},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":515240438,"num_objects":125807,"num_object_clones":0,"num_object_copies":251614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":661,"num_objects_unfound":0,"num_objects_dirty":125807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103436,"num_write_kb":413632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3557,"num_bytes_recovered":14488035,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1050222592,"data_stored":1050139493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125871,"ondisk_log_size":125871,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018919,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622296,"kb_used_data":388048,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93749544,"statfs":{"total":96636764160,"available":95999533056,"internally_reserved":0,"allocated":397361152,"data_stored":397117913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":239857328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,0,1,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":3.0899999999999999},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":6.694}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":3.1349999999999998},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.8680000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349496,"kb_used_data":191536,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94022344,"statfs":{"total":96636764160,"available":96278880256,"internally_reserved":0,"allocated":196132864,"data_stored":195817365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":161738410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":3.4089999999999998},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":5.1340000000000003},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":5.1959999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215147,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708036,"kb_used_data":445948,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93663804,"statfs":{"total":96636764160,"available":95911735296,"internally_reserved":0,"allocated":456650752,"data_stored":456411127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":268365480},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,0,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":2.3479999999999999},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.6660000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":456638464,"data_stored":456634851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195932160,"data_stored":195856287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397651968,"data_stored":397648355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:50.550 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 158 20 128269 128249 3.17001 1.82031 0.0202962 0.0246378 2024-09-16T10:25:51.550 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 159 20 128771 128751 3.16241 1.96094 0.0651486 0.0246959 2024-09-16T10:25:52.550 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:25:52.549296+0000 min lat: 0.00139595 max lat: 2.61294 avg lat: 0.0247549 2024-09-16T10:25:52.550 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:25:52.550 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 160 20 129275 129255 3.15495 1.96875 0.0267936 0.0247549 2024-09-16T10:25:53.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:53.550 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 161 20 129747 129727 3.1468 1.84375 0.0292708 0.0248172 2024-09-16T10:25:53.610 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:53.633 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-16T10:25:52.610187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":525042185,"num_objects":128202,"num_object_clones":0,"num_object_copies":256404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":441,"num_objects_unfound":0,"num_objects_dirty":128202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105831,"num_write_kb":423206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3778,"num_bytes_recovered":15389174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128267,"ondisk_log_size":128267,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1703712,"kb_used_data":1037124,"kb_used_omap":12,"kb_used_meta":666547,"kb_avail":281411808,"statfs":{"total":289910292480,"available":288165691392,"internally_reserved":0,"allocated":1062014976,"data_stored":1061216613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":682544130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,7,10,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22970368,"num_objects":5608,"num_object_clones":0,"num_object_copies":11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-671,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":22432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":671,"num_bytes_recovered":2748416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003346"},"pg_stats":[{"pgid":"2.f","version":"42'8017","reported_seq":8083,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.051430+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:50.051430+0000","last_peered":"2024-09-16T10:25:50.051430+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:50.051430+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32837632,"num_objects":8017,"num_object_clones":0,"num_object_copies":16034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":32068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3825},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7989","reported_seq":8143,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.689100+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:50.689100+0000","last_peered":"2024-09-16T10:25:50.689100+0000","last_clean":"2024-09-16T10:25:50.689100+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:50.689100+0000","last_undegraded":"2024-09-16T10:25:50.689100+0000","last_fullsized":"2024-09-16T10:25:50.689100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32698368,"num_objects":7983,"num_object_clones":0,"num_object_copies":15966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":16808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'8000","reported_seq":8182,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.715684+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:50.715684+0000","last_peered":"2024-09-16T10:25:50.715684+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:50.715684+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32751616,"num_objects":7996,"num_object_clones":0,"num_object_copies":15992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4057,"num_write_kb":16228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3661}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'8018","reported_seq":8072,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.763071+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:50.763071+0000","last_peered":"2024-09-16T10:25:50.763071+0000","last_clean":"2024-09-16T10:25:50.763071+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:50.763071+0000","last_undegraded":"2024-09-16T10:25:50.763071+0000","last_fullsized":"2024-09-16T10:25:50.763071+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32841728,"num_objects":8018,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8018,"num_write_kb":32072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'8007","reported_seq":8161,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.095866+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:50.095866+0000","last_peered":"2024-09-16T10:25:50.095866+0000","last_clean":"2024-09-16T10:25:50.095866+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:50.095866+0000","last_undegraded":"2024-09-16T10:25:50.095866+0000","last_fullsized":"2024-09-16T10:25:50.095866+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8007,"log_dups_size":0,"ondisk_log_size":8007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":8000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":18596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7896","reported_seq":8057,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.028566+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:50.028566+0000","last_peered":"2024-09-16T10:25:50.028566+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:50.028566+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32321536,"num_objects":7891,"num_object_clones":0,"num_object_copies":15782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4521,"num_write_kb":18084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3674},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8241","reported_seq":8410,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.050141+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:50.050141+0000","last_peered":"2024-09-16T10:25:50.050141+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:50.050141+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33730560,"num_objects":8235,"num_object_clones":0,"num_object_copies":16470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":17932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3832},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'8018","reported_seq":8190,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.028513+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:50.028513+0000","last_peered":"2024-09-16T10:25:50.028513+0000","last_clean":"2024-09-16T10:25:50.028513+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:50.028513+0000","last_undegraded":"2024-09-16T10:25:50.028513+0000","last_fullsized":"2024-09-16T10:25:50.028513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":8016,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3821,"num_write_kb":15284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7943","reported_seq":15553,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:50.787227+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:50.787227+0000","last_peered":"2024-09-16T10:25:50.787227+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:50.787227+0000","last_undegraded":"2024-09-16T10:25:50.787227+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32506822,"num_objects":7938,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":441,"num_objects_unfound":0,"num_objects_dirty":7938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":31748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3760,"num_bytes_recovered":15384576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":441},{"shards":"2","objects":7497}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8211","reported_seq":8286,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.071694+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:50.071694+0000","last_peered":"2024-09-16T10:25:50.071694+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:50.071694+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8211,"log_dups_size":0,"ondisk_log_size":8211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33600454,"num_objects":8205,"num_object_clones":0,"num_object_copies":16410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8209,"num_write_kb":32816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3771},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7943","reported_seq":8046,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.699925+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:50.699925+0000","last_peered":"2024-09-16T10:25:50.699925+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:50.699925+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32515014,"num_objects":7940,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":31755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3663}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'8016","reported_seq":8099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.789915+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:50.789915+0000","last_peered":"2024-09-16T10:25:50.789915+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:50.789915+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32809926,"num_objects":8012,"num_object_clones":0,"num_object_copies":16024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8014,"num_write_kb":32043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3820}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7827","reported_seq":7904,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.073422+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:50.073422+0000","last_peered":"2024-09-16T10:25:50.073422+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:50.073422+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7827,"log_dups_size":0,"ondisk_log_size":7827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32051683,"num_objects":7826,"num_object_clones":0,"num_object_copies":15652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7826,"num_write_kb":31301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3667},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'8127","reported_seq":8257,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.034671+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:50.034671+0000","last_peered":"2024-09-16T10:25:50.034671+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:50.034671+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33256873,"num_objects":8122,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":32480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3785},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7850","reported_seq":7927,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.059373+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:50.059373+0000","last_peered":"2024-09-16T10:25:50.059373+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:50.059373+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7850,"log_dups_size":0,"ondisk_log_size":7850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32138182,"num_objects":7848,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":31386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3681},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8161","reported_seq":8251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:49.907456+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:49.907456+0000","last_peered":"2024-09-16T10:25:49.907456+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:49.907456+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8161,"log_dups_size":0,"ondisk_log_size":8161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33380236,"num_objects":8153,"num_object_clones":0,"num_object_copies":16306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":32603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3758},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":525042166,"num_objects":128200,"num_object_clones":0,"num_object_copies":256400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":441,"num_objects_unfound":0,"num_objects_dirty":128200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105829,"num_write_kb":423204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3777,"num_bytes_recovered":15389155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1062285312,"data_stored":1062202213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128264,"ondisk_log_size":128264,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018920,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630940,"kb_used_data":392592,"kb_used_omap":4,"kb_used_meta":238331,"kb_avail":93740900,"statfs":{"total":96636764160,"available":95990681600,"internally_reserved":0,"allocated":402014208,"data_stored":401770969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":244051632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.84799999999999998},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":5.8179999999999996}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":10.492000000000001},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.51300000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349496,"kb_used_data":191536,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94022344,"statfs":{"total":96636764160,"available":96278880256,"internally_reserved":0,"allocated":196132864,"data_stored":195817365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":161738410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":3.4089999999999998},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":5.1340000000000003},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":5.1959999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723276,"kb_used_data":452996,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93648564,"statfs":{"total":96636764160,"available":95896129536,"internally_reserved":0,"allocated":463867904,"data_stored":463628279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":276754088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,8,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.218},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":5.96}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":5.4699999999999998},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":463949824,"data_stored":463946211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195932160,"data_stored":195856287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402403328,"data_stored":402399715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:53.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:54.198 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:54.201 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:54.223 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-16T10:25:52.610187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":525042185,"num_objects":128202,"num_object_clones":0,"num_object_copies":256404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":441,"num_objects_unfound":0,"num_objects_dirty":128202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105831,"num_write_kb":423206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3778,"num_bytes_recovered":15389174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128267,"ondisk_log_size":128267,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1703712,"kb_used_data":1037124,"kb_used_omap":12,"kb_used_meta":666547,"kb_avail":281411808,"statfs":{"total":289910292480,"available":288165691392,"internally_reserved":0,"allocated":1062014976,"data_stored":1061216613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":682544130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,7,10,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22970368,"num_objects":5608,"num_object_clones":0,"num_object_copies":11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-671,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5608,"num_write_kb":22432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":671,"num_bytes_recovered":2748416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003346"},"pg_stats":[{"pgid":"2.f","version":"42'8017","reported_seq":8083,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.051430+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:50.051430+0000","last_peered":"2024-09-16T10:25:50.051430+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:50.051430+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32837632,"num_objects":8017,"num_object_clones":0,"num_object_copies":16034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":32068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3825},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7989","reported_seq":8143,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.689100+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:50.689100+0000","last_peered":"2024-09-16T10:25:50.689100+0000","last_clean":"2024-09-16T10:25:50.689100+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:50.689100+0000","last_undegraded":"2024-09-16T10:25:50.689100+0000","last_fullsized":"2024-09-16T10:25:50.689100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32698368,"num_objects":7983,"num_object_clones":0,"num_object_copies":15966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":16808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'8000","reported_seq":8182,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.715684+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:50.715684+0000","last_peered":"2024-09-16T10:25:50.715684+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:50.715684+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32751616,"num_objects":7996,"num_object_clones":0,"num_object_copies":15992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4057,"num_write_kb":16228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3661}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'8018","reported_seq":8072,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.763071+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:50.763071+0000","last_peered":"2024-09-16T10:25:50.763071+0000","last_clean":"2024-09-16T10:25:50.763071+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:50.763071+0000","last_undegraded":"2024-09-16T10:25:50.763071+0000","last_fullsized":"2024-09-16T10:25:50.763071+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32841728,"num_objects":8018,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8018,"num_write_kb":32072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'8007","reported_seq":8161,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.095866+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:50.095866+0000","last_peered":"2024-09-16T10:25:50.095866+0000","last_clean":"2024-09-16T10:25:50.095866+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:50.095866+0000","last_undegraded":"2024-09-16T10:25:50.095866+0000","last_fullsized":"2024-09-16T10:25:50.095866+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8007,"log_dups_size":0,"ondisk_log_size":8007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":8000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":18596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7896","reported_seq":8057,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.028566+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:50.028566+0000","last_peered":"2024-09-16T10:25:50.028566+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:50.028566+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32321536,"num_objects":7891,"num_object_clones":0,"num_object_copies":15782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4521,"num_write_kb":18084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3674},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8241","reported_seq":8410,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.050141+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:50.050141+0000","last_peered":"2024-09-16T10:25:50.050141+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:50.050141+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33730560,"num_objects":8235,"num_object_clones":0,"num_object_copies":16470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":17932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3832},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'8018","reported_seq":8190,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.028513+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:50.028513+0000","last_peered":"2024-09-16T10:25:50.028513+0000","last_clean":"2024-09-16T10:25:50.028513+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:50.028513+0000","last_undegraded":"2024-09-16T10:25:50.028513+0000","last_fullsized":"2024-09-16T10:25:50.028513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":8016,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3821,"num_write_kb":15284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374107+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374107+0000","last_peered":"2024-09-16T10:23:56.374107+0000","last_clean":"2024-09-16T10:23:56.374107+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374107+0000","last_undegraded":"2024-09-16T10:23:56.374107+0000","last_fullsized":"2024-09-16T10:23:56.374107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7943","reported_seq":15553,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:50.787227+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:50.787227+0000","last_peered":"2024-09-16T10:25:50.787227+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:50.787227+0000","last_undegraded":"2024-09-16T10:25:50.787227+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32506822,"num_objects":7938,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":441,"num_objects_unfound":0,"num_objects_dirty":7938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":31748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3760,"num_bytes_recovered":15384576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":441},{"shards":"2","objects":7497}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8211","reported_seq":8286,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.071694+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:50.071694+0000","last_peered":"2024-09-16T10:25:50.071694+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:50.071694+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8211,"log_dups_size":0,"ondisk_log_size":8211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33600454,"num_objects":8205,"num_object_clones":0,"num_object_copies":16410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8209,"num_write_kb":32816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3771},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7943","reported_seq":8046,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.699925+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:50.699925+0000","last_peered":"2024-09-16T10:25:50.699925+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:50.699925+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32515014,"num_objects":7940,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":31755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3663}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'8016","reported_seq":8099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.789915+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:50.789915+0000","last_peered":"2024-09-16T10:25:50.789915+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:50.789915+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32809926,"num_objects":8012,"num_object_clones":0,"num_object_copies":16024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8014,"num_write_kb":32043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3820}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7827","reported_seq":7904,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.073422+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:50.073422+0000","last_peered":"2024-09-16T10:25:50.073422+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:50.073422+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7827,"log_dups_size":0,"ondisk_log_size":7827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32051683,"num_objects":7826,"num_object_clones":0,"num_object_copies":15652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7826,"num_write_kb":31301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3667},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'8127","reported_seq":8257,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.034671+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:50.034671+0000","last_peered":"2024-09-16T10:25:50.034671+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:50.034671+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33256873,"num_objects":8122,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":32480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3785},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7850","reported_seq":7927,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.059373+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:50.059373+0000","last_peered":"2024-09-16T10:25:50.059373+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:50.059373+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7850,"log_dups_size":0,"ondisk_log_size":7850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32138182,"num_objects":7848,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":31386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3681},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8161","reported_seq":8251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:49.907456+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:49.907456+0000","last_peered":"2024-09-16T10:25:49.907456+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:49.907456+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8161,"log_dups_size":0,"ondisk_log_size":8161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33380236,"num_objects":8153,"num_object_clones":0,"num_object_copies":16306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":32603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3758},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":525042166,"num_objects":128200,"num_object_clones":0,"num_object_copies":256400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":441,"num_objects_unfound":0,"num_objects_dirty":128200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105829,"num_write_kb":423204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3777,"num_bytes_recovered":15389155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1062285312,"data_stored":1062202213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128264,"ondisk_log_size":128264,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018920,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630940,"kb_used_data":392592,"kb_used_omap":4,"kb_used_meta":238331,"kb_avail":93740900,"statfs":{"total":96636764160,"available":95990681600,"internally_reserved":0,"allocated":402014208,"data_stored":401770969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":244051632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.84799999999999998},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":5.8179999999999996}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":10.492000000000001},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.51300000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691864,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":349496,"kb_used_data":191536,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94022344,"statfs":{"total":96636764160,"available":96278880256,"internally_reserved":0,"allocated":196132864,"data_stored":195817365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":161738410},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":3.4089999999999998},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":5.1340000000000003},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":5.1959999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723276,"kb_used_data":452996,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93648564,"statfs":{"total":96636764160,"available":95896129536,"internally_reserved":0,"allocated":463867904,"data_stored":463628279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":276754088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,8,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.218},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":5.96}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":5.4699999999999998},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":463949824,"data_stored":463946211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":195932160,"data_stored":195856287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402403328,"data_stored":402399715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:54.226 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:54.226 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'8017", 'reported_seq': 8083, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:50.051430+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:25:50.051430+0000', 'last_peered': '2024-09-16T10:25:50.051430+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:25:50.051430+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8017, 'log_dups_size': 0, 'ondisk_log_size': 8017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32837632, 'num_objects': 8017, 'num_object_clones': 0, 'num_object_copies': 16034, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8017, 'num_write_kb': 32068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3825}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:54.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:54.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'8000", 'reported_seq': 8182, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:50.715684+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:50.715684+0000', 'last_peered': '2024-09-16T10:25:50.715684+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:50.715684+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 8000, 'log_dups_size': 0, 'ondisk_log_size': 8000, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32751616, 'num_objects': 7996, 'num_object_clones': 0, 'num_object_copies': 15992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4057, 'num_write_kb': 16228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 3661}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:54.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:54.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'7896", 'reported_seq': 8057, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:50.028566+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:25:50.028566+0000', 'last_peered': '2024-09-16T10:25:50.028566+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:25:50.028566+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 7896, 'log_dups_size': 0, 'ondisk_log_size': 7896, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32321536, 'num_objects': 7891, 'num_object_clones': 0, 'num_object_copies': 15782, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4521, 'num_write_kb': 18084, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3674}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:54.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:54.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'8241", 'reported_seq': 8410, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:50.050141+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:25:50.050141+0000', 'last_peered': '2024-09-16T10:25:50.050141+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:25:50.050141+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 8241, 'log_dups_size': 0, 'ondisk_log_size': 8241, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33730560, 'num_objects': 8235, 'num_object_clones': 0, 'num_object_copies': 16470, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8235, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4483, 'num_write_kb': 17932, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3832}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:54.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:54.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'7943", 'reported_seq': 15553, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:50.787227+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:50.787227+0000', 'last_peered': '2024-09-16T10:25:50.787227+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:50.787227+0000', 'last_undegraded': '2024-09-16T10:25:50.787227+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7943, 'log_dups_size': 0, 'ondisk_log_size': 7943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32506822, 'num_objects': 7938, 'num_object_clones': 0, 'num_object_copies': 15876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 441, 'num_objects_unfound': 0, 'num_objects_dirty': 7938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7941, 'num_write_kb': 31748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3760, 'num_bytes_recovered': 15384576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 441}, {'shards': '2', 'objects': 7497}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:54.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:54.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'8211", 'reported_seq': 8286, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:50.071694+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:25:50.071694+0000', 'last_peered': '2024-09-16T10:25:50.071694+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:25:50.071694+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8211, 'log_dups_size': 0, 'ondisk_log_size': 8211, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33600454, 'num_objects': 8205, 'num_object_clones': 0, 'num_object_copies': 16410, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8209, 'num_write_kb': 32816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3771}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:54.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:54.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'7943", 'reported_seq': 8046, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:50.699925+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:50.699925+0000', 'last_peered': '2024-09-16T10:25:50.699925+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:50.699925+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 7943, 'log_dups_size': 0, 'ondisk_log_size': 7943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32515014, 'num_objects': 7940, 'num_object_clones': 0, 'num_object_copies': 15880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7941, 'num_write_kb': 31755, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 3663}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:54.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:54.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'8016", 'reported_seq': 8099, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:50.789915+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:50.789915+0000', 'last_peered': '2024-09-16T10:25:50.789915+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:50.789915+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8016, 'log_dups_size': 0, 'ondisk_log_size': 8016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32809926, 'num_objects': 8012, 'num_object_clones': 0, 'num_object_copies': 16024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8014, 'num_write_kb': 32043, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 3820}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:54.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:54.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'7827", 'reported_seq': 7904, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:50.073422+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:25:50.073422+0000', 'last_peered': '2024-09-16T10:25:50.073422+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:25:50.073422+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7827, 'log_dups_size': 0, 'ondisk_log_size': 7827, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32051683, 'num_objects': 7826, 'num_object_clones': 0, 'num_object_copies': 15652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7826, 'num_write_kb': 31301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3667}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:54.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:54.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'8127", 'reported_seq': 8257, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:50.034671+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:50.034671+0000', 'last_peered': '2024-09-16T10:25:50.034671+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:50.034671+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 8127, 'log_dups_size': 0, 'ondisk_log_size': 8127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33256873, 'num_objects': 8122, 'num_object_clones': 0, 'num_object_copies': 16244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8124, 'num_write_kb': 32480, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3785}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:54.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:54.229 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'7850", 'reported_seq': 7927, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:50.059373+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:50.059373+0000', 'last_peered': '2024-09-16T10:25:50.059373+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:50.059373+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7850, 'log_dups_size': 0, 'ondisk_log_size': 7850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32138182, 'num_objects': 7848, 'num_object_clones': 0, 'num_object_copies': 15696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7848, 'num_write_kb': 31386, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3681}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:54.230 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:54.230 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'8161", 'reported_seq': 8251, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:49.907456+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:25:49.907456+0000', 'last_peered': '2024-09-16T10:25:49.907456+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:25:49.907456+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8161, 'log_dups_size': 0, 'ondisk_log_size': 8161, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33380236, 'num_objects': 8153, 'num_object_clones': 0, 'num_object_copies': 16306, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8158, 'num_write_kb': 32603, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3758}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:54.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:25:54.550 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 162 20 130174 130154 3.13767 1.66797 0.125017 0.0248901 2024-09-16T10:25:54.888 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:54.889 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/256404 objects degraded (18.380%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":243,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":128202,"data_bytes":525042185,"bytes_used":1744601088,"bytes_avail":288165691392,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":256404,"degraded_ratio":0.18380368480990936,"misplaced_objects":441,"misplaced_total":256404,"misplaced_ratio":0.0017199419665839847,"recovering_objects_per_sec":55,"recovering_bytes_per_sec":228970,"recovering_keys_per_sec":0,"num_objects_recovered":671,"num_bytes_recovered":2748416,"num_keys_recovered":0,"write_bytes_sec":1913663,"read_op_per_sec":0,"write_op_per_sec":467},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:25:54.916 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:25:54.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:55.473 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:55.475 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:55.502 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-09-16T10:25:54.610697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":525042185,"num_objects":128202,"num_object_clones":0,"num_object_copies":256404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":441,"num_objects_unfound":0,"num_objects_dirty":128202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105831,"num_write_kb":423206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3778,"num_bytes_recovered":15389174,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128267,"ondisk_log_size":128267,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1715716,"kb_used_data":1045032,"kb_used_omap":12,"kb_used_meta":670643,"kb_avail":281399804,"statfs":{"total":289910292480,"available":288153399296,"internally_reserved":0,"allocated":1070112768,"data_stored":1069314405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":686738434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,10,13,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19308544,"num_objects":4714,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-463,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4714,"num_write_kb":18856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":1896448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003334"},"pg_stats":[{"pgid":"2.f","version":"42'8017","reported_seq":8083,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.051430+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:50.051430+0000","last_peered":"2024-09-16T10:25:50.051430+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:50.051430+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32837632,"num_objects":8017,"num_object_clones":0,"num_object_copies":16034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":32068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3825},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'7989","reported_seq":8143,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.689100+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:50.689100+0000","last_peered":"2024-09-16T10:25:50.689100+0000","last_clean":"2024-09-16T10:25:50.689100+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:50.689100+0000","last_undegraded":"2024-09-16T10:25:50.689100+0000","last_fullsized":"2024-09-16T10:25:50.689100+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32698368,"num_objects":7983,"num_object_clones":0,"num_object_copies":15966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":16808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'8000","reported_seq":8182,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.715684+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:50.715684+0000","last_peered":"2024-09-16T10:25:50.715684+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:50.715684+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32751616,"num_objects":7996,"num_object_clones":0,"num_object_copies":15992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4057,"num_write_kb":16228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3661}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'8018","reported_seq":8072,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.763071+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:50.763071+0000","last_peered":"2024-09-16T10:25:50.763071+0000","last_clean":"2024-09-16T10:25:50.763071+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:50.763071+0000","last_undegraded":"2024-09-16T10:25:50.763071+0000","last_fullsized":"2024-09-16T10:25:50.763071+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32841728,"num_objects":8018,"num_object_clones":0,"num_object_copies":16036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8018,"num_write_kb":32072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'8007","reported_seq":8161,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.095866+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:50.095866+0000","last_peered":"2024-09-16T10:25:50.095866+0000","last_clean":"2024-09-16T10:25:50.095866+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:50.095866+0000","last_undegraded":"2024-09-16T10:25:50.095866+0000","last_fullsized":"2024-09-16T10:25:50.095866+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8007,"log_dups_size":0,"ondisk_log_size":8007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32768000,"num_objects":8000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":18596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'7896","reported_seq":8057,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.028566+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:50.028566+0000","last_peered":"2024-09-16T10:25:50.028566+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:50.028566+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":7896,"log_dups_size":0,"ondisk_log_size":7896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32321536,"num_objects":7891,"num_object_clones":0,"num_object_copies":15782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4521,"num_write_kb":18084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3674},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8241","reported_seq":8410,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.050141+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:50.050141+0000","last_peered":"2024-09-16T10:25:50.050141+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:50.050141+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8241,"log_dups_size":0,"ondisk_log_size":8241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33730560,"num_objects":8235,"num_object_clones":0,"num_object_copies":16470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":17932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3832},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'8018","reported_seq":8190,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:50.028513+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:50.028513+0000","last_peered":"2024-09-16T10:25:50.028513+0000","last_clean":"2024-09-16T10:25:50.028513+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:50.028513+0000","last_undegraded":"2024-09-16T10:25:50.028513+0000","last_fullsized":"2024-09-16T10:25:50.028513+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":8016,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3821,"num_write_kb":15284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374859+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374859+0000","last_peered":"2024-09-16T10:23:56.374859+0000","last_clean":"2024-09-16T10:23:56.374859+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374859+0000","last_undegraded":"2024-09-16T10:23:56.374859+0000","last_fullsized":"2024-09-16T10:23:56.374859+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'7943","reported_seq":15553,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:50.787227+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:50.787227+0000","last_peered":"2024-09-16T10:25:50.787227+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:50.787227+0000","last_undegraded":"2024-09-16T10:25:50.787227+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32506822,"num_objects":7938,"num_object_clones":0,"num_object_copies":15876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":441,"num_objects_unfound":0,"num_objects_dirty":7938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":31748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3760,"num_bytes_recovered":15384576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":441},{"shards":"2","objects":7497}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8211","reported_seq":8286,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.071694+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:50.071694+0000","last_peered":"2024-09-16T10:25:50.071694+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:50.071694+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8211,"log_dups_size":0,"ondisk_log_size":8211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33600454,"num_objects":8205,"num_object_clones":0,"num_object_copies":16410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8209,"num_write_kb":32816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3771},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'7943","reported_seq":8046,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.699925+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:50.699925+0000","last_peered":"2024-09-16T10:25:50.699925+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:50.699925+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32515014,"num_objects":7940,"num_object_clones":0,"num_object_copies":15880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":31755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3663}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":65,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.435112+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.435112+0000","last_peered":"2024-09-16T10:23:56.435112+0000","last_clean":"2024-09-16T10:23:56.435112+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.435112+0000","last_undegraded":"2024-09-16T10:23:56.435112+0000","last_fullsized":"2024-09-16T10:23:56.435112+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'8016","reported_seq":8099,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.789915+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:50.789915+0000","last_peered":"2024-09-16T10:25:50.789915+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:50.789915+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8016,"log_dups_size":0,"ondisk_log_size":8016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32809926,"num_objects":8012,"num_object_clones":0,"num_object_copies":16024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8014,"num_write_kb":32043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3820}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7827","reported_seq":7904,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.073422+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:50.073422+0000","last_peered":"2024-09-16T10:25:50.073422+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:50.073422+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7827,"log_dups_size":0,"ondisk_log_size":7827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32051683,"num_objects":7826,"num_object_clones":0,"num_object_copies":15652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7826,"num_write_kb":31301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3667},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549863+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.549863+0000","last_peered":"2024-09-16T10:23:56.549863+0000","last_clean":"2024-09-16T10:23:56.549863+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.549863+0000","last_undegraded":"2024-09-16T10:23:56.549863+0000","last_fullsized":"2024-09-16T10:23:56.549863+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'8127","reported_seq":8257,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.034671+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:50.034671+0000","last_peered":"2024-09-16T10:25:50.034671+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:50.034671+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33256873,"num_objects":8122,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":32480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3785},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7850","reported_seq":7927,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:50.059373+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:50.059373+0000","last_peered":"2024-09-16T10:25:50.059373+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:50.059373+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7850,"log_dups_size":0,"ondisk_log_size":7850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32138182,"num_objects":7848,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7848,"num_write_kb":31386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3681},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":51,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.549308+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.549308+0000","last_peered":"2024-09-16T10:23:56.549308+0000","last_clean":"2024-09-16T10:23:56.549308+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.549308+0000","last_undegraded":"2024-09-16T10:23:56.549308+0000","last_fullsized":"2024-09-16T10:23:56.549308+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8161","reported_seq":8251,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:49.907456+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:49.907456+0000","last_peered":"2024-09-16T10:25:49.907456+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:49.907456+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8161,"log_dups_size":0,"ondisk_log_size":8161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33380236,"num_objects":8153,"num_object_clones":0,"num_object_copies":16306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8158,"num_write_kb":32603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3758},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":525042166,"num_objects":128200,"num_object_clones":0,"num_object_copies":256400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":441,"num_objects_unfound":0,"num_objects_dirty":128200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":105829,"num_write_kb":423204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3777,"num_bytes_recovered":15389155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1070579712,"data_stored":1070496613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128264,"ondisk_log_size":128264,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018920,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630940,"kb_used_data":392592,"kb_used_omap":4,"kb_used_meta":238331,"kb_avail":93740900,"statfs":{"total":96636764160,"available":95990681600,"internally_reserved":0,"allocated":402014208,"data_stored":401770969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":244051632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.84799999999999998},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":5.8179999999999996}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":10.492000000000001},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":0.51300000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":361500,"kb_used_data":199444,"kb_used_omap":4,"kb_used_meta":162043,"kb_avail":94010340,"statfs":{"total":96636764160,"available":96266588160,"internally_reserved":0,"allocated":204230656,"data_stored":203915157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":165932714},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":6.6879999999999997},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":6.7169999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":6.2199999999999998},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":5.0049999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723276,"kb_used_data":452996,"kb_used_omap":4,"kb_used_meta":270267,"kb_avail":93648564,"statfs":{"total":96636764160,"available":95896129536,"internally_reserved":0,"allocated":463867904,"data_stored":463628279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":276754088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,8,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":0.218},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":5.96}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":5.4699999999999998},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":463949824,"data_stored":463946211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204226560,"data_stored":204150687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402403328,"data_stored":402399715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:55.550 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 163 20 130620 130600 3.12911 1.74219 0.124563 0.0249589 2024-09-16T10:25:56.550 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 164 20 131126 131106 3.12208 1.97656 0.0291798 0.0250157 2024-09-16T10:25:57.550 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 165 20 131624 131604 3.11495 1.94531 0.0355915 0.0250723 2024-09-16T10:25:58.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:58.551 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 166 20 132097 132077 3.1073 1.84766 0.0174783 0.025133 2024-09-16T10:25:59.062 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:59.063 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:59.085 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-16T10:25:58.612458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534737417,"num_objects":130569,"num_object_clones":0,"num_object_copies":261138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":204,"num_objects_unfound":0,"num_objects_dirty":130569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108198,"num_write_kb":432674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4015,"num_bytes_recovered":16359926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130634,"ondisk_log_size":130634,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1771512,"kb_used_data":1064736,"kb_used_omap":12,"kb_used_meta":706739,"kb_avail":281344008,"statfs":{"total":289910292480,"available":288096264192,"internally_reserved":0,"allocated":1090289664,"data_stored":1089491301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":723700738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,4,3,9,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19496960,"num_objects":4760,"num_object_clones":0,"num_object_copies":9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-457,"num_objects_unfound":0,"num_objects_dirty":4760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4760,"num_write_kb":19040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":1871872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004145"},"pg_stats":[{"pgid":"2.f","version":"42'8160","reported_seq":8226,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.092538+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:55.092538+0000","last_peered":"2024-09-16T10:25:55.092538+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:55.092538+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33423360,"num_objects":8160,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":32640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3968},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'8151","reported_seq":8305,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.793121+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:55.793121+0000","last_peered":"2024-09-16T10:25:55.793121+0000","last_clean":"2024-09-16T10:25:55.793121+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:55.793121+0000","last_undegraded":"2024-09-16T10:25:55.793121+0000","last_fullsized":"2024-09-16T10:25:55.793121+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8151,"log_dups_size":0,"ondisk_log_size":8151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33361920,"num_objects":8145,"num_object_clones":0,"num_object_copies":16290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4364,"num_write_kb":17456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'8164","reported_seq":8347,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.725095+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:55.725095+0000","last_peered":"2024-09-16T10:25:55.725095+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:55.725095+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33423360,"num_objects":8160,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":16884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'8178","reported_seq":8232,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.793795+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:55.793795+0000","last_peered":"2024-09-16T10:25:55.793795+0000","last_clean":"2024-09-16T10:25:55.793795+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:55.793795+0000","last_undegraded":"2024-09-16T10:25:55.793795+0000","last_fullsized":"2024-09-16T10:25:55.793795+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8178,"log_dups_size":0,"ondisk_log_size":8178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33497088,"num_objects":8178,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8178,"num_write_kb":32712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'8149","reported_seq":8303,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.110547+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:55.110547+0000","last_peered":"2024-09-16T10:25:55.110547+0000","last_clean":"2024-09-16T10:25:55.110547+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:55.110547+0000","last_undegraded":"2024-09-16T10:25:55.110547+0000","last_fullsized":"2024-09-16T10:25:55.110547+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8149,"log_dups_size":0,"ondisk_log_size":8149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33349632,"num_objects":8142,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4791,"num_write_kb":19164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'8039","reported_seq":8200,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.052537+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:55.052537+0000","last_peered":"2024-09-16T10:25:55.052537+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:55.052537+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8039,"log_dups_size":0,"ondisk_log_size":8039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32907264,"num_objects":8034,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":18656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3817},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8396","reported_seq":8567,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.090595+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:55.090595+0000","last_peered":"2024-09-16T10:25:55.090595+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:55.090595+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8396,"log_dups_size":0,"ondisk_log_size":8396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34365440,"num_objects":8390,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":18552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3987},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'8176","reported_seq":8348,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.079131+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:55.079131+0000","last_peered":"2024-09-16T10:25:55.079131+0000","last_clean":"2024-09-16T10:25:55.079131+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:55.079131+0000","last_undegraded":"2024-09-16T10:25:55.079131+0000","last_fullsized":"2024-09-16T10:25:55.079131+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8176,"log_dups_size":0,"ondisk_log_size":8176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":8174,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3979,"num_write_kb":15916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374859+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374859+0000","last_peered":"2024-09-16T10:23:56.374859+0000","last_clean":"2024-09-16T10:23:56.374859+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374859+0000","last_undegraded":"2024-09-16T10:23:56.374859+0000","last_fullsized":"2024-09-16T10:23:56.374859+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'8096","reported_seq":16180,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:55.799332+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:55.799332+0000","last_peered":"2024-09-16T10:25:55.799332+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:55.799332+0000","last_undegraded":"2024-09-16T10:25:55.799332+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33133510,"num_objects":8091,"num_object_clones":0,"num_object_copies":16182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":204,"num_objects_unfound":0,"num_objects_dirty":8091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":32360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3997,"num_bytes_recovered":16355328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":204},{"shards":"2","objects":7887}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8348","reported_seq":8423,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.063259+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:55.063259+0000","last_peered":"2024-09-16T10:25:55.063259+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:55.063259+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8348,"log_dups_size":0,"ondisk_log_size":8348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34161606,"num_objects":8342,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8346,"num_write_kb":33364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3908},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'8080","reported_seq":8183,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.674858+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:55.674858+0000","last_peered":"2024-09-16T10:25:55.674858+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:55.674858+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8080,"log_dups_size":0,"ondisk_log_size":8080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076166,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8078,"num_write_kb":32303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3800}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":66,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.771942+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.771942+0000","last_peered":"2024-09-16T10:23:56.771942+0000","last_clean":"2024-09-16T10:23:56.771942+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.771942+0000","last_undegraded":"2024-09-16T10:23:56.771942+0000","last_fullsized":"2024-09-16T10:23:56.771942+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'8150","reported_seq":8234,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.758605+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:55.758605+0000","last_peered":"2024-09-16T10:25:55.758605+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:55.758605+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8150,"log_dups_size":0,"ondisk_log_size":8150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33358790,"num_objects":8146,"num_object_clones":0,"num_object_copies":16292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8148,"num_write_kb":32579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7976","reported_seq":8053,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.084974+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:55.084974+0000","last_peered":"2024-09-16T10:25:55.084974+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:55.084974+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7976,"log_dups_size":0,"ondisk_log_size":7976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32661987,"num_objects":7975,"num_object_clones":0,"num_object_copies":15950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7975,"num_write_kb":31897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3816},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.893933+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.893933+0000","last_peered":"2024-09-16T10:23:56.893933+0000","last_clean":"2024-09-16T10:23:56.893933+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.893933+0000","last_undegraded":"2024-09-16T10:23:56.893933+0000","last_fullsized":"2024-09-16T10:23:56.893933+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'8282","reported_seq":8412,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.086076+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:55.086076+0000","last_peered":"2024-09-16T10:25:55.086076+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:55.086076+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33891753,"num_objects":8277,"num_object_clones":0,"num_object_copies":16554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8279,"num_write_kb":33100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3940},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7987","reported_seq":8065,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.117141+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:55.117141+0000","last_peered":"2024-09-16T10:25:55.117141+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:55.117141+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699334,"num_objects":7985,"num_object_clones":0,"num_object_copies":15970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7985,"num_write_kb":31934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3818},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896307+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.896307+0000","last_peered":"2024-09-16T10:23:56.896307+0000","last_clean":"2024-09-16T10:23:56.896307+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.896307+0000","last_undegraded":"2024-09-16T10:23:56.896307+0000","last_fullsized":"2024-09-16T10:23:56.896307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8299","reported_seq":8389,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.026098+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:55.026098+0000","last_peered":"2024-09-16T10:25:55.026098+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:55.026098+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8299,"log_dups_size":0,"ondisk_log_size":8299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33945484,"num_objects":8291,"num_object_clones":0,"num_object_copies":16582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8296,"num_write_kb":33155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3896},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":534737398,"num_objects":130567,"num_object_clones":0,"num_object_copies":261134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":204,"num_objects_unfound":0,"num_objects_dirty":130567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108196,"num_write_kb":432672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4014,"num_bytes_recovered":16359907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1090949120,"data_stored":1090866021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130631,"ondisk_log_size":130631,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018921,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623544,"kb_used_data":397488,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93748296,"statfs":{"total":96636764160,"available":95998255104,"internally_reserved":0,"allocated":407027712,"data_stored":406784473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":231468720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.878},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":1.6619999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":1.5589999999999999},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.022}]}]},{"osd":1,"up_from":40,"seq":171798691866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395532,"kb_used_data":207236,"kb_used_omap":4,"kb_used_meta":188283,"kb_avail":93976308,"statfs":{"total":96636764160,"available":96231739392,"internally_reserved":0,"allocated":212209664,"data_stored":211894165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":192802474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":2.8860000000000001},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.442}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":0.627}]}]},{"osd":0,"up_from":24,"seq":103079215149,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752436,"kb_used_data":460012,"kb_used_omap":4,"kb_used_meta":292411,"kb_avail":93619404,"statfs":{"total":96636764160,"available":95866269696,"internally_reserved":0,"allocated":471052288,"data_stored":470812663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":299429544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,0,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":6.0659999999999998},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.0030000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":4.4340000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.226}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471199744,"data_stored":471196131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212385792,"data_stored":212309919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407363584,"data_stored":407359971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:59.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:25:59.551 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 167 20 132540 132520 3.09905 1.73047 0.0420255 0.0252007 2024-09-16T10:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:25:59.680 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:25:59.705 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-16T10:25:58.612458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534737417,"num_objects":130569,"num_object_clones":0,"num_object_copies":261138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":204,"num_objects_unfound":0,"num_objects_dirty":130569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108198,"num_write_kb":432674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4015,"num_bytes_recovered":16359926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130634,"ondisk_log_size":130634,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1771512,"kb_used_data":1064736,"kb_used_omap":12,"kb_used_meta":706739,"kb_avail":281344008,"statfs":{"total":289910292480,"available":288096264192,"internally_reserved":0,"allocated":1090289664,"data_stored":1089491301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":723700738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,8,4,3,9,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19496960,"num_objects":4760,"num_object_clones":0,"num_object_copies":9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-457,"num_objects_unfound":0,"num_objects_dirty":4760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4760,"num_write_kb":19040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":457,"num_bytes_recovered":1871872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004145"},"pg_stats":[{"pgid":"2.f","version":"42'8160","reported_seq":8226,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.092538+0000","last_change":"2024-09-16T10:23:56.901468+0000","last_active":"2024-09-16T10:25:55.092538+0000","last_peered":"2024-09-16T10:25:55.092538+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:25:55.092538+0000","last_undegraded":"2024-09-16T10:23:56.485426+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33423360,"num_objects":8160,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":32640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3968},{"shards":"0,2","objects":4192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'8151","reported_seq":8305,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.793121+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:55.793121+0000","last_peered":"2024-09-16T10:25:55.793121+0000","last_clean":"2024-09-16T10:25:55.793121+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:55.793121+0000","last_undegraded":"2024-09-16T10:25:55.793121+0000","last_fullsized":"2024-09-16T10:25:55.793121+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8151,"log_dups_size":0,"ondisk_log_size":8151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33361920,"num_objects":8145,"num_object_clones":0,"num_object_copies":16290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4364,"num_write_kb":17456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'8164","reported_seq":8347,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.725095+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:55.725095+0000","last_peered":"2024-09-16T10:25:55.725095+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:55.725095+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33423360,"num_objects":8160,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":16884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'8178","reported_seq":8232,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.793795+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:55.793795+0000","last_peered":"2024-09-16T10:25:55.793795+0000","last_clean":"2024-09-16T10:25:55.793795+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:55.793795+0000","last_undegraded":"2024-09-16T10:25:55.793795+0000","last_fullsized":"2024-09-16T10:25:55.793795+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8178,"log_dups_size":0,"ondisk_log_size":8178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33497088,"num_objects":8178,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8178,"num_write_kb":32712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'8149","reported_seq":8303,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.110547+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:25:55.110547+0000","last_peered":"2024-09-16T10:25:55.110547+0000","last_clean":"2024-09-16T10:25:55.110547+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:25:55.110547+0000","last_undegraded":"2024-09-16T10:25:55.110547+0000","last_fullsized":"2024-09-16T10:25:55.110547+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8149,"log_dups_size":0,"ondisk_log_size":8149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33349632,"num_objects":8142,"num_object_clones":0,"num_object_copies":16284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4791,"num_write_kb":19164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'8039","reported_seq":8200,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.052537+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:25:55.052537+0000","last_peered":"2024-09-16T10:25:55.052537+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:25:55.052537+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8039,"log_dups_size":0,"ondisk_log_size":8039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32907264,"num_objects":8034,"num_object_clones":0,"num_object_copies":16068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":18656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3817},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8396","reported_seq":8567,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.090595+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:25:55.090595+0000","last_peered":"2024-09-16T10:25:55.090595+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:25:55.090595+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8396,"log_dups_size":0,"ondisk_log_size":8396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34365440,"num_objects":8390,"num_object_clones":0,"num_object_copies":16780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4638,"num_write_kb":18552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3987},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'8176","reported_seq":8348,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.079131+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:25:55.079131+0000","last_peered":"2024-09-16T10:25:55.079131+0000","last_clean":"2024-09-16T10:25:55.079131+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:25:55.079131+0000","last_undegraded":"2024-09-16T10:25:55.079131+0000","last_fullsized":"2024-09-16T10:25:55.079131+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8176,"log_dups_size":0,"ondisk_log_size":8176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33480704,"num_objects":8174,"num_object_clones":0,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3979,"num_write_kb":15916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374859+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374859+0000","last_peered":"2024-09-16T10:23:56.374859+0000","last_clean":"2024-09-16T10:23:56.374859+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374859+0000","last_undegraded":"2024-09-16T10:23:56.374859+0000","last_fullsized":"2024-09-16T10:23:56.374859+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'8096","reported_seq":16180,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:55.799332+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:55.799332+0000","last_peered":"2024-09-16T10:25:55.799332+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:55.799332+0000","last_undegraded":"2024-09-16T10:25:55.799332+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33133510,"num_objects":8091,"num_object_clones":0,"num_object_copies":16182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":204,"num_objects_unfound":0,"num_objects_dirty":8091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":32360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3997,"num_bytes_recovered":16355328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":204},{"shards":"2","objects":7887}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8348","reported_seq":8423,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.063259+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:25:55.063259+0000","last_peered":"2024-09-16T10:25:55.063259+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:25:55.063259+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8348,"log_dups_size":0,"ondisk_log_size":8348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34161606,"num_objects":8342,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8346,"num_write_kb":33364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3908},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'8080","reported_seq":8183,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.674858+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:55.674858+0000","last_peered":"2024-09-16T10:25:55.674858+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:55.674858+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8080,"log_dups_size":0,"ondisk_log_size":8080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076166,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8078,"num_write_kb":32303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3800}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":66,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.771942+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.771942+0000","last_peered":"2024-09-16T10:23:56.771942+0000","last_clean":"2024-09-16T10:23:56.771942+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.771942+0000","last_undegraded":"2024-09-16T10:23:56.771942+0000","last_fullsized":"2024-09-16T10:23:56.771942+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'8150","reported_seq":8234,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.758605+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:55.758605+0000","last_peered":"2024-09-16T10:25:55.758605+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:55.758605+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8150,"log_dups_size":0,"ondisk_log_size":8150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33358790,"num_objects":8146,"num_object_clones":0,"num_object_copies":16292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8148,"num_write_kb":32579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'7976","reported_seq":8053,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.084974+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:25:55.084974+0000","last_peered":"2024-09-16T10:25:55.084974+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:25:55.084974+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7976,"log_dups_size":0,"ondisk_log_size":7976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32661987,"num_objects":7975,"num_object_clones":0,"num_object_copies":15950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7975,"num_write_kb":31897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3816},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.893933+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.893933+0000","last_peered":"2024-09-16T10:23:56.893933+0000","last_clean":"2024-09-16T10:23:56.893933+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.893933+0000","last_undegraded":"2024-09-16T10:23:56.893933+0000","last_fullsized":"2024-09-16T10:23:56.893933+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'8282","reported_seq":8412,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.086076+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:55.086076+0000","last_peered":"2024-09-16T10:25:55.086076+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:55.086076+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33891753,"num_objects":8277,"num_object_clones":0,"num_object_copies":16554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8279,"num_write_kb":33100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3940},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'7987","reported_seq":8065,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.117141+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:55.117141+0000","last_peered":"2024-09-16T10:25:55.117141+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:55.117141+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699334,"num_objects":7985,"num_object_clones":0,"num_object_copies":15970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7985,"num_write_kb":31934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3818},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896307+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.896307+0000","last_peered":"2024-09-16T10:23:56.896307+0000","last_clean":"2024-09-16T10:23:56.896307+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.896307+0000","last_undegraded":"2024-09-16T10:23:56.896307+0000","last_fullsized":"2024-09-16T10:23:56.896307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8299","reported_seq":8389,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.026098+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:25:55.026098+0000","last_peered":"2024-09-16T10:25:55.026098+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:25:55.026098+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8299,"log_dups_size":0,"ondisk_log_size":8299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33945484,"num_objects":8291,"num_object_clones":0,"num_object_copies":16582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8296,"num_write_kb":33155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3896},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":534737398,"num_objects":130567,"num_object_clones":0,"num_object_copies":261134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47128,"num_objects_misplaced":204,"num_objects_unfound":0,"num_objects_dirty":130567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108196,"num_write_kb":432672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4014,"num_bytes_recovered":16359907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1090949120,"data_stored":1090866021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130631,"ondisk_log_size":130631,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018921,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623544,"kb_used_data":397488,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93748296,"statfs":{"total":96636764160,"available":95998255104,"internally_reserved":0,"allocated":407027712,"data_stored":406784473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":231468720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.878},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":1.6619999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":1.5589999999999999},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.022}]}]},{"osd":1,"up_from":40,"seq":171798691866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395532,"kb_used_data":207236,"kb_used_omap":4,"kb_used_meta":188283,"kb_avail":93976308,"statfs":{"total":96636764160,"available":96231739392,"internally_reserved":0,"allocated":212209664,"data_stored":211894165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":192802474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":2.8860000000000001},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.442}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":0.627}]}]},{"osd":0,"up_from":24,"seq":103079215149,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752436,"kb_used_data":460012,"kb_used_omap":4,"kb_used_meta":292411,"kb_avail":93619404,"statfs":{"total":96636764160,"available":95866269696,"internally_reserved":0,"allocated":471052288,"data_stored":470812663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":299429544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,0,1,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:24:55 2024","interfaces":[{"interface":"back","average":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":10.272,"5min":10.272,"15min":10.272},"last":6.0659999999999998},{"interface":"front","average":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":2.0030000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":4.4340000000000002},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":2.226}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":471199744,"data_stored":471196131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212385792,"data_stored":212309919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407363584,"data_stored":407359971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:25:59.706 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:25:59.706 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'8160", 'reported_seq': 8226, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.092538+0000', 'last_change': '2024-09-16T10:23:56.901468+0000', 'last_active': '2024-09-16T10:25:55.092538+0000', 'last_peered': '2024-09-16T10:25:55.092538+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:25:55.092538+0000', 'last_undegraded': '2024-09-16T10:23:56.485426+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8160, 'log_dups_size': 0, 'ondisk_log_size': 8160, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33423360, 'num_objects': 8160, 'num_object_clones': 0, 'num_object_copies': 16320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4192, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8160, 'num_write_kb': 32640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3968}, {'shards': '0,2', 'objects': 4192}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:59.706 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:25:59.706 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'8164", 'reported_seq': 8347, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.725095+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:25:55.725095+0000', 'last_peered': '2024-09-16T10:25:55.725095+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:25:55.725095+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 8164, 'log_dups_size': 0, 'ondisk_log_size': 8164, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33423360, 'num_objects': 8160, 'num_object_clones': 0, 'num_object_copies': 16320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8160, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4221, 'num_write_kb': 16884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 3825}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:59.706 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:25:59.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'8039", 'reported_seq': 8200, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.052537+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:25:55.052537+0000', 'last_peered': '2024-09-16T10:25:55.052537+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:25:55.052537+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 8039, 'log_dups_size': 0, 'ondisk_log_size': 8039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32907264, 'num_objects': 8034, 'num_object_clones': 0, 'num_object_copies': 16068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8034, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4664, 'num_write_kb': 18656, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3817}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:59.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:25:59.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'8396", 'reported_seq': 8567, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.090595+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:25:55.090595+0000', 'last_peered': '2024-09-16T10:25:55.090595+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:25:55.090595+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 8396, 'log_dups_size': 0, 'ondisk_log_size': 8396, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34365440, 'num_objects': 8390, 'num_object_clones': 0, 'num_object_copies': 16780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4638, 'num_write_kb': 18552, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3987}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:59.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.3 is not active+clean 2024-09-16T10:25:59.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.3', 'version': "42'8096", 'reported_seq': 16180, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:25:55.799332+0000', 'last_change': '2024-09-16T10:23:56.821675+0000', 'last_active': '2024-09-16T10:25:55.799332+0000', 'last_peered': '2024-09-16T10:25:55.799332+0000', 'last_clean': '2024-09-16T10:23:54.345012+0000', 'last_became_active': '2024-09-16T10:23:56.776179+0000', 'last_became_peered': '2024-09-16T10:23:56.776179+0000', 'last_unstale': '2024-09-16T10:25:55.799332+0000', 'last_undegraded': '2024-09-16T10:25:55.799332+0000', 'last_fullsized': '2024-09-16T10:23:56.365307+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8096, 'log_dups_size': 0, 'ondisk_log_size': 8096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33133510, 'num_objects': 8091, 'num_object_clones': 0, 'num_object_copies': 16182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 204, 'num_objects_unfound': 0, 'num_objects_dirty': 8091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8094, 'num_write_kb': 32360, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3997, 'num_bytes_recovered': 16355328, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 204}, {'shards': '2', 'objects': 7887}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:59.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:25:59.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'8348", 'reported_seq': 8423, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.063259+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:25:55.063259+0000', 'last_peered': '2024-09-16T10:25:55.063259+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:25:55.063259+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8348, 'log_dups_size': 0, 'ondisk_log_size': 8348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34161606, 'num_objects': 8342, 'num_object_clones': 0, 'num_object_copies': 16684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8346, 'num_write_kb': 33364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3908}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:59.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:25:59.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'8080", 'reported_seq': 8183, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.674858+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:25:55.674858+0000', 'last_peered': '2024-09-16T10:25:55.674858+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:25:55.674858+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 8080, 'log_dups_size': 0, 'ondisk_log_size': 8080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33076166, 'num_objects': 8077, 'num_object_clones': 0, 'num_object_copies': 16154, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8078, 'num_write_kb': 32303, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 3800}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:59.707 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:25:59.708 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'8150", 'reported_seq': 8234, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.758605+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:25:55.758605+0000', 'last_peered': '2024-09-16T10:25:55.758605+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:25:55.758605+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8150, 'log_dups_size': 0, 'ondisk_log_size': 8150, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33358790, 'num_objects': 8146, 'num_object_clones': 0, 'num_object_copies': 16292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8148, 'num_write_kb': 32579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 3954}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:25:59.708 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:25:59.708 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'7976", 'reported_seq': 8053, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.084974+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:25:55.084974+0000', 'last_peered': '2024-09-16T10:25:55.084974+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:25:55.084974+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7976, 'log_dups_size': 0, 'ondisk_log_size': 7976, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32661987, 'num_objects': 7975, 'num_object_clones': 0, 'num_object_copies': 15950, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7975, 'num_write_kb': 31897, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3816}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:59.708 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:25:59.708 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'8282", 'reported_seq': 8412, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.086076+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:55.086076+0000', 'last_peered': '2024-09-16T10:25:55.086076+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:55.086076+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 8282, 'log_dups_size': 0, 'ondisk_log_size': 8282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33891753, 'num_objects': 8277, 'num_object_clones': 0, 'num_object_copies': 16554, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8277, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8279, 'num_write_kb': 33100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3940}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:59.708 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:25:59.708 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'7987", 'reported_seq': 8065, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.117141+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:55.117141+0000', 'last_peered': '2024-09-16T10:25:55.117141+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:55.117141+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 7987, 'log_dups_size': 0, 'ondisk_log_size': 7987, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32699334, 'num_objects': 7985, 'num_object_clones': 0, 'num_object_copies': 15970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7985, 'num_write_kb': 31934, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3818}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:59.708 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:25:59.708 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'8299", 'reported_seq': 8389, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:55.026098+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:25:55.026098+0000', 'last_peered': '2024-09-16T10:25:55.026098+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:25:55.026098+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8299, 'log_dups_size': 0, 'ondisk_log_size': 8299, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33945484, 'num_objects': 8291, 'num_object_clones': 0, 'num_object_copies': 16582, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8296, 'num_write_kb': 33155, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3896}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:25:59.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:00.393 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:00.393 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 47128/261138 objects degraded (18.047%), 11 pgs degraded, 12 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":249,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":42,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":130569,"data_bytes":534737417,"bytes_used":1814028288,"bytes_avail":288096264192,"bytes_total":289910292480,"degraded_objects":47128,"degraded_total":261138,"degraded_ratio":0.18047162802809244,"misplaced_objects":204,"misplaced_total":261138,"misplaced_ratio":0.00078119614916251172,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":155935,"recovering_keys_per_sec":0,"num_objects_recovered":457,"num_bytes_recovered":1871872,"num_keys_recovered":0,"write_bytes_sec":1624185,"read_op_per_sec":0,"write_op_per_sec":396},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:00.411 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:00.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:00.551 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 168 20 132974 132954 3.0907 1.69531 0.0500657 0.0252692 2024-09-16T10:26:00.946 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:00.946 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:00.969 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-16T10:26:00.613175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":540713481,"num_objects":132028,"num_object_clones":0,"num_object_copies":264056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4395,"num_objects_unfound":0,"num_objects_dirty":132028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109657,"num_write_kb":438510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4016,"num_bytes_recovered":16364022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132093,"ondisk_log_size":132093,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1766332,"kb_used_data":1071844,"kb_used_omap":12,"kb_used_meta":694451,"kb_avail":281349188,"statfs":{"total":289910292480,"available":288101568512,"internally_reserved":0,"allocated":1097568256,"data_stored":1096769893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":711117826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,6,10,12,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25473024,"num_objects":6219,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4192,"num_objects_misplaced":3734,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":24876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":1875968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004337"},"pg_stats":[{"pgid":"2.f","version":"42'8300","reported_seq":8369,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:00.081379+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:00.081379+0000","last_peered":"2024-09-16T10:26:00.081379+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:00.081379+0000","last_undegraded":"2024-09-16T10:26:00.081379+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8300,"log_dups_size":0,"ondisk_log_size":8300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33996800,"num_objects":8300,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":8300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8300,"num_write_kb":33200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4109},{"shards":"0,2","objects":4191}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'8151","reported_seq":8305,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.793121+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:25:55.793121+0000","last_peered":"2024-09-16T10:25:55.793121+0000","last_clean":"2024-09-16T10:25:55.793121+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:25:55.793121+0000","last_undegraded":"2024-09-16T10:25:55.793121+0000","last_fullsized":"2024-09-16T10:25:55.793121+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8151,"log_dups_size":0,"ondisk_log_size":8151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33361920,"num_objects":8145,"num_object_clones":0,"num_object_copies":16290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4364,"num_write_kb":17456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'8164","reported_seq":8347,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.725095+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:25:55.725095+0000","last_peered":"2024-09-16T10:25:55.725095+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:25:55.725095+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8164,"log_dups_size":0,"ondisk_log_size":8164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33423360,"num_objects":8160,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":16884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'8178","reported_seq":8232,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:25:55.793795+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:25:55.793795+0000","last_peered":"2024-09-16T10:25:55.793795+0000","last_clean":"2024-09-16T10:25:55.793795+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:25:55.793795+0000","last_undegraded":"2024-09-16T10:25:55.793795+0000","last_fullsized":"2024-09-16T10:25:55.793795+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8178,"log_dups_size":0,"ondisk_log_size":8178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33497088,"num_objects":8178,"num_object_clones":0,"num_object_copies":16356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8178,"num_write_kb":32712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'8326","reported_seq":8480,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.090939+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:00.090939+0000","last_peered":"2024-09-16T10:26:00.090939+0000","last_clean":"2024-09-16T10:26:00.090939+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:00.090939+0000","last_undegraded":"2024-09-16T10:26:00.090939+0000","last_fullsized":"2024-09-16T10:26:00.090939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34074624,"num_objects":8319,"num_object_clones":0,"num_object_copies":16638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":19872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'8185","reported_seq":8346,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.030371+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:00.030371+0000","last_peered":"2024-09-16T10:26:00.030371+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:00.030371+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8185,"log_dups_size":0,"ondisk_log_size":8185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":8180,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4810,"num_write_kb":19240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3963},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8540","reported_seq":8711,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.046302+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:00.046302+0000","last_peered":"2024-09-16T10:26:00.046302+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:00.046302+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8540,"log_dups_size":0,"ondisk_log_size":8540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34955264,"num_objects":8534,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4782,"num_write_kb":19128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4131},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'8324","reported_seq":8496,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.118652+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:00.118652+0000","last_peered":"2024-09-16T10:26:00.118652+0000","last_clean":"2024-09-16T10:26:00.118652+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:00.118652+0000","last_undegraded":"2024-09-16T10:26:00.118652+0000","last_fullsized":"2024-09-16T10:26:00.118652+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34086912,"num_objects":8322,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":16508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374859+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374859+0000","last_peered":"2024-09-16T10:23:56.374859+0000","last_clean":"2024-09-16T10:23:56.374859+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374859+0000","last_undegraded":"2024-09-16T10:23:56.374859+0000","last_fullsized":"2024-09-16T10:23:56.374859+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'8096","reported_seq":16180,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:25:55.799332+0000","last_change":"2024-09-16T10:23:56.821675+0000","last_active":"2024-09-16T10:25:55.799332+0000","last_peered":"2024-09-16T10:25:55.799332+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:25:55.799332+0000","last_undegraded":"2024-09-16T10:25:55.799332+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8096,"log_dups_size":0,"ondisk_log_size":8096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33133510,"num_objects":8091,"num_object_clones":0,"num_object_copies":16182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":204,"num_objects_unfound":0,"num_objects_dirty":8091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":32360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3997,"num_bytes_recovered":16355328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":204},{"shards":"2","objects":7887}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8497","reported_seq":8572,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.052273+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:00.052273+0000","last_peered":"2024-09-16T10:26:00.052273+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:00.052273+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8497,"log_dups_size":0,"ondisk_log_size":8497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34771910,"num_objects":8491,"num_object_clones":0,"num_object_copies":16982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8495,"num_write_kb":33960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4057},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'8080","reported_seq":8183,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.674858+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:25:55.674858+0000","last_peered":"2024-09-16T10:25:55.674858+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:25:55.674858+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8080,"log_dups_size":0,"ondisk_log_size":8080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076166,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8078,"num_write_kb":32303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3800}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":66,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.771942+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.771942+0000","last_peered":"2024-09-16T10:23:56.771942+0000","last_clean":"2024-09-16T10:23:56.771942+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.771942+0000","last_undegraded":"2024-09-16T10:23:56.771942+0000","last_fullsized":"2024-09-16T10:23:56.771942+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'8150","reported_seq":8234,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:55.758605+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:25:55.758605+0000","last_peered":"2024-09-16T10:25:55.758605+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:25:55.758605+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8150,"log_dups_size":0,"ondisk_log_size":8150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33358790,"num_objects":8146,"num_object_clones":0,"num_object_copies":16292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8148,"num_write_kb":32579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":3954}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'8108","reported_seq":8185,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.121214+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:00.121214+0000","last_peered":"2024-09-16T10:26:00.121214+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:00.121214+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8108,"log_dups_size":0,"ondisk_log_size":8108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33202659,"num_objects":8107,"num_object_clones":0,"num_object_copies":16214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8107,"num_write_kb":32425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3948},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.893933+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.893933+0000","last_peered":"2024-09-16T10:23:56.893933+0000","last_clean":"2024-09-16T10:23:56.893933+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.893933+0000","last_undegraded":"2024-09-16T10:23:56.893933+0000","last_fullsized":"2024-09-16T10:23:56.893933+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'8428","reported_seq":8558,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:59.960516+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:59.960516+0000","last_peered":"2024-09-16T10:25:59.960516+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:59.960516+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8428,"log_dups_size":0,"ondisk_log_size":8428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34489769,"num_objects":8423,"num_object_clones":0,"num_object_copies":16846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":33684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4086},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'8124","reported_seq":8202,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:59.791499+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:59.791499+0000","last_peered":"2024-09-16T10:25:59.791499+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:59.791499+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8124,"log_dups_size":0,"ondisk_log_size":8124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33260486,"num_objects":8122,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8122,"num_write_kb":32482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3955},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896307+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.896307+0000","last_peered":"2024-09-16T10:23:56.896307+0000","last_clean":"2024-09-16T10:23:56.896307+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.896307+0000","last_undegraded":"2024-09-16T10:23:56.896307+0000","last_fullsized":"2024-09-16T10:23:56.896307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8439","reported_seq":8529,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.118042+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:00.118042+0000","last_peered":"2024-09-16T10:26:00.118042+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:00.118042+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34518924,"num_objects":8431,"num_object_clones":0,"num_object_copies":16862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":33715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4036},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":540713462,"num_objects":132026,"num_object_clones":0,"num_object_copies":264052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4395,"num_objects_unfound":0,"num_objects_dirty":132026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109655,"num_write_kb":438508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4015,"num_bytes_recovered":16364003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1098186752,"data_stored":1098103653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132090,"ondisk_log_size":132090,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018921,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":623544,"kb_used_data":397488,"kb_used_omap":4,"kb_used_meta":226043,"kb_avail":93748296,"statfs":{"total":96636764160,"available":95998255104,"internally_reserved":0,"allocated":407027712,"data_stored":406784473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":231468720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":0.98999999999999999,"15min":0.90400000000000003},"min":{"1min":0.246,"5min":0.184,"15min":0.184},"max":{"1min":6.3579999999999997,"5min":6.3579999999999997,"15min":6.3579999999999997},"last":0.878},{"interface":"front","average":{"1min":2.7810000000000001,"5min":1.276,"15min":1.0249999999999999},"min":{"1min":0.254,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":7.976,"5min":7.976,"15min":7.976},"last":1.6619999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:24:56 2024","interfaces":[{"interface":"back","average":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.8879999999999999,"5min":6.8879999999999999,"15min":6.8879999999999999},"last":1.5589999999999999},{"interface":"front","average":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":7.3200000000000003,"5min":7.3200000000000003,"15min":7.3200000000000003},"last":1.022}]}]},{"osd":1,"up_from":40,"seq":171798691866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395532,"kb_used_data":207236,"kb_used_omap":4,"kb_used_meta":188283,"kb_avail":93976308,"statfs":{"total":96636764160,"available":96231739392,"internally_reserved":0,"allocated":212209664,"data_stored":211894165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":192802474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":2.8860000000000001},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.442}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":0.627}]}]},{"osd":0,"up_from":24,"seq":103079215150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747256,"kb_used_data":467120,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93624584,"statfs":{"total":96636764160,"available":95871574016,"internally_reserved":0,"allocated":478330880,"data_stored":478091255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":286846632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,8,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.2}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":5.5640000000000001},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":1.859}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":478437376,"data_stored":478433763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212385792,"data_stored":212309919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407363584,"data_stored":407359971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:01.551 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 169 20 133344 133324 3.08096 1.44531 0.0078696 0.0252957 2024-09-16T10:26:02.552 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 170 20 133631 133611 3.06944 1.12109 0.0346131 0.0254449 2024-09-16T10:26:03.552 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 171 20 134129 134109 3.06286 1.94531 0.022367 0.0255 2024-09-16T10:26:03.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:04.541 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:04.542 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:04.552 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 172 20 134590 134570 3.05552 1.80078 0.0431096 0.0255617 2024-09-16T10:26:04.566 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-16T10:26:02.613685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":544350729,"num_objects":132916,"num_object_clones":0,"num_object_copies":265832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":132916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110545,"num_write_kb":442062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4220,"num_bytes_recovered":17199606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132981,"ondisk_log_size":132981,"up":48,"acting":36,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1775268,"kb_used_data":1076684,"kb_used_omap":12,"kb_used_meta":698547,"kb_avail":281340252,"statfs":{"total":289910292480,"available":288092418048,"internally_reserved":0,"allocated":1102524416,"data_stored":1101726053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":715312130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,10,11,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19308544,"num_objects":4714,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4192,"num_objects_misplaced":3750,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4714,"num_write_kb":18856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":442,"num_bytes_recovered":1810432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002987"},"pg_stats":[{"pgid":"2.f","version":"42'8300","reported_seq":8369,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:00.081379+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:00.081379+0000","last_peered":"2024-09-16T10:26:00.081379+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:00.081379+0000","last_undegraded":"2024-09-16T10:26:00.081379+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8300,"log_dups_size":0,"ondisk_log_size":8300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33996800,"num_objects":8300,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":8300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8300,"num_write_kb":33200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4109},{"shards":"0,2","objects":4191}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'8295","reported_seq":8450,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.775886+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:00.775886+0000","last_peered":"2024-09-16T10:26:00.775886+0000","last_clean":"2024-09-16T10:26:00.775886+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:00.775886+0000","last_undegraded":"2024-09-16T10:26:00.775886+0000","last_fullsized":"2024-09-16T10:26:00.775886+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8295,"log_dups_size":0,"ondisk_log_size":8295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33951744,"num_objects":8289,"num_object_clones":0,"num_object_copies":16578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4508,"num_write_kb":18032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'8332","reported_seq":8515,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.761239+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:00.761239+0000","last_peered":"2024-09-16T10:26:00.761239+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:00.761239+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8332,"log_dups_size":0,"ondisk_log_size":8332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34111488,"num_objects":8328,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":17556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'8328","reported_seq":8381,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.736426+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:00.736426+0000","last_peered":"2024-09-16T10:26:00.736426+0000","last_clean":"2024-09-16T10:26:00.736426+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:00.736426+0000","last_undegraded":"2024-09-16T10:26:00.736426+0000","last_fullsized":"2024-09-16T10:26:00.736426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8328,"log_dups_size":0,"ondisk_log_size":8328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34111488,"num_objects":8328,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":33312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'8326","reported_seq":8480,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.090939+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:00.090939+0000","last_peered":"2024-09-16T10:26:00.090939+0000","last_clean":"2024-09-16T10:26:00.090939+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:00.090939+0000","last_undegraded":"2024-09-16T10:26:00.090939+0000","last_fullsized":"2024-09-16T10:26:00.090939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34074624,"num_objects":8319,"num_object_clones":0,"num_object_copies":16638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":19872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'8185","reported_seq":8346,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.030371+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:00.030371+0000","last_peered":"2024-09-16T10:26:00.030371+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:00.030371+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8185,"log_dups_size":0,"ondisk_log_size":8185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":8180,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4810,"num_write_kb":19240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3963},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8540","reported_seq":8711,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.046302+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:00.046302+0000","last_peered":"2024-09-16T10:26:00.046302+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:00.046302+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8540,"log_dups_size":0,"ondisk_log_size":8540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34955264,"num_objects":8534,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4782,"num_write_kb":19128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4131},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'8324","reported_seq":8496,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.118652+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:00.118652+0000","last_peered":"2024-09-16T10:26:00.118652+0000","last_clean":"2024-09-16T10:26:00.118652+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:00.118652+0000","last_undegraded":"2024-09-16T10:26:00.118652+0000","last_fullsized":"2024-09-16T10:26:00.118652+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34086912,"num_objects":8322,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":16508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.374859+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:23:56.374859+0000","last_peered":"2024-09-16T10:23:56.374859+0000","last_clean":"2024-09-16T10:23:56.374859+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:23:56.374859+0000","last_undegraded":"2024-09-16T10:23:56.374859+0000","last_fullsized":"2024-09-16T10:23:56.374859+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"42'8242","reported_seq":16735,"reported_epoch":42,"state":"active+undersized+remapped","last_fresh":"2024-09-16T10:26:00.761892+0000","last_change":"2024-09-16T10:25:59.980749+0000","last_active":"2024-09-16T10:26:00.761892+0000","last_peered":"2024-09-16T10:26:00.761892+0000","last_clean":"2024-09-16T10:23:54.345012+0000","last_became_active":"2024-09-16T10:23:56.776179+0000","last_became_peered":"2024-09-16T10:23:56.776179+0000","last_unstale":"2024-09-16T10:26:00.761892+0000","last_undegraded":"2024-09-16T10:26:00.761892+0000","last_fullsized":"2024-09-16T10:23:56.365307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33731526,"num_objects":8237,"num_object_clones":0,"num_object_copies":16474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8240,"num_write_kb":32944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"2","objects":8237}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8497","reported_seq":8572,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.052273+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:00.052273+0000","last_peered":"2024-09-16T10:26:00.052273+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:00.052273+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8497,"log_dups_size":0,"ondisk_log_size":8497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34771910,"num_objects":8491,"num_object_clones":0,"num_object_copies":16982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8495,"num_write_kb":33960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4057},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'8220","reported_seq":8323,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.800913+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:00.800913+0000","last_peered":"2024-09-16T10:26:00.800913+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:00.800913+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8220,"log_dups_size":0,"ondisk_log_size":8220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33649606,"num_objects":8217,"num_object_clones":0,"num_object_copies":16434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8218,"num_write_kb":32863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":66,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.771942+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:23:56.771942+0000","last_peered":"2024-09-16T10:23:56.771942+0000","last_clean":"2024-09-16T10:23:56.771942+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:23:56.771942+0000","last_undegraded":"2024-09-16T10:23:56.771942+0000","last_fullsized":"2024-09-16T10:23:56.771942+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'8290","reported_seq":8374,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.671137+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:00.671137+0000","last_peered":"2024-09-16T10:26:00.671137+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:00.671137+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8290,"log_dups_size":0,"ondisk_log_size":8290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33932230,"num_objects":8286,"num_object_clones":0,"num_object_copies":16572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8288,"num_write_kb":33139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4094}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'8108","reported_seq":8185,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.121214+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:00.121214+0000","last_peered":"2024-09-16T10:26:00.121214+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:00.121214+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8108,"log_dups_size":0,"ondisk_log_size":8108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33202659,"num_objects":8107,"num_object_clones":0,"num_object_copies":16214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8107,"num_write_kb":32425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3948},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.893933+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:23:56.893933+0000","last_peered":"2024-09-16T10:23:56.893933+0000","last_clean":"2024-09-16T10:23:56.893933+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:23:56.893933+0000","last_undegraded":"2024-09-16T10:23:56.893933+0000","last_fullsized":"2024-09-16T10:23:56.893933+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'8428","reported_seq":8558,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:59.960516+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:59.960516+0000","last_peered":"2024-09-16T10:25:59.960516+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:59.960516+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8428,"log_dups_size":0,"ondisk_log_size":8428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34489769,"num_objects":8423,"num_object_clones":0,"num_object_copies":16846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":33684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4086},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'8124","reported_seq":8202,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:59.791499+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:59.791499+0000","last_peered":"2024-09-16T10:25:59.791499+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:59.791499+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8124,"log_dups_size":0,"ondisk_log_size":8124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33260486,"num_objects":8122,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8122,"num_write_kb":32482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3955},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.896307+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:23:56.896307+0000","last_peered":"2024-09-16T10:23:56.896307+0000","last_clean":"2024-09-16T10:23:56.896307+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:23:56.896307+0000","last_undegraded":"2024-09-16T10:23:56.896307+0000","last_fullsized":"2024-09-16T10:23:56.896307+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":60,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.369471+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:23:56.369471+0000","last_peered":"2024-09-16T10:23:56.369471+0000","last_clean":"2024-09-16T10:23:56.369471+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:23:56.369471+0000","last_undegraded":"2024-09-16T10:23:56.369471+0000","last_fullsized":"2024-09-16T10:23:56.369471+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8439","reported_seq":8529,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.118042+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:00.118042+0000","last_peered":"2024-09-16T10:26:00.118042+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:00.118042+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34518924,"num_objects":8431,"num_object_clones":0,"num_object_copies":16862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":33715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4036},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":544350710,"num_objects":132914,"num_object_clones":0,"num_object_copies":265828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":132914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110543,"num_write_kb":442060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":17199587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1103097856,"data_stored":1103014757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132978,"ondisk_log_size":132978,"up":32,"acting":20,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018922,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632480,"kb_used_data":402328,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93739360,"statfs":{"total":96636764160,"available":95989104640,"internally_reserved":0,"allocated":411983872,"data_stored":411740633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":235663024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":8.9879999999999995}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":4.0540000000000003},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.46999999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":395532,"kb_used_data":207236,"kb_used_omap":4,"kb_used_meta":188283,"kb_avail":93976308,"statfs":{"total":96636764160,"available":96231739392,"internally_reserved":0,"allocated":212209664,"data_stored":211894165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":192802474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":3.6000000000000001,"5min":3.6000000000000001,"15min":3.6000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":14.244,"5min":14.244,"15min":14.244},"last":2.8860000000000001},{"interface":"front","average":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.7460000000000004,"5min":8.7460000000000004,"15min":8.7460000000000004},"last":0.442}]},{"osd":2,"last update":"Mon Sep 16 10:24:58 2024","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.319,"15min":2.319},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.60199999999999998},{"interface":"front","average":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.394,"5min":11.394,"15min":11.394},"last":0.627}]}]},{"osd":0,"up_from":24,"seq":103079215150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747256,"kb_used_data":467120,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93624584,"statfs":{"total":96636764160,"available":95871574016,"internally_reserved":0,"allocated":478330880,"data_stored":478091255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":286846632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,8,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.2}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":5.5640000000000001},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":1.859}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":478437376,"data_stored":478433763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":212385792,"data_stored":212309919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412274688,"data_stored":412271075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:04.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:05.158 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:05.158 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:05.181 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-16T10:26:04.614945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":544649737,"num_objects":132989,"num_object_clones":0,"num_object_copies":265978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":132989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110618,"num_write_kb":442354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4220,"num_bytes_recovered":17199606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133054,"ondisk_log_size":133054,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1769696,"kb_used_data":1083396,"kb_used_omap":12,"kb_used_meta":686259,"kb_avail":281345824,"statfs":{"total":289910292480,"available":288098123776,"internally_reserved":0,"allocated":1109397504,"data_stored":1108588746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":702729218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,5,10,9,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9912320,"num_objects":2420,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4192,"num_objects_misplaced":3987,"num_objects_unfound":0,"num_objects_dirty":2420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2420,"num_write_kb":9680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":839680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003395"},"pg_stats":[{"pgid":"2.f","version":"42'8300","reported_seq":8369,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:00.081379+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:00.081379+0000","last_peered":"2024-09-16T10:26:00.081379+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:00.081379+0000","last_undegraded":"2024-09-16T10:26:00.081379+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8300,"log_dups_size":0,"ondisk_log_size":8300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33996800,"num_objects":8300,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":8300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8300,"num_write_kb":33200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4109},{"shards":"0,2","objects":4191}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'8295","reported_seq":8450,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.775886+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:00.775886+0000","last_peered":"2024-09-16T10:26:00.775886+0000","last_clean":"2024-09-16T10:26:00.775886+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:00.775886+0000","last_undegraded":"2024-09-16T10:26:00.775886+0000","last_fullsized":"2024-09-16T10:26:00.775886+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8295,"log_dups_size":0,"ondisk_log_size":8295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33951744,"num_objects":8289,"num_object_clones":0,"num_object_copies":16578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4508,"num_write_kb":18032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'8332","reported_seq":8515,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.761239+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:00.761239+0000","last_peered":"2024-09-16T10:26:00.761239+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:00.761239+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8332,"log_dups_size":0,"ondisk_log_size":8332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34111488,"num_objects":8328,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":17556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'8328","reported_seq":8381,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.736426+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:00.736426+0000","last_peered":"2024-09-16T10:26:00.736426+0000","last_clean":"2024-09-16T10:26:00.736426+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:00.736426+0000","last_undegraded":"2024-09-16T10:26:00.736426+0000","last_fullsized":"2024-09-16T10:26:00.736426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8328,"log_dups_size":0,"ondisk_log_size":8328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34111488,"num_objects":8328,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":33312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'8326","reported_seq":8480,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.090939+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:00.090939+0000","last_peered":"2024-09-16T10:26:00.090939+0000","last_clean":"2024-09-16T10:26:00.090939+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:00.090939+0000","last_undegraded":"2024-09-16T10:26:00.090939+0000","last_fullsized":"2024-09-16T10:26:00.090939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34074624,"num_objects":8319,"num_object_clones":0,"num_object_copies":16638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":19872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'8185","reported_seq":8346,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.030371+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:00.030371+0000","last_peered":"2024-09-16T10:26:00.030371+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:00.030371+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8185,"log_dups_size":0,"ondisk_log_size":8185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":8180,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4810,"num_write_kb":19240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3963},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8540","reported_seq":8711,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.046302+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:00.046302+0000","last_peered":"2024-09-16T10:26:00.046302+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:00.046302+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8540,"log_dups_size":0,"ondisk_log_size":8540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34955264,"num_objects":8534,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4782,"num_write_kb":19128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4131},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'8324","reported_seq":8496,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.118652+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:00.118652+0000","last_peered":"2024-09-16T10:26:00.118652+0000","last_clean":"2024-09-16T10:26:00.118652+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:00.118652+0000","last_undegraded":"2024-09-16T10:26:00.118652+0000","last_fullsized":"2024-09-16T10:26:00.118652+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34086912,"num_objects":8322,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":16508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8315","reported_seq":16817,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.443940+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:03.443940+0000","last_peered":"2024-09-16T10:26:03.443940+0000","last_clean":"2024-09-16T10:26:03.443940+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:03.443940+0000","last_undegraded":"2024-09-16T10:26:03.443940+0000","last_fullsized":"2024-09-16T10:26:03.443940+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8315,"log_dups_size":0,"ondisk_log_size":8315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34030534,"num_objects":8310,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8313,"num_write_kb":33236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8497","reported_seq":8572,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.052273+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:00.052273+0000","last_peered":"2024-09-16T10:26:00.052273+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:00.052273+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8497,"log_dups_size":0,"ondisk_log_size":8497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34771910,"num_objects":8491,"num_object_clones":0,"num_object_copies":16982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8495,"num_write_kb":33960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4057},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'8220","reported_seq":8323,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.800913+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:00.800913+0000","last_peered":"2024-09-16T10:26:00.800913+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:00.800913+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8220,"log_dups_size":0,"ondisk_log_size":8220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33649606,"num_objects":8217,"num_object_clones":0,"num_object_copies":16434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8218,"num_write_kb":32863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'8290","reported_seq":8374,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.671137+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:00.671137+0000","last_peered":"2024-09-16T10:26:00.671137+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:00.671137+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8290,"log_dups_size":0,"ondisk_log_size":8290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33932230,"num_objects":8286,"num_object_clones":0,"num_object_copies":16572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8288,"num_write_kb":33139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4094}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'8108","reported_seq":8185,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.121214+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:00.121214+0000","last_peered":"2024-09-16T10:26:00.121214+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:00.121214+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8108,"log_dups_size":0,"ondisk_log_size":8108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33202659,"num_objects":8107,"num_object_clones":0,"num_object_copies":16214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8107,"num_write_kb":32425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3948},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'8428","reported_seq":8558,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:59.960516+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:59.960516+0000","last_peered":"2024-09-16T10:25:59.960516+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:59.960516+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8428,"log_dups_size":0,"ondisk_log_size":8428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34489769,"num_objects":8423,"num_object_clones":0,"num_object_copies":16846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":33684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4086},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'8124","reported_seq":8202,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:59.791499+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:59.791499+0000","last_peered":"2024-09-16T10:25:59.791499+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:59.791499+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8124,"log_dups_size":0,"ondisk_log_size":8124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33260486,"num_objects":8122,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8122,"num_write_kb":32482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3955},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8439","reported_seq":8529,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.118042+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:00.118042+0000","last_peered":"2024-09-16T10:26:00.118042+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:00.118042+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34518924,"num_objects":8431,"num_object_clones":0,"num_object_copies":16862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":33715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4036},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":544649718,"num_objects":132987,"num_object_clones":0,"num_object_copies":265974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":132987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110616,"num_write_kb":442352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":17199587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1110102016,"data_stored":1110018917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133051,"ondisk_log_size":133051,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018922,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632480,"kb_used_data":402328,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93739360,"statfs":{"total":96636764160,"available":95989104640,"internally_reserved":0,"allocated":411983872,"data_stored":411740633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":235663024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":8.9879999999999995}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":4.0540000000000003},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.46999999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389960,"kb_used_data":213948,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93981880,"statfs":{"total":96636764160,"available":96237445120,"internally_reserved":0,"allocated":219082752,"data_stored":218756858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":180219562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":3.2160000000000002},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.1859999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":3.254},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":0.63900000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747256,"kb_used_data":467120,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93624584,"statfs":{"total":96636764160,"available":95871574016,"internally_reserved":0,"allocated":478330880,"data_stored":478091255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":286846632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,8,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.2}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":5.5640000000000001},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":1.859}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":478437376,"data_stored":478433763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":219389952,"data_stored":219314079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412274688,"data_stored":412271075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:05.182 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:05.182 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "42'8300", 'reported_seq': 8369, 'reported_epoch': 42, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:00.081379+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:00.081379+0000', 'last_peered': '2024-09-16T10:26:00.081379+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:00.081379+0000', 'last_undegraded': '2024-09-16T10:26:00.081379+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8300, 'log_dups_size': 0, 'ondisk_log_size': 8300, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33996800, 'num_objects': 8300, 'num_object_clones': 0, 'num_object_copies': 16600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4191, 'num_objects_unfound': 0, 'num_objects_dirty': 8300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8300, 'num_write_kb': 33200, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 4096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4109}, {'shards': '0,2', 'objects': 4191}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:05.182 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:05.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "42'8332", 'reported_seq': 8515, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:00.761239+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:00.761239+0000', 'last_peered': '2024-09-16T10:26:00.761239+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:00.761239+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 8332, 'log_dups_size': 0, 'ondisk_log_size': 8332, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34111488, 'num_objects': 8328, 'num_object_clones': 0, 'num_object_copies': 16656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4389, 'num_write_kb': 17556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 3993}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:05.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:05.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "42'8185", 'reported_seq': 8346, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:00.030371+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:00.030371+0000', 'last_peered': '2024-09-16T10:26:00.030371+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:00.030371+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 8185, 'log_dups_size': 0, 'ondisk_log_size': 8185, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33505280, 'num_objects': 8180, 'num_object_clones': 0, 'num_object_copies': 16360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4810, 'num_write_kb': 19240, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3963}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:05.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:05.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "42'8540", 'reported_seq': 8711, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:00.046302+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:00.046302+0000', 'last_peered': '2024-09-16T10:26:00.046302+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:00.046302+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 8540, 'log_dups_size': 0, 'ondisk_log_size': 8540, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34955264, 'num_objects': 8534, 'num_object_clones': 0, 'num_object_copies': 17068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8534, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4782, 'num_write_kb': 19128, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4131}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:05.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:05.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "42'8497", 'reported_seq': 8572, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:00.052273+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:00.052273+0000', 'last_peered': '2024-09-16T10:26:00.052273+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:00.052273+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8497, 'log_dups_size': 0, 'ondisk_log_size': 8497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34771910, 'num_objects': 8491, 'num_object_clones': 0, 'num_object_copies': 16982, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8495, 'num_write_kb': 33960, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4057}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:05.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:05.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "42'8220", 'reported_seq': 8323, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:00.800913+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:00.800913+0000', 'last_peered': '2024-09-16T10:26:00.800913+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:00.800913+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 8220, 'log_dups_size': 0, 'ondisk_log_size': 8220, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33649606, 'num_objects': 8217, 'num_object_clones': 0, 'num_object_copies': 16434, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8218, 'num_write_kb': 32863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 3940}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:05.183 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:05.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "42'8290", 'reported_seq': 8374, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:00.671137+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:00.671137+0000', 'last_peered': '2024-09-16T10:26:00.671137+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:00.671137+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8290, 'log_dups_size': 0, 'ondisk_log_size': 8290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33932230, 'num_objects': 8286, 'num_object_clones': 0, 'num_object_copies': 16572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8288, 'num_write_kb': 33139, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 4094}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:05.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:05.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "42'8108", 'reported_seq': 8185, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:00.121214+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:00.121214+0000', 'last_peered': '2024-09-16T10:26:00.121214+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:00.121214+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8108, 'log_dups_size': 0, 'ondisk_log_size': 8108, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33202659, 'num_objects': 8107, 'num_object_clones': 0, 'num_object_copies': 16214, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8107, 'num_write_kb': 32425, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3948}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:05.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:05.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "42'8428", 'reported_seq': 8558, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:59.960516+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:25:59.960516+0000', 'last_peered': '2024-09-16T10:25:59.960516+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:25:59.960516+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 8428, 'log_dups_size': 0, 'ondisk_log_size': 8428, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34489769, 'num_objects': 8423, 'num_object_clones': 0, 'num_object_copies': 16846, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8423, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8425, 'num_write_kb': 33684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4086}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:05.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:05.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "42'8124", 'reported_seq': 8202, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:25:59.791499+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:25:59.791499+0000', 'last_peered': '2024-09-16T10:25:59.791499+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:25:59.791499+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8124, 'log_dups_size': 0, 'ondisk_log_size': 8124, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33260486, 'num_objects': 8122, 'num_object_clones': 0, 'num_object_copies': 16244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8122, 'num_write_kb': 32482, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3955}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:05.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:05.184 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "42'8439", 'reported_seq': 8529, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:00.118042+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:00.118042+0000', 'last_peered': '2024-09-16T10:26:00.118042+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:00.118042+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8439, 'log_dups_size': 0, 'ondisk_log_size': 8439, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34518924, 'num_objects': 8431, 'num_object_clones': 0, 'num_object_copies': 16862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8436, 'num_write_kb': 33715, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4036}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:05.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:05.552 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 173 20 135024 135004 3.04766 1.69531 0.0423094 0.0256252 2024-09-16T10:26:05.861 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:05.861 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/265978 objects degraded (16.143%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":254,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":132989,"data_bytes":544649737,"bytes_used":1812168704,"bytes_avail":288098123776,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":265978,"degraded_ratio":0.16142688493033258,"misplaced_objects":4191,"misplaced_total":265978,"misplaced_ratio":0.015756942303498785,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":104915,"recovering_keys_per_sec":0,"num_objects_recovered":205,"num_bytes_recovered":839680,"num_keys_recovered":0,"write_bytes_sec":1238514,"read_op_per_sec":0,"write_op_per_sec":302},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:05.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:05.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:06.456 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:06.457 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-16T10:26:04.614945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":544649737,"num_objects":132989,"num_object_clones":0,"num_object_copies":265978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":132989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110618,"num_write_kb":442354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4220,"num_bytes_recovered":17199606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133054,"ondisk_log_size":133054,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1769696,"kb_used_data":1083396,"kb_used_omap":12,"kb_used_meta":686259,"kb_avail":281345824,"statfs":{"total":289910292480,"available":288098123776,"internally_reserved":0,"allocated":1109397504,"data_stored":1108588746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":702729218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,5,10,9,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9912320,"num_objects":2420,"num_object_clones":0,"num_object_copies":4840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4192,"num_objects_misplaced":3987,"num_objects_unfound":0,"num_objects_dirty":2420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2420,"num_write_kb":9680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":839680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003395"},"pg_stats":[{"pgid":"2.f","version":"42'8300","reported_seq":8369,"reported_epoch":42,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:00.081379+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:00.081379+0000","last_peered":"2024-09-16T10:26:00.081379+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:00.081379+0000","last_undegraded":"2024-09-16T10:26:00.081379+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8300,"log_dups_size":0,"ondisk_log_size":8300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33996800,"num_objects":8300,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":8300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8300,"num_write_kb":33200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":4096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4109},{"shards":"0,2","objects":4191}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"42'8295","reported_seq":8450,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.775886+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:00.775886+0000","last_peered":"2024-09-16T10:26:00.775886+0000","last_clean":"2024-09-16T10:26:00.775886+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:00.775886+0000","last_undegraded":"2024-09-16T10:26:00.775886+0000","last_fullsized":"2024-09-16T10:26:00.775886+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8295,"log_dups_size":0,"ondisk_log_size":8295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33951744,"num_objects":8289,"num_object_clones":0,"num_object_copies":16578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4508,"num_write_kb":18032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"42'8332","reported_seq":8515,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.761239+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:00.761239+0000","last_peered":"2024-09-16T10:26:00.761239+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:00.761239+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8332,"log_dups_size":0,"ondisk_log_size":8332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34111488,"num_objects":8328,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4389,"num_write_kb":17556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":3993}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"42'8328","reported_seq":8381,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.736426+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:00.736426+0000","last_peered":"2024-09-16T10:26:00.736426+0000","last_clean":"2024-09-16T10:26:00.736426+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:00.736426+0000","last_undegraded":"2024-09-16T10:26:00.736426+0000","last_fullsized":"2024-09-16T10:26:00.736426+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8328,"log_dups_size":0,"ondisk_log_size":8328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34111488,"num_objects":8328,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8328,"num_write_kb":33312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"42'8326","reported_seq":8480,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.090939+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:00.090939+0000","last_peered":"2024-09-16T10:26:00.090939+0000","last_clean":"2024-09-16T10:26:00.090939+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:00.090939+0000","last_undegraded":"2024-09-16T10:26:00.090939+0000","last_fullsized":"2024-09-16T10:26:00.090939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34074624,"num_objects":8319,"num_object_clones":0,"num_object_copies":16638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4968,"num_write_kb":19872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"42'8185","reported_seq":8346,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.030371+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:00.030371+0000","last_peered":"2024-09-16T10:26:00.030371+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:00.030371+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8185,"log_dups_size":0,"ondisk_log_size":8185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":8180,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4810,"num_write_kb":19240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3963},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"42'8540","reported_seq":8711,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.046302+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:00.046302+0000","last_peered":"2024-09-16T10:26:00.046302+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:00.046302+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8540,"log_dups_size":0,"ondisk_log_size":8540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34955264,"num_objects":8534,"num_object_clones":0,"num_object_copies":17068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4782,"num_write_kb":19128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4131},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"42'8324","reported_seq":8496,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.118652+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:00.118652+0000","last_peered":"2024-09-16T10:26:00.118652+0000","last_clean":"2024-09-16T10:26:00.118652+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:00.118652+0000","last_undegraded":"2024-09-16T10:26:00.118652+0000","last_fullsized":"2024-09-16T10:26:00.118652+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8324,"log_dups_size":0,"ondisk_log_size":8324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34086912,"num_objects":8322,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4127,"num_write_kb":16508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8315","reported_seq":16817,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.443940+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:03.443940+0000","last_peered":"2024-09-16T10:26:03.443940+0000","last_clean":"2024-09-16T10:26:03.443940+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:03.443940+0000","last_undegraded":"2024-09-16T10:26:03.443940+0000","last_fullsized":"2024-09-16T10:26:03.443940+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8315,"log_dups_size":0,"ondisk_log_size":8315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34030534,"num_objects":8310,"num_object_clones":0,"num_object_copies":16620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8313,"num_write_kb":33236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.363311+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:23:56.363311+0000","last_peered":"2024-09-16T10:23:56.363311+0000","last_clean":"2024-09-16T10:23:56.363311+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:23:56.363311+0000","last_undegraded":"2024-09-16T10:23:56.363311+0000","last_fullsized":"2024-09-16T10:23:56.363311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"42'8497","reported_seq":8572,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.052273+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:00.052273+0000","last_peered":"2024-09-16T10:26:00.052273+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:00.052273+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8497,"log_dups_size":0,"ondisk_log_size":8497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34771910,"num_objects":8491,"num_object_clones":0,"num_object_copies":16982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8495,"num_write_kb":33960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4057},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":85,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.917420+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:23:56.917420+0000","last_peered":"2024-09-16T10:23:56.917420+0000","last_clean":"2024-09-16T10:23:56.917420+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:23:56.917420+0000","last_undegraded":"2024-09-16T10:23:56.917420+0000","last_fullsized":"2024-09-16T10:23:56.917420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"42'8220","reported_seq":8323,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.800913+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:00.800913+0000","last_peered":"2024-09-16T10:26:00.800913+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:00.800913+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8220,"log_dups_size":0,"ondisk_log_size":8220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33649606,"num_objects":8217,"num_object_clones":0,"num_object_copies":16434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8218,"num_write_kb":32863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":3940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"42'8290","reported_seq":8374,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.671137+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:00.671137+0000","last_peered":"2024-09-16T10:26:00.671137+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:00.671137+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8290,"log_dups_size":0,"ondisk_log_size":8290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33932230,"num_objects":8286,"num_object_clones":0,"num_object_copies":16572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8288,"num_write_kb":33139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4094}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"42'8108","reported_seq":8185,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.121214+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:00.121214+0000","last_peered":"2024-09-16T10:26:00.121214+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:00.121214+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8108,"log_dups_size":0,"ondisk_log_size":8108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33202659,"num_objects":8107,"num_object_clones":0,"num_object_copies":16214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8107,"num_write_kb":32425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3948},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"42'8428","reported_seq":8558,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:59.960516+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:25:59.960516+0000","last_peered":"2024-09-16T10:25:59.960516+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:25:59.960516+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8428,"log_dups_size":0,"ondisk_log_size":8428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34489769,"num_objects":8423,"num_object_clones":0,"num_object_copies":16846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":33684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4086},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":54,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-16T10:23:56.366479+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:23:56.366479+0000","last_peered":"2024-09-16T10:23:56.366479+0000","last_clean":"2024-09-16T10:23:56.366479+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:23:56.366479+0000","last_undegraded":"2024-09-16T10:23:56.366479+0000","last_fullsized":"2024-09-16T10:23:56.366479+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"42'8124","reported_seq":8202,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:25:59.791499+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:25:59.791499+0000","last_peered":"2024-09-16T10:25:59.791499+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:25:59.791499+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8124,"log_dups_size":0,"ondisk_log_size":8124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33260486,"num_objects":8122,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8122,"num_write_kb":32482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3955},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"42'8439","reported_seq":8529,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:00.118042+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:00.118042+0000","last_peered":"2024-09-16T10:26:00.118042+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:00.118042+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8439,"log_dups_size":0,"ondisk_log_size":8439,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34518924,"num_objects":8431,"num_object_clones":0,"num_object_copies":16862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8436,"num_write_kb":33715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4036},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":544649718,"num_objects":132987,"num_object_clones":0,"num_object_copies":265974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":132987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110616,"num_write_kb":442352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":17199587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1110102016,"data_stored":1110018917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133051,"ondisk_log_size":133051,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018922,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":632480,"kb_used_data":402328,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93739360,"statfs":{"total":96636764160,"available":95989104640,"internally_reserved":0,"allocated":411983872,"data_stored":411740633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":235663024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":8.9879999999999995}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":4.0540000000000003},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.46999999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":389960,"kb_used_data":213948,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93981880,"statfs":{"total":96636764160,"available":96237445120,"internally_reserved":0,"allocated":219082752,"data_stored":218756858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":180219562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":3.2160000000000002},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.1859999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":3.254},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":0.63900000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215150,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747256,"kb_used_data":467120,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93624584,"statfs":{"total":96636764160,"available":95871574016,"internally_reserved":0,"allocated":478330880,"data_stored":478091255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":286846632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,8,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.8899999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.2}]},{"osd":2,"last update":"Mon Sep 16 10:24:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4350000000000001,"5min":0.96899999999999997,"15min":0.72499999999999998},"min":{"1min":0.254,"5min":0.182,"15min":0.182},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":9.5060000000000002},"last":5.5640000000000001},{"interface":"front","average":{"1min":2.5950000000000002,"5min":1.4490000000000001,"15min":1.258},"min":{"1min":0.27900000000000003,"5min":0.192,"15min":0.192},"max":{"1min":6.7000000000000002,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":1.859}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":478437376,"data_stored":478433763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":219389952,"data_stored":219314079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":412274688,"data_stored":412271075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:06.552 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 174 20 135461 135441 3.03996 1.70703 0.0402889 0.0256904 2024-09-16T10:26:07.552 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 175 20 135953 135933 3.03357 1.92188 0.0263809 0.025746 2024-09-16T10:26:08.552 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 176 20 136434 136414 3.02701 1.87891 0.0140332 0.0258012 2024-09-16T10:26:09.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:09.552 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 177 20 136931 136911 3.02087 1.94141 0.026543 0.0258561 2024-09-16T10:26:10.037 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:10.038 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:10.069 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-09-16T10:26:08.616716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":553214473,"num_objects":135080,"num_object_clones":0,"num_object_copies":270160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4054,"num_objects_unfound":0,"num_objects_dirty":135080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112709,"num_write_kb":450718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4357,"num_bytes_recovered":17760758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135145,"ondisk_log_size":135145,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1799584,"kb_used_data":1101000,"kb_used_omap":12,"kb_used_meta":698547,"kb_avail":281315936,"statfs":{"total":289910292480,"available":288067518464,"internally_reserved":0,"allocated":1127424000,"data_stored":1126594452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":715312130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,2,2,13,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12500992,"num_objects":3052,"num_object_clones":0,"num_object_copies":6104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-341,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":12208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":341,"num_bytes_recovered":1396736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003540"},"pg_stats":[{"pgid":"2.f","version":"44'8425","reported_seq":8770,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:05.180108+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:05.180108+0000","last_peered":"2024-09-16T10:26:05.180108+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:05.180108+0000","last_undegraded":"2024-09-16T10:26:05.180108+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34508800,"num_objects":8425,"num_object_clones":0,"num_object_copies":16850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4054,"num_objects_unfound":0,"num_objects_dirty":8425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":33700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":565248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4371},{"shards":"0,2","objects":4054}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8421","reported_seq":8578,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:05.726637+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:05.726637+0000","last_peered":"2024-09-16T10:26:05.726637+0000","last_clean":"2024-09-16T10:26:05.726637+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:05.726637+0000","last_undegraded":"2024-09-16T10:26:05.726637+0000","last_fullsized":"2024-09-16T10:26:05.726637+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34467840,"num_objects":8415,"num_object_clones":0,"num_object_copies":16830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4634,"num_write_kb":18536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'8463","reported_seq":8648,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.789698+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:05.789698+0000","last_peered":"2024-09-16T10:26:05.789698+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:05.789698+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8463,"log_dups_size":0,"ondisk_log_size":8463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34648064,"num_objects":8459,"num_object_clones":0,"num_object_copies":16918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":18080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'8471","reported_seq":8527,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:05.818908+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:05.818908+0000","last_peered":"2024-09-16T10:26:05.818908+0000","last_clean":"2024-09-16T10:26:05.818908+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:05.818908+0000","last_undegraded":"2024-09-16T10:26:05.818908+0000","last_fullsized":"2024-09-16T10:26:05.818908+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34697216,"num_objects":8471,"num_object_clones":0,"num_object_copies":16942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":33884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8449","reported_seq":8605,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:05.147110+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:05.147110+0000","last_peered":"2024-09-16T10:26:05.147110+0000","last_clean":"2024-09-16T10:26:05.147110+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:05.147110+0000","last_undegraded":"2024-09-16T10:26:05.147110+0000","last_fullsized":"2024-09-16T10:26:05.147110+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8449,"log_dups_size":0,"ondisk_log_size":8449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34578432,"num_objects":8442,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":20364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8326","reported_seq":8488,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.076997+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:05.076997+0000","last_peered":"2024-09-16T10:26:05.076997+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:05.076997+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34082816,"num_objects":8321,"num_object_clones":0,"num_object_copies":16642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":19804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4104},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'8652","reported_seq":8825,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.130713+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:05.130713+0000","last_peered":"2024-09-16T10:26:05.130713+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:05.130713+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8652,"log_dups_size":0,"ondisk_log_size":8652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35414016,"num_objects":8646,"num_object_clones":0,"num_object_copies":17292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4894,"num_write_kb":19576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4243},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8448","reported_seq":8623,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:05.177075+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:05.177075+0000","last_peered":"2024-09-16T10:26:05.177075+0000","last_clean":"2024-09-16T10:26:05.177075+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:05.177075+0000","last_undegraded":"2024-09-16T10:26:05.177075+0000","last_fullsized":"2024-09-16T10:26:05.177075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8448,"log_dups_size":0,"ondisk_log_size":8448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34594816,"num_objects":8446,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4251,"num_write_kb":17004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8480","reported_seq":16983,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.388792+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:08.388792+0000","last_peered":"2024-09-16T10:26:08.388792+0000","last_clean":"2024-09-16T10:26:08.388792+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:08.388792+0000","last_undegraded":"2024-09-16T10:26:08.388792+0000","last_fullsized":"2024-09-16T10:26:08.388792+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34706374,"num_objects":8475,"num_object_clones":0,"num_object_copies":16950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":33896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'8635","reported_seq":8712,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.080838+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:05.080838+0000","last_peered":"2024-09-16T10:26:05.080838+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:05.080838+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8635,"log_dups_size":0,"ondisk_log_size":8635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35337158,"num_objects":8629,"num_object_clones":0,"num_object_copies":17258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8633,"num_write_kb":34512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4195},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8369","reported_seq":8474,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.765633+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:05.765633+0000","last_peered":"2024-09-16T10:26:05.765633+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:05.765633+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8369,"log_dups_size":0,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34259910,"num_objects":8366,"num_object_clones":0,"num_object_copies":16732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":33459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8403","reported_seq":8489,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.817619+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:05.817619+0000","last_peered":"2024-09-16T10:26:05.817619+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:05.817619+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34395078,"num_objects":8399,"num_object_clones":0,"num_object_copies":16798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8401,"num_write_kb":33591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4207}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8208","reported_seq":8287,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.067859+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:05.067859+0000","last_peered":"2024-09-16T10:26:05.067859+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:05.067859+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8208,"log_dups_size":0,"ondisk_log_size":8208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33612259,"num_objects":8207,"num_object_clones":0,"num_object_copies":16414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":32825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4048},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'8560","reported_seq":8691,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.170965+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:05.170965+0000","last_peered":"2024-09-16T10:26:05.170965+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:05.170965+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8560,"log_dups_size":0,"ondisk_log_size":8560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35030441,"num_objects":8555,"num_object_clones":0,"num_object_copies":17110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8557,"num_write_kb":34212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4218},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8256","reported_seq":8334,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.154095+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:05.154095+0000","last_peered":"2024-09-16T10:26:05.154095+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:05.154095+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8256,"log_dups_size":0,"ondisk_log_size":8256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33801158,"num_objects":8254,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8254,"num_write_kb":33010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4087},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'8576","reported_seq":8667,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.121177+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:05.121177+0000","last_peered":"2024-09-16T10:26:05.121177+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:05.121177+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8576,"log_dups_size":0,"ondisk_log_size":8576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35080076,"num_objects":8568,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8573,"num_write_kb":34263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4173},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":553214454,"num_objects":135078,"num_object_clones":0,"num_object_copies":270156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4054,"num_objects_unfound":0,"num_objects_dirty":135078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112707,"num_write_kb":450716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4356,"num_bytes_recovered":17760739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1128247296,"data_stored":1128164197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135142,"ondisk_log_size":135142,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018923,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640752,"kb_used_data":406504,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93731088,"statfs":{"total":96636764160,"available":95980634112,"internally_reserved":0,"allocated":416260096,"data_stored":416006462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":239857328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.0579999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":1.859},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.74399999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401456,"kb_used_data":221352,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93970384,"statfs":{"total":96636764160,"available":96225673216,"internally_reserved":0,"allocated":226664448,"data_stored":226338554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":184413866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":8.4169999999999998},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":8.3849999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":8.5500000000000007},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":11.412000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215151,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757376,"kb_used_data":473144,"kb_used_omap":4,"kb_used_meta":284219,"kb_avail":93614464,"statfs":{"total":96636764160,"available":95861211136,"internally_reserved":0,"allocated":484499456,"data_stored":484249436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":291040936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":0.93100000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":0.39500000000000002},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.46500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":484732928,"data_stored":484729315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226930688,"data_stored":226854815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":416583680,"data_stored":416580067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:10.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:10.552 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 178 20 137357 137337 3.01325 1.66406 0.0581755 0.0259201 2024-09-16T10:26:10.612 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:10.613 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:10.637 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":166,"stamp":"2024-09-16T10:26:08.616716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":553214473,"num_objects":135080,"num_object_clones":0,"num_object_copies":270160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4054,"num_objects_unfound":0,"num_objects_dirty":135080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112709,"num_write_kb":450718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4357,"num_bytes_recovered":17760758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135145,"ondisk_log_size":135145,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1799584,"kb_used_data":1101000,"kb_used_omap":12,"kb_used_meta":698547,"kb_avail":281315936,"statfs":{"total":289910292480,"available":288067518464,"internally_reserved":0,"allocated":1127424000,"data_stored":1126594452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":715312130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,2,2,13,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12500992,"num_objects":3052,"num_object_clones":0,"num_object_copies":6104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-341,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":12208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":341,"num_bytes_recovered":1396736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003540"},"pg_stats":[{"pgid":"2.f","version":"44'8425","reported_seq":8770,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:05.180108+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:05.180108+0000","last_peered":"2024-09-16T10:26:05.180108+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:05.180108+0000","last_undegraded":"2024-09-16T10:26:05.180108+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8425,"log_dups_size":0,"ondisk_log_size":8425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34508800,"num_objects":8425,"num_object_clones":0,"num_object_copies":16850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4054,"num_objects_unfound":0,"num_objects_dirty":8425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":33700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":565248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4371},{"shards":"0,2","objects":4054}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8421","reported_seq":8578,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:05.726637+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:05.726637+0000","last_peered":"2024-09-16T10:26:05.726637+0000","last_clean":"2024-09-16T10:26:05.726637+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:05.726637+0000","last_undegraded":"2024-09-16T10:26:05.726637+0000","last_fullsized":"2024-09-16T10:26:05.726637+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34467840,"num_objects":8415,"num_object_clones":0,"num_object_copies":16830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4634,"num_write_kb":18536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'8463","reported_seq":8648,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.789698+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:05.789698+0000","last_peered":"2024-09-16T10:26:05.789698+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:05.789698+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8463,"log_dups_size":0,"ondisk_log_size":8463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34648064,"num_objects":8459,"num_object_clones":0,"num_object_copies":16918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":18080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'8471","reported_seq":8527,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:05.818908+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:05.818908+0000","last_peered":"2024-09-16T10:26:05.818908+0000","last_clean":"2024-09-16T10:26:05.818908+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:05.818908+0000","last_undegraded":"2024-09-16T10:26:05.818908+0000","last_fullsized":"2024-09-16T10:26:05.818908+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34697216,"num_objects":8471,"num_object_clones":0,"num_object_copies":16942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":33884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8449","reported_seq":8605,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:05.147110+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:05.147110+0000","last_peered":"2024-09-16T10:26:05.147110+0000","last_clean":"2024-09-16T10:26:05.147110+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:05.147110+0000","last_undegraded":"2024-09-16T10:26:05.147110+0000","last_fullsized":"2024-09-16T10:26:05.147110+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8449,"log_dups_size":0,"ondisk_log_size":8449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34578432,"num_objects":8442,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":20364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8326","reported_seq":8488,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.076997+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:05.076997+0000","last_peered":"2024-09-16T10:26:05.076997+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:05.076997+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8326,"log_dups_size":0,"ondisk_log_size":8326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34082816,"num_objects":8321,"num_object_clones":0,"num_object_copies":16642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4951,"num_write_kb":19804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4104},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'8652","reported_seq":8825,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.130713+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:05.130713+0000","last_peered":"2024-09-16T10:26:05.130713+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:05.130713+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8652,"log_dups_size":0,"ondisk_log_size":8652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35414016,"num_objects":8646,"num_object_clones":0,"num_object_copies":17292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4894,"num_write_kb":19576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4243},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8448","reported_seq":8623,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:05.177075+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:05.177075+0000","last_peered":"2024-09-16T10:26:05.177075+0000","last_clean":"2024-09-16T10:26:05.177075+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:05.177075+0000","last_undegraded":"2024-09-16T10:26:05.177075+0000","last_fullsized":"2024-09-16T10:26:05.177075+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8448,"log_dups_size":0,"ondisk_log_size":8448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34594816,"num_objects":8446,"num_object_clones":0,"num_object_copies":16892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4251,"num_write_kb":17004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8480","reported_seq":16983,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.388792+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:08.388792+0000","last_peered":"2024-09-16T10:26:08.388792+0000","last_clean":"2024-09-16T10:26:08.388792+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:08.388792+0000","last_undegraded":"2024-09-16T10:26:08.388792+0000","last_fullsized":"2024-09-16T10:26:08.388792+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34706374,"num_objects":8475,"num_object_clones":0,"num_object_copies":16950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":33896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'8635","reported_seq":8712,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.080838+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:05.080838+0000","last_peered":"2024-09-16T10:26:05.080838+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:05.080838+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8635,"log_dups_size":0,"ondisk_log_size":8635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35337158,"num_objects":8629,"num_object_clones":0,"num_object_copies":17258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8633,"num_write_kb":34512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4195},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8369","reported_seq":8474,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.765633+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:05.765633+0000","last_peered":"2024-09-16T10:26:05.765633+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:05.765633+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8369,"log_dups_size":0,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34259910,"num_objects":8366,"num_object_clones":0,"num_object_copies":16732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":33459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8403","reported_seq":8489,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.817619+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:05.817619+0000","last_peered":"2024-09-16T10:26:05.817619+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:05.817619+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34395078,"num_objects":8399,"num_object_clones":0,"num_object_copies":16798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8401,"num_write_kb":33591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4207}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8208","reported_seq":8287,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.067859+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:05.067859+0000","last_peered":"2024-09-16T10:26:05.067859+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:05.067859+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8208,"log_dups_size":0,"ondisk_log_size":8208,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33612259,"num_objects":8207,"num_object_clones":0,"num_object_copies":16414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":32825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4048},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'8560","reported_seq":8691,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.170965+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:05.170965+0000","last_peered":"2024-09-16T10:26:05.170965+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:05.170965+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8560,"log_dups_size":0,"ondisk_log_size":8560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35030441,"num_objects":8555,"num_object_clones":0,"num_object_copies":17110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8557,"num_write_kb":34212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4218},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8256","reported_seq":8334,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.154095+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:05.154095+0000","last_peered":"2024-09-16T10:26:05.154095+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:05.154095+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8256,"log_dups_size":0,"ondisk_log_size":8256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33801158,"num_objects":8254,"num_object_clones":0,"num_object_copies":16508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8254,"num_write_kb":33010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4087},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'8576","reported_seq":8667,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.121177+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:05.121177+0000","last_peered":"2024-09-16T10:26:05.121177+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:05.121177+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8576,"log_dups_size":0,"ondisk_log_size":8576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35080076,"num_objects":8568,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8573,"num_write_kb":34263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4173},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":553214454,"num_objects":135078,"num_object_clones":0,"num_object_copies":270156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":4054,"num_objects_unfound":0,"num_objects_dirty":135078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112707,"num_write_kb":450716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4356,"num_bytes_recovered":17760739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1128247296,"data_stored":1128164197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135142,"ondisk_log_size":135142,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018923,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640752,"kb_used_data":406504,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93731088,"statfs":{"total":96636764160,"available":95980634112,"internally_reserved":0,"allocated":416260096,"data_stored":416006462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":239857328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.0579999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":1.859},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.74399999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401456,"kb_used_data":221352,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93970384,"statfs":{"total":96636764160,"available":96225673216,"internally_reserved":0,"allocated":226664448,"data_stored":226338554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":184413866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":8.4169999999999998},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":8.3849999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":8.5500000000000007},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":11.412000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215151,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757376,"kb_used_data":473144,"kb_used_omap":4,"kb_used_meta":284219,"kb_avail":93614464,"statfs":{"total":96636764160,"available":95861211136,"internally_reserved":0,"allocated":484499456,"data_stored":484249436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":291040936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":0.93100000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":0.39500000000000002},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.46500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":484732928,"data_stored":484729315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226930688,"data_stored":226854815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":416583680,"data_stored":416580067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:10.638 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:10.638 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'8425", 'reported_seq': 8770, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:05.180108+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:05.180108+0000', 'last_peered': '2024-09-16T10:26:05.180108+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:05.180108+0000', 'last_undegraded': '2024-09-16T10:26:05.180108+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8425, 'log_dups_size': 0, 'ondisk_log_size': 8425, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34508800, 'num_objects': 8425, 'num_object_clones': 0, 'num_object_copies': 16850, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4054, 'num_objects_unfound': 0, 'num_objects_dirty': 8425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8425, 'num_write_kb': 33700, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 138, 'num_bytes_recovered': 565248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4371}, {'shards': '0,2', 'objects': 4054}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:10.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:10.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'8463", 'reported_seq': 8648, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:05.789698+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:05.789698+0000', 'last_peered': '2024-09-16T10:26:05.789698+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:05.789698+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 8463, 'log_dups_size': 0, 'ondisk_log_size': 8463, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34648064, 'num_objects': 8459, 'num_object_clones': 0, 'num_object_copies': 16918, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4520, 'num_write_kb': 18080, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 4124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:10.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:10.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'8326", 'reported_seq': 8488, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:05.076997+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:05.076997+0000', 'last_peered': '2024-09-16T10:26:05.076997+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:05.076997+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 8326, 'log_dups_size': 0, 'ondisk_log_size': 8326, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34082816, 'num_objects': 8321, 'num_object_clones': 0, 'num_object_copies': 16642, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4951, 'num_write_kb': 19804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4104}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:10.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:10.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'8652", 'reported_seq': 8825, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:05.130713+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:05.130713+0000', 'last_peered': '2024-09-16T10:26:05.130713+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:05.130713+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 8652, 'log_dups_size': 0, 'ondisk_log_size': 8652, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35414016, 'num_objects': 8646, 'num_object_clones': 0, 'num_object_copies': 17292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4894, 'num_write_kb': 19576, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4243}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:10.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:10.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'8635", 'reported_seq': 8712, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:05.080838+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:05.080838+0000', 'last_peered': '2024-09-16T10:26:05.080838+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:05.080838+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8635, 'log_dups_size': 0, 'ondisk_log_size': 8635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35337158, 'num_objects': 8629, 'num_object_clones': 0, 'num_object_copies': 17258, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8633, 'num_write_kb': 34512, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4195}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:10.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:10.639 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'8369", 'reported_seq': 8474, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:05.765633+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:05.765633+0000', 'last_peered': '2024-09-16T10:26:05.765633+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:05.765633+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 8369, 'log_dups_size': 0, 'ondisk_log_size': 8369, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34259910, 'num_objects': 8366, 'num_object_clones': 0, 'num_object_copies': 16732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8367, 'num_write_kb': 33459, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 4089}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:10.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:10.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'8403", 'reported_seq': 8489, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:05.817619+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:05.817619+0000', 'last_peered': '2024-09-16T10:26:05.817619+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:05.817619+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8403, 'log_dups_size': 0, 'ondisk_log_size': 8403, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34395078, 'num_objects': 8399, 'num_object_clones': 0, 'num_object_copies': 16798, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8401, 'num_write_kb': 33591, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 4207}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:10.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:10.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'8208", 'reported_seq': 8287, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:05.067859+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:05.067859+0000', 'last_peered': '2024-09-16T10:26:05.067859+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:05.067859+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8208, 'log_dups_size': 0, 'ondisk_log_size': 8208, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33612259, 'num_objects': 8207, 'num_object_clones': 0, 'num_object_copies': 16414, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8207, 'num_write_kb': 32825, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4048}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:10.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:10.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'8560", 'reported_seq': 8691, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:05.170965+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:26:05.170965+0000', 'last_peered': '2024-09-16T10:26:05.170965+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:26:05.170965+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 8560, 'log_dups_size': 0, 'ondisk_log_size': 8560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35030441, 'num_objects': 8555, 'num_object_clones': 0, 'num_object_copies': 17110, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8557, 'num_write_kb': 34212, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4218}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:10.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:10.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'8256", 'reported_seq': 8334, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:05.154095+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:26:05.154095+0000', 'last_peered': '2024-09-16T10:26:05.154095+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:26:05.154095+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8256, 'log_dups_size': 0, 'ondisk_log_size': 8256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33801158, 'num_objects': 8254, 'num_object_clones': 0, 'num_object_copies': 16508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8254, 'num_write_kb': 33010, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4087}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:10.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:10.640 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'8576", 'reported_seq': 8667, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:05.121177+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:05.121177+0000', 'last_peered': '2024-09-16T10:26:05.121177+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:05.121177+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8576, 'log_dups_size': 0, 'ondisk_log_size': 8576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35080076, 'num_objects': 8568, 'num_object_clones': 0, 'num_object_copies': 17136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8568, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8573, 'num_write_kb': 34263, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4173}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:10.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:11.327 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:11.327 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/273012 objects degraded (15.727%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":260,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":136506,"data_bytes":559055369,"bytes_used":1872482304,"bytes_avail":288037810176,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":273012,"degraded_ratio":0.15726781240385038,"misplaced_objects":3983,"misplaced_total":273012,"misplaced_ratio":0.01458910231052115,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":88066,"recovering_keys_per_sec":0,"num_objects_recovered":208,"num_bytes_recovered":851968,"num_keys_recovered":0,"write_bytes_sec":1519990,"read_op_per_sec":0,"write_op_per_sec":371},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:11.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:11.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:11.559 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 179 20 137780 137760 3.0056 1.65234 0.0346451 0.0259861 2024-09-16T10:26:11.905 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:11.906 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:11.942 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-16T10:26:10.617727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":559055369,"num_objects":136506,"num_object_clones":0,"num_object_copies":273012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3983,"num_objects_unfound":0,"num_objects_dirty":136506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114135,"num_write_kb":456422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4428,"num_bytes_recovered":18051574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136571,"ondisk_log_size":136571,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1828596,"kb_used_data":1107868,"kb_used_omap":12,"kb_used_meta":720691,"kb_avail":281286924,"statfs":{"total":289910292480,"available":288037810176,"internally_reserved":0,"allocated":1134456832,"data_stored":1133627284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":737987586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,3,6,7,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14704640,"num_objects":3590,"num_object_clones":0,"num_object_copies":7180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":3590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3590,"num_write_kb":14360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.674166"},"pg_stats":[{"pgid":"2.f","version":"44'8579","reported_seq":9065,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:10.192581+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:10.192581+0000","last_peered":"2024-09-16T10:26:10.192581+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:10.192581+0000","last_undegraded":"2024-09-16T10:26:10.192581+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35139584,"num_objects":8579,"num_object_clones":0,"num_object_copies":17158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3983,"num_objects_unfound":0,"num_objects_dirty":8579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8579,"num_write_kb":34316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":856064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4596},{"shards":"0,2","objects":3983}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8421","reported_seq":8578,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:05.726637+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:05.726637+0000","last_peered":"2024-09-16T10:26:05.726637+0000","last_clean":"2024-09-16T10:26:05.726637+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:05.726637+0000","last_undegraded":"2024-09-16T10:26:05.726637+0000","last_fullsized":"2024-09-16T10:26:05.726637+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34467840,"num_objects":8415,"num_object_clones":0,"num_object_copies":16830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4634,"num_write_kb":18536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'8463","reported_seq":8648,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.789698+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:05.789698+0000","last_peered":"2024-09-16T10:26:05.789698+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:05.789698+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8463,"log_dups_size":0,"ondisk_log_size":8463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34648064,"num_objects":8459,"num_object_clones":0,"num_object_copies":16918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4520,"num_write_kb":18080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'8471","reported_seq":8527,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:05.818908+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:05.818908+0000","last_peered":"2024-09-16T10:26:05.818908+0000","last_clean":"2024-09-16T10:26:05.818908+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:05.818908+0000","last_undegraded":"2024-09-16T10:26:05.818908+0000","last_fullsized":"2024-09-16T10:26:05.818908+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8471,"log_dups_size":0,"ondisk_log_size":8471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34697216,"num_objects":8471,"num_object_clones":0,"num_object_copies":16942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8471,"num_write_kb":33884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8573","reported_seq":8729,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:10.100180+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:10.100180+0000","last_peered":"2024-09-16T10:26:10.100180+0000","last_clean":"2024-09-16T10:26:10.100180+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:10.100180+0000","last_undegraded":"2024-09-16T10:26:10.100180+0000","last_fullsized":"2024-09-16T10:26:10.100180+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35086336,"num_objects":8566,"num_object_clones":0,"num_object_copies":17132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":20860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8456","reported_seq":8619,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.175469+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:10.175469+0000","last_peered":"2024-09-16T10:26:10.175469+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:10.175469+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34615296,"num_objects":8451,"num_object_clones":0,"num_object_copies":16902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5081,"num_write_kb":20324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4234},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'8802","reported_seq":8975,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.002428+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:10.002428+0000","last_peered":"2024-09-16T10:26:10.002428+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:10.002428+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8802,"log_dups_size":0,"ondisk_log_size":8802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36028416,"num_objects":8796,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":20176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4393},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8588","reported_seq":8763,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:10.185486+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:10.185486+0000","last_peered":"2024-09-16T10:26:10.185486+0000","last_clean":"2024-09-16T10:26:10.185486+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:10.185486+0000","last_undegraded":"2024-09-16T10:26:10.185486+0000","last_fullsized":"2024-09-16T10:26:10.185486+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8588,"log_dups_size":0,"ondisk_log_size":8588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35168256,"num_objects":8586,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":17564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8480","reported_seq":16983,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:08.388792+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:08.388792+0000","last_peered":"2024-09-16T10:26:08.388792+0000","last_clean":"2024-09-16T10:26:08.388792+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:08.388792+0000","last_undegraded":"2024-09-16T10:26:08.388792+0000","last_fullsized":"2024-09-16T10:26:08.388792+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8480,"log_dups_size":0,"ondisk_log_size":8480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34706374,"num_objects":8475,"num_object_clones":0,"num_object_copies":16950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8478,"num_write_kb":33896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'8777","reported_seq":8855,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.173317+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:10.173317+0000","last_peered":"2024-09-16T10:26:10.173317+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:10.173317+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8777,"log_dups_size":0,"ondisk_log_size":8777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35918790,"num_objects":8771,"num_object_clones":0,"num_object_copies":17542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8775,"num_write_kb":35080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4337},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8369","reported_seq":8474,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.765633+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:05.765633+0000","last_peered":"2024-09-16T10:26:05.765633+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:05.765633+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8369,"log_dups_size":0,"ondisk_log_size":8369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34259910,"num_objects":8366,"num_object_clones":0,"num_object_copies":16732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":33459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4089}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8403","reported_seq":8489,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:05.817619+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:05.817619+0000","last_peered":"2024-09-16T10:26:05.817619+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:05.817619+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8403,"log_dups_size":0,"ondisk_log_size":8403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34395078,"num_objects":8399,"num_object_clones":0,"num_object_copies":16798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8401,"num_write_kb":33591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4207}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8352","reported_seq":8431,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.137054+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:10.137054+0000","last_peered":"2024-09-16T10:26:10.137054+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:10.137054+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8352,"log_dups_size":0,"ondisk_log_size":8352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34202083,"num_objects":8351,"num_object_clones":0,"num_object_copies":16702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8351,"num_write_kb":33401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4192},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'8706","reported_seq":8839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.142891+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:10.142891+0000","last_peered":"2024-09-16T10:26:10.142891+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:10.142891+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8706,"log_dups_size":0,"ondisk_log_size":8706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35628457,"num_objects":8701,"num_object_clones":0,"num_object_copies":17402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8703,"num_write_kb":34796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4364},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8402","reported_seq":8480,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.014065+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:10.014065+0000","last_peered":"2024-09-16T10:26:10.014065+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:10.014065+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8402,"log_dups_size":0,"ondisk_log_size":8402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34399174,"num_objects":8400,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":33594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4233},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'8726","reported_seq":8818,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.154679+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:10.154679+0000","last_peered":"2024-09-16T10:26:10.154679+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:10.154679+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8726,"log_dups_size":0,"ondisk_log_size":8726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35694476,"num_objects":8718,"num_object_clones":0,"num_object_copies":17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8723,"num_write_kb":34863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4323},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":559055350,"num_objects":136504,"num_object_clones":0,"num_object_copies":273008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3983,"num_objects_unfound":0,"num_objects_dirty":136504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114133,"num_write_kb":456420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4427,"num_bytes_recovered":18051555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1135239168,"data_stored":1135156069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136568,"ondisk_log_size":136568,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018923,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":640752,"kb_used_data":406504,"kb_used_omap":4,"kb_used_meta":234235,"kb_avail":93731088,"statfs":{"total":96636764160,"available":95980634112,"internally_reserved":0,"allocated":416260096,"data_stored":416006462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":239857328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.0579999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":1.859},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.74399999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401456,"kb_used_data":221352,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93970384,"statfs":{"total":96636764160,"available":96225673216,"internally_reserved":0,"allocated":226664448,"data_stored":226338554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":184413866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":8.4169999999999998},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":8.3849999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":8.5500000000000007},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":11.412000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215152,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":786388,"kb_used_data":480012,"kb_used_omap":4,"kb_used_meta":306363,"kb_avail":93585452,"statfs":{"total":96636764160,"available":95831502848,"internally_reserved":0,"allocated":491532288,"data_stored":491282268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":313716392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":4.5229999999999997},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.8029999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":7.6390000000000002},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.5709999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":491724800,"data_stored":491721187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":226930688,"data_stored":226854815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":416583680,"data_stored":416580067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:12.555 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:26:12.554874+0000 min lat: 0.00139595 max lat: 2.61294 avg lat: 0.0260427 2024-09-16T10:26:12.555 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:26:12.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 180 20 138245 138225 2.99899 1.81641 0.0359644 0.0260427 2024-09-16T10:26:13.555 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 181 20 138726 138706 2.99281 1.87891 0.0247775 0.0260963 2024-09-16T10:26:14.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 182 20 139207 139187 2.98669 1.87891 0.0305736 0.0261496 2024-09-16T10:26:14.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:15.477 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:15.480 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:15.501 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-16T10:26:14.619153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562651657,"num_objects":137384,"num_object_clones":0,"num_object_copies":274768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3983,"num_objects_unfound":0,"num_objects_dirty":137384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115013,"num_write_kb":459934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4428,"num_bytes_recovered":18051574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137449,"ondisk_log_size":137449,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1866540,"kb_used_data":1119572,"kb_used_omap":12,"kb_used_meta":746931,"kb_avail":281248980,"statfs":{"total":289910292480,"available":287998955520,"internally_reserved":0,"allocated":1146441728,"data_stored":1145612180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":764857346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,11,3,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18300928,"num_objects":4468,"num_object_clones":0,"num_object_copies":8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4468,"num_write_kb":17872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005467"},"pg_stats":[{"pgid":"2.f","version":"44'8579","reported_seq":9065,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:10.192581+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:10.192581+0000","last_peered":"2024-09-16T10:26:10.192581+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:10.192581+0000","last_undegraded":"2024-09-16T10:26:10.192581+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35139584,"num_objects":8579,"num_object_clones":0,"num_object_copies":17158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3983,"num_objects_unfound":0,"num_objects_dirty":8579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8579,"num_write_kb":34316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":856064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4596},{"shards":"0,2","objects":3983}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8562","reported_seq":8718,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:10.802958+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:10.802958+0000","last_peered":"2024-09-16T10:26:10.802958+0000","last_clean":"2024-09-16T10:26:10.802958+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:10.802958+0000","last_undegraded":"2024-09-16T10:26:10.802958+0000","last_fullsized":"2024-09-16T10:26:10.802958+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8562,"log_dups_size":0,"ondisk_log_size":8562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35045376,"num_objects":8556,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4775,"num_write_kb":19100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'8614","reported_seq":8799,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.782039+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:10.782039+0000","last_peered":"2024-09-16T10:26:10.782039+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:10.782039+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8614,"log_dups_size":0,"ondisk_log_size":8614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35266560,"num_objects":8610,"num_object_clones":0,"num_object_copies":17220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4671,"num_write_kb":18684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4275}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'8615","reported_seq":8671,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:10.731810+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:10.731810+0000","last_peered":"2024-09-16T10:26:10.731810+0000","last_clean":"2024-09-16T10:26:10.731810+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:10.731810+0000","last_undegraded":"2024-09-16T10:26:10.731810+0000","last_fullsized":"2024-09-16T10:26:10.731810+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8615,"log_dups_size":0,"ondisk_log_size":8615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35287040,"num_objects":8615,"num_object_clones":0,"num_object_copies":17230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8615,"num_write_kb":34460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8573","reported_seq":8729,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:10.100180+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:10.100180+0000","last_peered":"2024-09-16T10:26:10.100180+0000","last_clean":"2024-09-16T10:26:10.100180+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:10.100180+0000","last_undegraded":"2024-09-16T10:26:10.100180+0000","last_fullsized":"2024-09-16T10:26:10.100180+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35086336,"num_objects":8566,"num_object_clones":0,"num_object_copies":17132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":20860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8456","reported_seq":8619,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.175469+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:10.175469+0000","last_peered":"2024-09-16T10:26:10.175469+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:10.175469+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34615296,"num_objects":8451,"num_object_clones":0,"num_object_copies":16902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5081,"num_write_kb":20324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4234},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'8802","reported_seq":8975,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.002428+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:10.002428+0000","last_peered":"2024-09-16T10:26:10.002428+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:10.002428+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8802,"log_dups_size":0,"ondisk_log_size":8802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36028416,"num_objects":8796,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":20176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4393},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8588","reported_seq":8763,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:10.185486+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:10.185486+0000","last_peered":"2024-09-16T10:26:10.185486+0000","last_clean":"2024-09-16T10:26:10.185486+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:10.185486+0000","last_undegraded":"2024-09-16T10:26:10.185486+0000","last_fullsized":"2024-09-16T10:26:10.185486+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8588,"log_dups_size":0,"ondisk_log_size":8588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35168256,"num_objects":8586,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":17564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8622","reported_seq":17125,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:13.508478+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:13.508478+0000","last_peered":"2024-09-16T10:26:13.508478+0000","last_clean":"2024-09-16T10:26:13.508478+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:13.508478+0000","last_undegraded":"2024-09-16T10:26:13.508478+0000","last_fullsized":"2024-09-16T10:26:13.508478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8622,"log_dups_size":0,"ondisk_log_size":8622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35288006,"num_objects":8617,"num_object_clones":0,"num_object_copies":17234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":34464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'8777","reported_seq":8855,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.173317+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:10.173317+0000","last_peered":"2024-09-16T10:26:10.173317+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:10.173317+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8777,"log_dups_size":0,"ondisk_log_size":8777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35918790,"num_objects":8771,"num_object_clones":0,"num_object_copies":17542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8775,"num_write_kb":35080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4337},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8536","reported_seq":8641,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.754499+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:10.754499+0000","last_peered":"2024-09-16T10:26:10.754499+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:10.754499+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34943942,"num_objects":8533,"num_object_clones":0,"num_object_copies":17066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8534,"num_write_kb":34127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8536","reported_seq":8622,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.784845+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:10.784845+0000","last_peered":"2024-09-16T10:26:10.784845+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:10.784845+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34939846,"num_objects":8532,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8534,"num_write_kb":34123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8352","reported_seq":8431,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.137054+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:10.137054+0000","last_peered":"2024-09-16T10:26:10.137054+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:10.137054+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8352,"log_dups_size":0,"ondisk_log_size":8352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34202083,"num_objects":8351,"num_object_clones":0,"num_object_copies":16702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8351,"num_write_kb":33401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4192},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'8706","reported_seq":8839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.142891+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:10.142891+0000","last_peered":"2024-09-16T10:26:10.142891+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:10.142891+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8706,"log_dups_size":0,"ondisk_log_size":8706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35628457,"num_objects":8701,"num_object_clones":0,"num_object_copies":17402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8703,"num_write_kb":34796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4364},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8402","reported_seq":8480,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.014065+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:10.014065+0000","last_peered":"2024-09-16T10:26:10.014065+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:10.014065+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8402,"log_dups_size":0,"ondisk_log_size":8402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34399174,"num_objects":8400,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":33594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4233},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'8726","reported_seq":8818,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.154679+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:10.154679+0000","last_peered":"2024-09-16T10:26:10.154679+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:10.154679+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8726,"log_dups_size":0,"ondisk_log_size":8726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35694476,"num_objects":8718,"num_object_clones":0,"num_object_copies":17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8723,"num_write_kb":34863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4323},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":562651638,"num_objects":137382,"num_object_clones":0,"num_object_copies":274764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3983,"num_objects_unfound":0,"num_objects_dirty":137382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115011,"num_write_kb":459932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4427,"num_bytes_recovered":18051555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1147486208,"data_stored":1147403109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137446,"ondisk_log_size":137446,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018924,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667556,"kb_used_data":411164,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93704284,"statfs":{"total":96636764160,"available":95953186816,"internally_reserved":0,"allocated":421031936,"data_stored":420778302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":262532784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.2890000000000001},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.2160000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":2.2559999999999998},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":3.1869999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412596,"kb_used_data":228396,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93959244,"statfs":{"total":96636764160,"available":96214265856,"internally_reserved":0,"allocated":233877504,"data_stored":233551610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":188608170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":6.282},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.2669999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":1.3109999999999999},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":1.351}]}]},{"osd":0,"up_from":24,"seq":103079215152,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":786388,"kb_used_data":480012,"kb_used_omap":4,"kb_used_meta":306363,"kb_avail":93585452,"statfs":{"total":96636764160,"available":95831502848,"internally_reserved":0,"allocated":491532288,"data_stored":491282268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":313716392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":4.5229999999999997},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.8029999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":7.6390000000000002},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.5709999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":491724800,"data_stored":491721187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234344448,"data_stored":234268575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":421416960,"data_stored":421413347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:15.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:15.555 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 183 20 139667 139647 2.98018 1.79688 0.04062 0.0262069 2024-09-16T10:26:16.072 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:16.073 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:16.098 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-16T10:26:14.619153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":562651657,"num_objects":137384,"num_object_clones":0,"num_object_copies":274768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3983,"num_objects_unfound":0,"num_objects_dirty":137384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115013,"num_write_kb":459934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4428,"num_bytes_recovered":18051574,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137449,"ondisk_log_size":137449,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1866540,"kb_used_data":1119572,"kb_used_omap":12,"kb_used_meta":746931,"kb_avail":281248980,"statfs":{"total":289910292480,"available":287998955520,"internally_reserved":0,"allocated":1146441728,"data_stored":1145612180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":764857346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,11,3,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18300928,"num_objects":4468,"num_object_clones":0,"num_object_copies":8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-208,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4468,"num_write_kb":17872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005467"},"pg_stats":[{"pgid":"2.f","version":"44'8579","reported_seq":9065,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:10.192581+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:10.192581+0000","last_peered":"2024-09-16T10:26:10.192581+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:10.192581+0000","last_undegraded":"2024-09-16T10:26:10.192581+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35139584,"num_objects":8579,"num_object_clones":0,"num_object_copies":17158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3983,"num_objects_unfound":0,"num_objects_dirty":8579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8579,"num_write_kb":34316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":856064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4596},{"shards":"0,2","objects":3983}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8562","reported_seq":8718,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:10.802958+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:10.802958+0000","last_peered":"2024-09-16T10:26:10.802958+0000","last_clean":"2024-09-16T10:26:10.802958+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:10.802958+0000","last_undegraded":"2024-09-16T10:26:10.802958+0000","last_fullsized":"2024-09-16T10:26:10.802958+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8562,"log_dups_size":0,"ondisk_log_size":8562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35045376,"num_objects":8556,"num_object_clones":0,"num_object_copies":17112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4775,"num_write_kb":19100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'8614","reported_seq":8799,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.782039+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:10.782039+0000","last_peered":"2024-09-16T10:26:10.782039+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:10.782039+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8614,"log_dups_size":0,"ondisk_log_size":8614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35266560,"num_objects":8610,"num_object_clones":0,"num_object_copies":17220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4671,"num_write_kb":18684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4275}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'8615","reported_seq":8671,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:10.731810+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:10.731810+0000","last_peered":"2024-09-16T10:26:10.731810+0000","last_clean":"2024-09-16T10:26:10.731810+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:10.731810+0000","last_undegraded":"2024-09-16T10:26:10.731810+0000","last_fullsized":"2024-09-16T10:26:10.731810+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8615,"log_dups_size":0,"ondisk_log_size":8615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35287040,"num_objects":8615,"num_object_clones":0,"num_object_copies":17230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8615,"num_write_kb":34460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8573","reported_seq":8729,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:10.100180+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:10.100180+0000","last_peered":"2024-09-16T10:26:10.100180+0000","last_clean":"2024-09-16T10:26:10.100180+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:10.100180+0000","last_undegraded":"2024-09-16T10:26:10.100180+0000","last_fullsized":"2024-09-16T10:26:10.100180+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8573,"log_dups_size":0,"ondisk_log_size":8573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35086336,"num_objects":8566,"num_object_clones":0,"num_object_copies":17132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5215,"num_write_kb":20860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8456","reported_seq":8619,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.175469+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:10.175469+0000","last_peered":"2024-09-16T10:26:10.175469+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:10.175469+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8456,"log_dups_size":0,"ondisk_log_size":8456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34615296,"num_objects":8451,"num_object_clones":0,"num_object_copies":16902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5081,"num_write_kb":20324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4234},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'8802","reported_seq":8975,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.002428+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:10.002428+0000","last_peered":"2024-09-16T10:26:10.002428+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:10.002428+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8802,"log_dups_size":0,"ondisk_log_size":8802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36028416,"num_objects":8796,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5044,"num_write_kb":20176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4393},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8588","reported_seq":8763,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:10.185486+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:10.185486+0000","last_peered":"2024-09-16T10:26:10.185486+0000","last_clean":"2024-09-16T10:26:10.185486+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:10.185486+0000","last_undegraded":"2024-09-16T10:26:10.185486+0000","last_fullsized":"2024-09-16T10:26:10.185486+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8588,"log_dups_size":0,"ondisk_log_size":8588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35168256,"num_objects":8586,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":17564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8622","reported_seq":17125,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:13.508478+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:13.508478+0000","last_peered":"2024-09-16T10:26:13.508478+0000","last_clean":"2024-09-16T10:26:13.508478+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:13.508478+0000","last_undegraded":"2024-09-16T10:26:13.508478+0000","last_fullsized":"2024-09-16T10:26:13.508478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8622,"log_dups_size":0,"ondisk_log_size":8622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35288006,"num_objects":8617,"num_object_clones":0,"num_object_copies":17234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":34464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'8777","reported_seq":8855,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.173317+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:10.173317+0000","last_peered":"2024-09-16T10:26:10.173317+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:10.173317+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8777,"log_dups_size":0,"ondisk_log_size":8777,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35918790,"num_objects":8771,"num_object_clones":0,"num_object_copies":17542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8775,"num_write_kb":35080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4337},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8536","reported_seq":8641,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.754499+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:10.754499+0000","last_peered":"2024-09-16T10:26:10.754499+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:10.754499+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34943942,"num_objects":8533,"num_object_clones":0,"num_object_copies":17066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8534,"num_write_kb":34127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4256}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8536","reported_seq":8622,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.784845+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:10.784845+0000","last_peered":"2024-09-16T10:26:10.784845+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:10.784845+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8536,"log_dups_size":0,"ondisk_log_size":8536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34939846,"num_objects":8532,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8534,"num_write_kb":34123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4340}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8352","reported_seq":8431,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.137054+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:10.137054+0000","last_peered":"2024-09-16T10:26:10.137054+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:10.137054+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8352,"log_dups_size":0,"ondisk_log_size":8352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34202083,"num_objects":8351,"num_object_clones":0,"num_object_copies":16702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8351,"num_write_kb":33401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4192},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'8706","reported_seq":8839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.142891+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:10.142891+0000","last_peered":"2024-09-16T10:26:10.142891+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:10.142891+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8706,"log_dups_size":0,"ondisk_log_size":8706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35628457,"num_objects":8701,"num_object_clones":0,"num_object_copies":17402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8703,"num_write_kb":34796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4364},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8402","reported_seq":8480,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.014065+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:10.014065+0000","last_peered":"2024-09-16T10:26:10.014065+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:10.014065+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8402,"log_dups_size":0,"ondisk_log_size":8402,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34399174,"num_objects":8400,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8400,"num_write_kb":33594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4233},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'8726","reported_seq":8818,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:10.154679+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:10.154679+0000","last_peered":"2024-09-16T10:26:10.154679+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:10.154679+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8726,"log_dups_size":0,"ondisk_log_size":8726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35694476,"num_objects":8718,"num_object_clones":0,"num_object_copies":17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8723,"num_write_kb":34863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4323},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":562651638,"num_objects":137382,"num_object_clones":0,"num_object_copies":274764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3983,"num_objects_unfound":0,"num_objects_dirty":137382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115011,"num_write_kb":459932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4427,"num_bytes_recovered":18051555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1147486208,"data_stored":1147403109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137446,"ondisk_log_size":137446,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018924,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667556,"kb_used_data":411164,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93704284,"statfs":{"total":96636764160,"available":95953186816,"internally_reserved":0,"allocated":421031936,"data_stored":420778302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":262532784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.2890000000000001},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.2160000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":2.2559999999999998},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":3.1869999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412596,"kb_used_data":228396,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93959244,"statfs":{"total":96636764160,"available":96214265856,"internally_reserved":0,"allocated":233877504,"data_stored":233551610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":188608170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":6.282},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.2669999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":1.3109999999999999},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":1.351}]}]},{"osd":0,"up_from":24,"seq":103079215152,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":786388,"kb_used_data":480012,"kb_used_omap":4,"kb_used_meta":306363,"kb_avail":93585452,"statfs":{"total":96636764160,"available":95831502848,"internally_reserved":0,"allocated":491532288,"data_stored":491282268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":313716392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,2,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":4.5229999999999997},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.8029999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":7.6390000000000002},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.5709999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":491724800,"data_stored":491721187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234344448,"data_stored":234268575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":421416960,"data_stored":421413347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:16.099 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:16.099 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'8579", 'reported_seq': 9065, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:10.192581+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:10.192581+0000', 'last_peered': '2024-09-16T10:26:10.192581+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:10.192581+0000', 'last_undegraded': '2024-09-16T10:26:10.192581+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8579, 'log_dups_size': 0, 'ondisk_log_size': 8579, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35139584, 'num_objects': 8579, 'num_object_clones': 0, 'num_object_copies': 17158, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3983, 'num_objects_unfound': 0, 'num_objects_dirty': 8579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8579, 'num_write_kb': 34316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 209, 'num_bytes_recovered': 856064, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4596}, {'shards': '0,2', 'objects': 3983}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:16.099 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:16.100 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'8614", 'reported_seq': 8799, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:10.782039+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:10.782039+0000', 'last_peered': '2024-09-16T10:26:10.782039+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:10.782039+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 8614, 'log_dups_size': 0, 'ondisk_log_size': 8614, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35266560, 'num_objects': 8610, 'num_object_clones': 0, 'num_object_copies': 17220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8610, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4671, 'num_write_kb': 18684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 4275}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:16.100 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:16.100 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'8456", 'reported_seq': 8619, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:10.175469+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:10.175469+0000', 'last_peered': '2024-09-16T10:26:10.175469+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:10.175469+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 8456, 'log_dups_size': 0, 'ondisk_log_size': 8456, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34615296, 'num_objects': 8451, 'num_object_clones': 0, 'num_object_copies': 16902, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5081, 'num_write_kb': 20324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4234}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:16.100 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:16.100 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'8802", 'reported_seq': 8975, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:10.002428+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:10.002428+0000', 'last_peered': '2024-09-16T10:26:10.002428+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:10.002428+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 8802, 'log_dups_size': 0, 'ondisk_log_size': 8802, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36028416, 'num_objects': 8796, 'num_object_clones': 0, 'num_object_copies': 17592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5044, 'num_write_kb': 20176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4393}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:16.100 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:16.100 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'8777", 'reported_seq': 8855, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:10.173317+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:10.173317+0000', 'last_peered': '2024-09-16T10:26:10.173317+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:10.173317+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8777, 'log_dups_size': 0, 'ondisk_log_size': 8777, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35918790, 'num_objects': 8771, 'num_object_clones': 0, 'num_object_copies': 17542, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8771, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8775, 'num_write_kb': 35080, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4337}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:16.100 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:16.100 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'8536", 'reported_seq': 8641, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:10.754499+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:10.754499+0000', 'last_peered': '2024-09-16T10:26:10.754499+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:10.754499+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 8536, 'log_dups_size': 0, 'ondisk_log_size': 8536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34943942, 'num_objects': 8533, 'num_object_clones': 0, 'num_object_copies': 17066, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8534, 'num_write_kb': 34127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 4256}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:16.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:16.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'8536", 'reported_seq': 8622, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:10.784845+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:10.784845+0000', 'last_peered': '2024-09-16T10:26:10.784845+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:10.784845+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8536, 'log_dups_size': 0, 'ondisk_log_size': 8536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34939846, 'num_objects': 8532, 'num_object_clones': 0, 'num_object_copies': 17064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8532, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8534, 'num_write_kb': 34123, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 4340}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:16.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:16.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'8352", 'reported_seq': 8431, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:10.137054+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:10.137054+0000', 'last_peered': '2024-09-16T10:26:10.137054+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:10.137054+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8352, 'log_dups_size': 0, 'ondisk_log_size': 8352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34202083, 'num_objects': 8351, 'num_object_clones': 0, 'num_object_copies': 16702, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8351, 'num_write_kb': 33401, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4192}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:16.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:16.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'8706", 'reported_seq': 8839, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:10.142891+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:26:10.142891+0000', 'last_peered': '2024-09-16T10:26:10.142891+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:26:10.142891+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 8706, 'log_dups_size': 0, 'ondisk_log_size': 8706, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35628457, 'num_objects': 8701, 'num_object_clones': 0, 'num_object_copies': 17402, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8701, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8703, 'num_write_kb': 34796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4364}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:16.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:16.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'8402", 'reported_seq': 8480, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:10.014065+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:26:10.014065+0000', 'last_peered': '2024-09-16T10:26:10.014065+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:26:10.014065+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8402, 'log_dups_size': 0, 'ondisk_log_size': 8402, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34399174, 'num_objects': 8400, 'num_object_clones': 0, 'num_object_copies': 16800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8400, 'num_write_kb': 33594, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4233}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:16.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:16.101 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'8726", 'reported_seq': 8818, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:10.154679+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:10.154679+0000', 'last_peered': '2024-09-16T10:26:10.154679+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:10.154679+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8726, 'log_dups_size': 0, 'ondisk_log_size': 8726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35694476, 'num_objects': 8718, 'num_object_clones': 0, 'num_object_copies': 17436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8718, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8723, 'num_write_kb': 34863, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4323}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:16.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:16.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 184 20 140089 140069 2.97295 1.64844 0.0301764 0.0262712 2024-09-16T10:26:16.833 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:16.833 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/274768 objects degraded (15.626%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":265,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":137384,"data_bytes":562651657,"bytes_used":1911336960,"bytes_avail":287998955520,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":274768,"degraded_ratio":0.1562627380189833,"misplaced_objects":3983,"misplaced_total":274768,"misplaced_ratio":0.014495865602981424,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":70964,"recovering_keys_per_sec":0,"num_objects_recovered":208,"num_bytes_recovered":851968,"num_keys_recovered":0,"write_bytes_sec":1524382,"read_op_per_sec":0,"write_op_per_sec":372},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:16.858 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:16.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:17.395 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:17.396 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:17.417 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-16T10:26:16.619830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":571568649,"num_objects":139561,"num_object_clones":0,"num_object_copies":279122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3912,"num_objects_unfound":0,"num_objects_dirty":139561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117190,"num_write_kb":468642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4499,"num_bytes_recovered":18342390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139626,"ondisk_log_size":139626,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1886292,"kb_used_data":1131132,"kb_used_omap":12,"kb_used_meta":755123,"kb_avail":281229228,"statfs":{"total":289910292480,"available":287978729472,"internally_reserved":0,"allocated":1158279168,"data_stored":1157449620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":773245954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,5,5,5,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26918912,"num_objects":6572,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-279,"num_objects_unfound":0,"num_objects_dirty":6572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6572,"num_write_kb":26288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":1142784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004884"},"pg_stats":[{"pgid":"2.f","version":"44'8737","reported_seq":9366,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:15.183103+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:15.183103+0000","last_peered":"2024-09-16T10:26:15.183103+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:15.183103+0000","last_undegraded":"2024-09-16T10:26:15.183103+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8737,"log_dups_size":0,"ondisk_log_size":8737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35786752,"num_objects":8737,"num_object_clones":0,"num_object_copies":17474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3912,"num_objects_unfound":0,"num_objects_dirty":8737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8737,"num_write_kb":34948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4825},{"shards":"0,2","objects":3912}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8710","reported_seq":8867,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:15.817143+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:15.817143+0000","last_peered":"2024-09-16T10:26:15.817143+0000","last_clean":"2024-09-16T10:26:15.817143+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:15.817143+0000","last_undegraded":"2024-09-16T10:26:15.817143+0000","last_fullsized":"2024-09-16T10:26:15.817143+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8710,"log_dups_size":0,"ondisk_log_size":8710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35651584,"num_objects":8704,"num_object_clones":0,"num_object_copies":17408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4923,"num_write_kb":19692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'8754","reported_seq":8938,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.818736+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:15.818736+0000","last_peered":"2024-09-16T10:26:15.818736+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:15.818736+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35840000,"num_objects":8750,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4811,"num_write_kb":19244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'8761","reported_seq":8817,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:15.825192+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:15.825192+0000","last_peered":"2024-09-16T10:26:15.825192+0000","last_clean":"2024-09-16T10:26:15.825192+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:15.825192+0000","last_undegraded":"2024-09-16T10:26:15.825192+0000","last_fullsized":"2024-09-16T10:26:15.825192+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8761,"log_dups_size":0,"ondisk_log_size":8761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35885056,"num_objects":8761,"num_object_clones":0,"num_object_copies":17522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8761,"num_write_kb":35044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8716","reported_seq":8872,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:15.178115+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:15.178115+0000","last_peered":"2024-09-16T10:26:15.178115+0000","last_clean":"2024-09-16T10:26:15.178115+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:15.178115+0000","last_undegraded":"2024-09-16T10:26:15.178115+0000","last_fullsized":"2024-09-16T10:26:15.178115+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8716,"log_dups_size":0,"ondisk_log_size":8716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35672064,"num_objects":8709,"num_object_clones":0,"num_object_copies":17418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":21432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8602","reported_seq":8760,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.202808+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:15.202808+0000","last_peered":"2024-09-16T10:26:15.202808+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:15.202808+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8602,"log_dups_size":0,"ondisk_log_size":8602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35213312,"num_objects":8597,"num_object_clones":0,"num_object_copies":17194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5227,"num_write_kb":20908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4380},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'8953","reported_seq":9126,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.132564+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:15.132564+0000","last_peered":"2024-09-16T10:26:15.132564+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:15.132564+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":8953,"log_dups_size":0,"ondisk_log_size":8953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36646912,"num_objects":8947,"num_object_clones":0,"num_object_copies":17894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5195,"num_write_kb":20780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4544},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8721","reported_seq":8896,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:15.163227+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:15.163227+0000","last_peered":"2024-09-16T10:26:15.163227+0000","last_clean":"2024-09-16T10:26:15.163227+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:15.163227+0000","last_undegraded":"2024-09-16T10:26:15.163227+0000","last_fullsized":"2024-09-16T10:26:15.163227+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35713024,"num_objects":8719,"num_object_clones":0,"num_object_copies":17438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4524,"num_write_kb":18096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8622","reported_seq":17125,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:13.508478+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:13.508478+0000","last_peered":"2024-09-16T10:26:13.508478+0000","last_clean":"2024-09-16T10:26:13.508478+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:13.508478+0000","last_undegraded":"2024-09-16T10:26:13.508478+0000","last_fullsized":"2024-09-16T10:26:13.508478+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8622,"log_dups_size":0,"ondisk_log_size":8622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35288006,"num_objects":8617,"num_object_clones":0,"num_object_copies":17234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":34464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'8933","reported_seq":9011,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.202765+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:15.202765+0000","last_peered":"2024-09-16T10:26:15.202765+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:15.202765+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8933,"log_dups_size":0,"ondisk_log_size":8933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36557766,"num_objects":8927,"num_object_clones":0,"num_object_copies":17854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8931,"num_write_kb":35704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4493},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8675","reported_seq":8780,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.756544+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:15.756544+0000","last_peered":"2024-09-16T10:26:15.756544+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:15.756544+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8675,"log_dups_size":0,"ondisk_log_size":8675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35513286,"num_objects":8672,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8673,"num_write_kb":34683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8694","reported_seq":8780,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.785923+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:15.785923+0000","last_peered":"2024-09-16T10:26:15.785923+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:15.785923+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8694,"log_dups_size":0,"ondisk_log_size":8694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35587014,"num_objects":8690,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8692,"num_write_kb":34755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4498}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8493","reported_seq":8572,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.172589+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:15.172589+0000","last_peered":"2024-09-16T10:26:15.172589+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:15.172589+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8493,"log_dups_size":0,"ondisk_log_size":8493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34779619,"num_objects":8492,"num_object_clones":0,"num_object_copies":16984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8492,"num_write_kb":33965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4333},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'8855","reported_seq":8988,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:14.994688+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:14.994688+0000","last_peered":"2024-09-16T10:26:14.994688+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:14.994688+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8855,"log_dups_size":0,"ondisk_log_size":8855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36238761,"num_objects":8850,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8852,"num_write_kb":35392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4513},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8537","reported_seq":8617,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.116545+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:15.116545+0000","last_peered":"2024-09-16T10:26:15.116545+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:15.116545+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8537,"log_dups_size":0,"ondisk_log_size":8537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34952134,"num_objects":8535,"num_object_clones":0,"num_object_copies":17070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8535,"num_write_kb":34134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4368},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'8860","reported_seq":8952,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.135191+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:15.135191+0000","last_peered":"2024-09-16T10:26:15.135191+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:15.135191+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8860,"log_dups_size":0,"ondisk_log_size":8860,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36243340,"num_objects":8852,"num_object_clones":0,"num_object_copies":17704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8857,"num_write_kb":35399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4457},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":571568630,"num_objects":139559,"num_object_clones":0,"num_object_copies":279118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3912,"num_objects_unfound":0,"num_objects_dirty":139559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":117188,"num_write_kb":468640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4498,"num_bytes_recovered":18342371,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1159348224,"data_stored":1159265125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139623,"ondisk_log_size":139623,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018925,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676164,"kb_used_data":415676,"kb_used_omap":4,"kb_used_meta":260475,"kb_avail":93695676,"statfs":{"total":96636764160,"available":95944372224,"internally_reserved":0,"allocated":425652224,"data_stored":425398590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":266727088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.2599999999999998},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.74199999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":9.6059999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691869,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412596,"kb_used_data":228396,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93959244,"statfs":{"total":96636764160,"available":96214265856,"internally_reserved":0,"allocated":233877504,"data_stored":233551610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":188608170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":6.282},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.2669999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":1.3109999999999999},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":1.351}]}]},{"osd":0,"up_from":24,"seq":103079215153,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797532,"kb_used_data":487060,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93574308,"statfs":{"total":96636764160,"available":95820091392,"internally_reserved":0,"allocated":498749440,"data_stored":498499420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":317910696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":4.556},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":4.4880000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":6.96},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":4.5309999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":498843648,"data_stored":498840035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234344448,"data_stored":234268575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":426160128,"data_stored":426156515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:17.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 185 20 140520 140500 2.96598 1.68359 0.0215609 0.0263302 2024-09-16T10:26:18.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 186 20 141009 140989 2.9603 1.91016 0.0220588 0.0263835 2024-09-16T10:26:19.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 187 20 141474 141454 2.95418 1.81641 0.0311594 0.0264373 2024-09-16T10:26:20.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:20.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 188 20 141934 141914 2.94803 1.79688 0.020511 0.0264919 2024-09-16T10:26:20.975 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:20.976 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:21.004 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-16T10:26:20.621104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":577946121,"num_objects":141118,"num_object_clones":0,"num_object_copies":282236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":141118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118747,"num_write_kb":474870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4573,"num_bytes_recovered":18645494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141183,"ondisk_log_size":141183,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1883840,"kb_used_data":1145064,"kb_used_omap":12,"kb_used_meta":738739,"kb_avail":281231680,"statfs":{"total":289910292480,"available":287981240320,"internally_reserved":0,"allocated":1172545536,"data_stored":1171715988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":756468738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,7,8,5,10,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24731648,"num_objects":6038,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-216,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":24152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004388"},"pg_stats":[{"pgid":"2.f","version":"44'8866","reported_seq":9643,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:20.241065+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:20.241065+0000","last_peered":"2024-09-16T10:26:20.241065+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:20.241065+0000","last_undegraded":"2024-09-16T10:26:20.241065+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8866,"log_dups_size":0,"ondisk_log_size":8866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36315136,"num_objects":8866,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":8866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8866,"num_write_kb":35464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5028},{"shards":"0,2","objects":3838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8710","reported_seq":8867,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:15.817143+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:15.817143+0000","last_peered":"2024-09-16T10:26:15.817143+0000","last_clean":"2024-09-16T10:26:15.817143+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:15.817143+0000","last_undegraded":"2024-09-16T10:26:15.817143+0000","last_fullsized":"2024-09-16T10:26:15.817143+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8710,"log_dups_size":0,"ondisk_log_size":8710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35651584,"num_objects":8704,"num_object_clones":0,"num_object_copies":17408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4923,"num_write_kb":19692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'8754","reported_seq":8938,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.818736+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:15.818736+0000","last_peered":"2024-09-16T10:26:15.818736+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:15.818736+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35840000,"num_objects":8750,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4811,"num_write_kb":19244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'8761","reported_seq":8817,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:15.825192+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:15.825192+0000","last_peered":"2024-09-16T10:26:15.825192+0000","last_clean":"2024-09-16T10:26:15.825192+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:15.825192+0000","last_undegraded":"2024-09-16T10:26:15.825192+0000","last_fullsized":"2024-09-16T10:26:15.825192+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8761,"log_dups_size":0,"ondisk_log_size":8761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35885056,"num_objects":8761,"num_object_clones":0,"num_object_copies":17522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8761,"num_write_kb":35044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8843","reported_seq":8999,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.242686+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:20.242686+0000","last_peered":"2024-09-16T10:26:20.242686+0000","last_clean":"2024-09-16T10:26:20.242686+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:20.242686+0000","last_undegraded":"2024-09-16T10:26:20.242686+0000","last_fullsized":"2024-09-16T10:26:20.242686+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8843,"log_dups_size":0,"ondisk_log_size":8843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36192256,"num_objects":8836,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":21940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8743","reported_seq":8906,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.204886+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:20.204886+0000","last_peered":"2024-09-16T10:26:20.204886+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:20.204886+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8743,"log_dups_size":0,"ondisk_log_size":8743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35790848,"num_objects":8738,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5368,"num_write_kb":21472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4521},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9106","reported_seq":9279,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.140013+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:20.140013+0000","last_peered":"2024-09-16T10:26:20.140013+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:20.140013+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37273600,"num_objects":9100,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5348,"num_write_kb":21392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4697},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8840","reported_seq":9015,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.162037+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:20.162037+0000","last_peered":"2024-09-16T10:26:20.162037+0000","last_clean":"2024-09-16T10:26:20.162037+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:20.162037+0000","last_undegraded":"2024-09-16T10:26:20.162037+0000","last_fullsized":"2024-09-16T10:26:20.162037+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36200448,"num_objects":8838,"num_object_clones":0,"num_object_copies":17676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4643,"num_write_kb":18572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8795","reported_seq":17298,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:18.531978+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:18.531978+0000","last_peered":"2024-09-16T10:26:18.531978+0000","last_clean":"2024-09-16T10:26:18.531978+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:18.531978+0000","last_undegraded":"2024-09-16T10:26:18.531978+0000","last_fullsized":"2024-09-16T10:26:18.531978+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8795,"log_dups_size":0,"ondisk_log_size":8795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35996614,"num_objects":8790,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8793,"num_write_kb":35156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9077","reported_seq":9155,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.206321+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:20.206321+0000","last_peered":"2024-09-16T10:26:20.206321+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:20.206321+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9077,"log_dups_size":0,"ondisk_log_size":9077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37147590,"num_objects":9071,"num_object_clones":0,"num_object_copies":18142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9075,"num_write_kb":36280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4637},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8675","reported_seq":8780,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.756544+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:15.756544+0000","last_peered":"2024-09-16T10:26:15.756544+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:15.756544+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8675,"log_dups_size":0,"ondisk_log_size":8675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35513286,"num_objects":8672,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8673,"num_write_kb":34683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8694","reported_seq":8780,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.785923+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:15.785923+0000","last_peered":"2024-09-16T10:26:15.785923+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:15.785923+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8694,"log_dups_size":0,"ondisk_log_size":8694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35587014,"num_objects":8690,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8692,"num_write_kb":34755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4498}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8646","reported_seq":8724,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.123458+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:20.123458+0000","last_peered":"2024-09-16T10:26:20.123458+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:20.123458+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8646,"log_dups_size":0,"ondisk_log_size":8646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35406307,"num_objects":8645,"num_object_clones":0,"num_object_copies":17290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8645,"num_write_kb":34577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4486},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'8994","reported_seq":9126,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.248607+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:20.248607+0000","last_peered":"2024-09-16T10:26:20.248607+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:20.248607+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36808105,"num_objects":8989,"num_object_clones":0,"num_object_copies":17978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":35948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4652},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8673","reported_seq":8753,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.195278+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:20.195278+0000","last_peered":"2024-09-16T10:26:20.195278+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:20.195278+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8673,"log_dups_size":0,"ondisk_log_size":8673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35509190,"num_objects":8671,"num_object_clones":0,"num_object_copies":17342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8671,"num_write_kb":34678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4504},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9003","reported_seq":9095,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.232791+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:20.232791+0000","last_peered":"2024-09-16T10:26:20.232791+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:20.232791+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9003,"log_dups_size":0,"ondisk_log_size":9003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36829068,"num_objects":8995,"num_object_clones":0,"num_object_copies":17990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9000,"num_write_kb":35971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4600},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":577946102,"num_objects":141116,"num_object_clones":0,"num_object_copies":282232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":141116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118745,"num_write_kb":474868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4572,"num_bytes_recovered":18645475,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1173680128,"data_stored":1173597029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141180,"ondisk_log_size":141180,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018925,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676164,"kb_used_data":415676,"kb_used_omap":4,"kb_used_meta":260475,"kb_avail":93695676,"statfs":{"total":96636764160,"available":95944372224,"internally_reserved":0,"allocated":425652224,"data_stored":425398590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":266727088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.2599999999999998},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.74199999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":9.6059999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411696,"kb_used_data":235688,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93960144,"statfs":{"total":96636764160,"available":96215187456,"internally_reserved":0,"allocated":241344512,"data_stored":241018618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":180219562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":17.675000000000001},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.93999999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":6.9619999999999997},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":7.0119999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215154,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":795980,"kb_used_data":493700,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93575860,"statfs":{"total":96636764160,"available":95821680640,"internally_reserved":0,"allocated":505548800,"data_stored":505298780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":309522088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,3,3,1,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.5089999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":0.499},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.23200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":505683968,"data_stored":505680355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241836032,"data_stored":241760159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":426160128,"data_stored":426156515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:21.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:21.538 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:21.540 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:21.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 189 20 142354 142334 2.94111 1.64062 0.0727001 0.0265557 2024-09-16T10:26:21.560 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-16T10:26:20.621104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":577946121,"num_objects":141118,"num_object_clones":0,"num_object_copies":282236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":141118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118747,"num_write_kb":474870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4573,"num_bytes_recovered":18645494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141183,"ondisk_log_size":141183,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1883840,"kb_used_data":1145064,"kb_used_omap":12,"kb_used_meta":738739,"kb_avail":281231680,"statfs":{"total":289910292480,"available":287981240320,"internally_reserved":0,"allocated":1172545536,"data_stored":1171715988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":756468738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,7,8,5,10,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24731648,"num_objects":6038,"num_object_clones":0,"num_object_copies":12076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-216,"num_objects_unfound":0,"num_objects_dirty":6038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":24152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004388"},"pg_stats":[{"pgid":"2.f","version":"44'8866","reported_seq":9643,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:20.241065+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:20.241065+0000","last_peered":"2024-09-16T10:26:20.241065+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:20.241065+0000","last_undegraded":"2024-09-16T10:26:20.241065+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8866,"log_dups_size":0,"ondisk_log_size":8866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36315136,"num_objects":8866,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":8866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8866,"num_write_kb":35464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5028},{"shards":"0,2","objects":3838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8710","reported_seq":8867,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:15.817143+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:15.817143+0000","last_peered":"2024-09-16T10:26:15.817143+0000","last_clean":"2024-09-16T10:26:15.817143+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:15.817143+0000","last_undegraded":"2024-09-16T10:26:15.817143+0000","last_fullsized":"2024-09-16T10:26:15.817143+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8710,"log_dups_size":0,"ondisk_log_size":8710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35651584,"num_objects":8704,"num_object_clones":0,"num_object_copies":17408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4923,"num_write_kb":19692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'8754","reported_seq":8938,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.818736+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:15.818736+0000","last_peered":"2024-09-16T10:26:15.818736+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:15.818736+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8754,"log_dups_size":0,"ondisk_log_size":8754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35840000,"num_objects":8750,"num_object_clones":0,"num_object_copies":17500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4811,"num_write_kb":19244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'8761","reported_seq":8817,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:15.825192+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:15.825192+0000","last_peered":"2024-09-16T10:26:15.825192+0000","last_clean":"2024-09-16T10:26:15.825192+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:15.825192+0000","last_undegraded":"2024-09-16T10:26:15.825192+0000","last_fullsized":"2024-09-16T10:26:15.825192+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8761,"log_dups_size":0,"ondisk_log_size":8761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35885056,"num_objects":8761,"num_object_clones":0,"num_object_copies":17522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8761,"num_write_kb":35044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8843","reported_seq":8999,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.242686+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:20.242686+0000","last_peered":"2024-09-16T10:26:20.242686+0000","last_clean":"2024-09-16T10:26:20.242686+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:20.242686+0000","last_undegraded":"2024-09-16T10:26:20.242686+0000","last_fullsized":"2024-09-16T10:26:20.242686+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8843,"log_dups_size":0,"ondisk_log_size":8843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36192256,"num_objects":8836,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":21940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8743","reported_seq":8906,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.204886+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:20.204886+0000","last_peered":"2024-09-16T10:26:20.204886+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:20.204886+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8743,"log_dups_size":0,"ondisk_log_size":8743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35790848,"num_objects":8738,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5368,"num_write_kb":21472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4521},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9106","reported_seq":9279,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.140013+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:20.140013+0000","last_peered":"2024-09-16T10:26:20.140013+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:20.140013+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37273600,"num_objects":9100,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5348,"num_write_kb":21392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4697},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8840","reported_seq":9015,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.162037+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:20.162037+0000","last_peered":"2024-09-16T10:26:20.162037+0000","last_clean":"2024-09-16T10:26:20.162037+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:20.162037+0000","last_undegraded":"2024-09-16T10:26:20.162037+0000","last_fullsized":"2024-09-16T10:26:20.162037+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36200448,"num_objects":8838,"num_object_clones":0,"num_object_copies":17676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4643,"num_write_kb":18572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8795","reported_seq":17298,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:18.531978+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:18.531978+0000","last_peered":"2024-09-16T10:26:18.531978+0000","last_clean":"2024-09-16T10:26:18.531978+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:18.531978+0000","last_undegraded":"2024-09-16T10:26:18.531978+0000","last_fullsized":"2024-09-16T10:26:18.531978+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8795,"log_dups_size":0,"ondisk_log_size":8795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35996614,"num_objects":8790,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8793,"num_write_kb":35156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9077","reported_seq":9155,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.206321+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:20.206321+0000","last_peered":"2024-09-16T10:26:20.206321+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:20.206321+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9077,"log_dups_size":0,"ondisk_log_size":9077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37147590,"num_objects":9071,"num_object_clones":0,"num_object_copies":18142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9075,"num_write_kb":36280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4637},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8675","reported_seq":8780,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.756544+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:15.756544+0000","last_peered":"2024-09-16T10:26:15.756544+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:15.756544+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8675,"log_dups_size":0,"ondisk_log_size":8675,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35513286,"num_objects":8672,"num_object_clones":0,"num_object_copies":17344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8673,"num_write_kb":34683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8694","reported_seq":8780,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:15.785923+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:15.785923+0000","last_peered":"2024-09-16T10:26:15.785923+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:15.785923+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8694,"log_dups_size":0,"ondisk_log_size":8694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35587014,"num_objects":8690,"num_object_clones":0,"num_object_copies":17380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8692,"num_write_kb":34755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4498}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8646","reported_seq":8724,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.123458+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:20.123458+0000","last_peered":"2024-09-16T10:26:20.123458+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:20.123458+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8646,"log_dups_size":0,"ondisk_log_size":8646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35406307,"num_objects":8645,"num_object_clones":0,"num_object_copies":17290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8645,"num_write_kb":34577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4486},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'8994","reported_seq":9126,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.248607+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:20.248607+0000","last_peered":"2024-09-16T10:26:20.248607+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:20.248607+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36808105,"num_objects":8989,"num_object_clones":0,"num_object_copies":17978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":35948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4652},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8673","reported_seq":8753,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.195278+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:20.195278+0000","last_peered":"2024-09-16T10:26:20.195278+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:20.195278+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8673,"log_dups_size":0,"ondisk_log_size":8673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35509190,"num_objects":8671,"num_object_clones":0,"num_object_copies":17342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8671,"num_write_kb":34678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4504},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9003","reported_seq":9095,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.232791+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:20.232791+0000","last_peered":"2024-09-16T10:26:20.232791+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:20.232791+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9003,"log_dups_size":0,"ondisk_log_size":9003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36829068,"num_objects":8995,"num_object_clones":0,"num_object_copies":17990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9000,"num_write_kb":35971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4600},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":577946102,"num_objects":141116,"num_object_clones":0,"num_object_copies":282232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":141116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118745,"num_write_kb":474868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4572,"num_bytes_recovered":18645475,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1173680128,"data_stored":1173597029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141180,"ondisk_log_size":141180,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018925,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676164,"kb_used_data":415676,"kb_used_omap":4,"kb_used_meta":260475,"kb_avail":93695676,"statfs":{"total":96636764160,"available":95944372224,"internally_reserved":0,"allocated":425652224,"data_stored":425398590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":266727088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.2599999999999998},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.74199999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":9.6059999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411696,"kb_used_data":235688,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93960144,"statfs":{"total":96636764160,"available":96215187456,"internally_reserved":0,"allocated":241344512,"data_stored":241018618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":180219562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":17.675000000000001},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.93999999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":6.9619999999999997},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":7.0119999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215154,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":795980,"kb_used_data":493700,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93575860,"statfs":{"total":96636764160,"available":95821680640,"internally_reserved":0,"allocated":505548800,"data_stored":505298780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":309522088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,3,3,1,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.5089999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":0.499},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.23200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":505683968,"data_stored":505680355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241836032,"data_stored":241760159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":426160128,"data_stored":426156515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:21.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:21.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'8866", 'reported_seq': 9643, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:20.241065+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:20.241065+0000', 'last_peered': '2024-09-16T10:26:20.241065+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:20.241065+0000', 'last_undegraded': '2024-09-16T10:26:20.241065+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8866, 'log_dups_size': 0, 'ondisk_log_size': 8866, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36315136, 'num_objects': 8866, 'num_object_clones': 0, 'num_object_copies': 17732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3838, 'num_objects_unfound': 0, 'num_objects_dirty': 8866, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8866, 'num_write_kb': 35464, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 354, 'num_bytes_recovered': 1449984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5028}, {'shards': '0,2', 'objects': 3838}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:21.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:21.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'8754", 'reported_seq': 8938, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:15.818736+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:15.818736+0000', 'last_peered': '2024-09-16T10:26:15.818736+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:15.818736+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 8754, 'log_dups_size': 0, 'ondisk_log_size': 8754, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35840000, 'num_objects': 8750, 'num_object_clones': 0, 'num_object_copies': 17500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4811, 'num_write_kb': 19244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 4415}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:21.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:21.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'8743", 'reported_seq': 8906, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:20.204886+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:20.204886+0000', 'last_peered': '2024-09-16T10:26:20.204886+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:20.204886+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 8743, 'log_dups_size': 0, 'ondisk_log_size': 8743, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35790848, 'num_objects': 8738, 'num_object_clones': 0, 'num_object_copies': 17476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5368, 'num_write_kb': 21472, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4521}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'9106", 'reported_seq': 9279, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:20.140013+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:20.140013+0000', 'last_peered': '2024-09-16T10:26:20.140013+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:20.140013+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 9106, 'log_dups_size': 0, 'ondisk_log_size': 9106, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37273600, 'num_objects': 9100, 'num_object_clones': 0, 'num_object_copies': 18200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9100, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5348, 'num_write_kb': 21392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4697}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'9077", 'reported_seq': 9155, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:20.206321+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:20.206321+0000', 'last_peered': '2024-09-16T10:26:20.206321+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:20.206321+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9077, 'log_dups_size': 0, 'ondisk_log_size': 9077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37147590, 'num_objects': 9071, 'num_object_clones': 0, 'num_object_copies': 18142, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9075, 'num_write_kb': 36280, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4637}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'8675", 'reported_seq': 8780, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:15.756544+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:15.756544+0000', 'last_peered': '2024-09-16T10:26:15.756544+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:15.756544+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 8675, 'log_dups_size': 0, 'ondisk_log_size': 8675, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35513286, 'num_objects': 8672, 'num_object_clones': 0, 'num_object_copies': 17344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8673, 'num_write_kb': 34683, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'8694", 'reported_seq': 8780, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:15.785923+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:15.785923+0000', 'last_peered': '2024-09-16T10:26:15.785923+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:15.785923+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8694, 'log_dups_size': 0, 'ondisk_log_size': 8694, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35587014, 'num_objects': 8690, 'num_object_clones': 0, 'num_object_copies': 17380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8692, 'num_write_kb': 34755, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 4498}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'8646", 'reported_seq': 8724, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:20.123458+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:20.123458+0000', 'last_peered': '2024-09-16T10:26:20.123458+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:20.123458+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8646, 'log_dups_size': 0, 'ondisk_log_size': 8646, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35406307, 'num_objects': 8645, 'num_object_clones': 0, 'num_object_copies': 17290, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8645, 'num_write_kb': 34577, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4486}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:21.563 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'8994", 'reported_seq': 9126, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:20.248607+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:26:20.248607+0000', 'last_peered': '2024-09-16T10:26:20.248607+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:26:20.248607+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 8994, 'log_dups_size': 0, 'ondisk_log_size': 8994, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36808105, 'num_objects': 8989, 'num_object_clones': 0, 'num_object_copies': 17978, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8991, 'num_write_kb': 35948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4652}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:21.564 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:21.564 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'8673", 'reported_seq': 8753, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:20.195278+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:26:20.195278+0000', 'last_peered': '2024-09-16T10:26:20.195278+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:26:20.195278+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8673, 'log_dups_size': 0, 'ondisk_log_size': 8673, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35509190, 'num_objects': 8671, 'num_object_clones': 0, 'num_object_copies': 17342, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8671, 'num_write_kb': 34678, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4504}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:21.564 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:21.564 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'9003", 'reported_seq': 9095, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:20.232791+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:20.232791+0000', 'last_peered': '2024-09-16T10:26:20.232791+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:20.232791+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9003, 'log_dups_size': 0, 'ondisk_log_size': 9003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36829068, 'num_objects': 8995, 'num_object_clones': 0, 'num_object_copies': 17990, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9000, 'num_write_kb': 35971, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4600}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:21.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:22.228 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:22.229 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/282236 objects degraded (15.213%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":271,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":141118,"data_bytes":577946121,"bytes_used":1929052160,"bytes_avail":287981240320,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":282236,"degraded_ratio":0.15212800634929632,"misplaced_objects":3838,"misplaced_total":282236,"misplaced_ratio":0.013598548732266614,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":73701,"recovering_keys_per_sec":0,"num_objects_recovered":216,"num_bytes_recovered":884736,"num_keys_recovered":0,"write_bytes_sec":2060217,"read_op_per_sec":0,"write_op_per_sec":502},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:22.262 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:22.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:22.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 190 20 142757 142737 2.93391 1.57422 0.0239793 0.0266183 2024-09-16T10:26:22.805 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:22.807 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:22.839 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-16T10:26:22.622102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":580993545,"num_objects":141862,"num_object_clones":0,"num_object_copies":283724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":141862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119491,"num_write_kb":477846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4573,"num_bytes_recovered":18645494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141927,"ondisk_log_size":141927,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1872044,"kb_used_data":1149652,"kb_used_omap":12,"kb_used_meta":722355,"kb_avail":281243476,"statfs":{"total":289910292480,"available":287993319424,"internally_reserved":0,"allocated":1177243648,"data_stored":1176414100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":739691522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,6,7,5,8,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21938176,"num_objects":5356,"num_object_clones":0,"num_object_copies":10712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5356,"num_write_kb":21424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":593920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004375"},"pg_stats":[{"pgid":"2.f","version":"44'8866","reported_seq":9643,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:20.241065+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:20.241065+0000","last_peered":"2024-09-16T10:26:20.241065+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:20.241065+0000","last_undegraded":"2024-09-16T10:26:20.241065+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8866,"log_dups_size":0,"ondisk_log_size":8866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36315136,"num_objects":8866,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":8866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8866,"num_write_kb":35464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5028},{"shards":"0,2","objects":3838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8842","reported_seq":8999,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.797110+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:20.797110+0000","last_peered":"2024-09-16T10:26:20.797110+0000","last_clean":"2024-09-16T10:26:20.797110+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:20.797110+0000","last_undegraded":"2024-09-16T10:26:20.797110+0000","last_fullsized":"2024-09-16T10:26:20.797110+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8842,"log_dups_size":0,"ondisk_log_size":8842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36192256,"num_objects":8836,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5055,"num_write_kb":20220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'8915","reported_seq":9099,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.860475+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:20.860475+0000","last_peered":"2024-09-16T10:26:20.860475+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:20.860475+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36499456,"num_objects":8911,"num_object_clones":0,"num_object_copies":17822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4972,"num_write_kb":19888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4576}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'8910","reported_seq":8966,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.857858+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:20.857858+0000","last_peered":"2024-09-16T10:26:20.857858+0000","last_clean":"2024-09-16T10:26:20.857858+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:20.857858+0000","last_undegraded":"2024-09-16T10:26:20.857858+0000","last_fullsized":"2024-09-16T10:26:20.857858+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36495360,"num_objects":8910,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":35640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8843","reported_seq":8999,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.242686+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:20.242686+0000","last_peered":"2024-09-16T10:26:20.242686+0000","last_clean":"2024-09-16T10:26:20.242686+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:20.242686+0000","last_undegraded":"2024-09-16T10:26:20.242686+0000","last_fullsized":"2024-09-16T10:26:20.242686+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8843,"log_dups_size":0,"ondisk_log_size":8843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36192256,"num_objects":8836,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":21940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8743","reported_seq":8906,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.204886+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:20.204886+0000","last_peered":"2024-09-16T10:26:20.204886+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:20.204886+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8743,"log_dups_size":0,"ondisk_log_size":8743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35790848,"num_objects":8738,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5368,"num_write_kb":21472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4521},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9106","reported_seq":9279,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.140013+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:20.140013+0000","last_peered":"2024-09-16T10:26:20.140013+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:20.140013+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37273600,"num_objects":9100,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5348,"num_write_kb":21392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4697},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8840","reported_seq":9015,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.162037+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:20.162037+0000","last_peered":"2024-09-16T10:26:20.162037+0000","last_clean":"2024-09-16T10:26:20.162037+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:20.162037+0000","last_undegraded":"2024-09-16T10:26:20.162037+0000","last_fullsized":"2024-09-16T10:26:20.162037+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36200448,"num_objects":8838,"num_object_clones":0,"num_object_copies":17676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4643,"num_write_kb":18572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8795","reported_seq":17298,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:18.531978+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:18.531978+0000","last_peered":"2024-09-16T10:26:18.531978+0000","last_clean":"2024-09-16T10:26:18.531978+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:18.531978+0000","last_undegraded":"2024-09-16T10:26:18.531978+0000","last_fullsized":"2024-09-16T10:26:18.531978+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8795,"log_dups_size":0,"ondisk_log_size":8795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35996614,"num_objects":8790,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8793,"num_write_kb":35156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9077","reported_seq":9155,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.206321+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:20.206321+0000","last_peered":"2024-09-16T10:26:20.206321+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:20.206321+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9077,"log_dups_size":0,"ondisk_log_size":9077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37147590,"num_objects":9071,"num_object_clones":0,"num_object_copies":18142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9075,"num_write_kb":36280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4637},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8824","reported_seq":8929,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.857627+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:20.857627+0000","last_peered":"2024-09-16T10:26:20.857627+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:20.857627+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8824,"log_dups_size":0,"ondisk_log_size":8824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36123590,"num_objects":8821,"num_object_clones":0,"num_object_copies":17642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8822,"num_write_kb":35279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8847","reported_seq":8933,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.820964+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:20.820964+0000","last_peered":"2024-09-16T10:26:20.820964+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:20.820964+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8847,"log_dups_size":0,"ondisk_log_size":8847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36213702,"num_objects":8843,"num_object_clones":0,"num_object_copies":17686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8845,"num_write_kb":35367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4651}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8646","reported_seq":8724,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.123458+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:20.123458+0000","last_peered":"2024-09-16T10:26:20.123458+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:20.123458+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8646,"log_dups_size":0,"ondisk_log_size":8646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35406307,"num_objects":8645,"num_object_clones":0,"num_object_copies":17290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8645,"num_write_kb":34577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4486},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'8994","reported_seq":9126,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.248607+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:20.248607+0000","last_peered":"2024-09-16T10:26:20.248607+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:20.248607+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36808105,"num_objects":8989,"num_object_clones":0,"num_object_copies":17978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":35948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4652},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8673","reported_seq":8753,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.195278+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:20.195278+0000","last_peered":"2024-09-16T10:26:20.195278+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:20.195278+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8673,"log_dups_size":0,"ondisk_log_size":8673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35509190,"num_objects":8671,"num_object_clones":0,"num_object_copies":17342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8671,"num_write_kb":34678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4504},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9003","reported_seq":9095,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.232791+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:20.232791+0000","last_peered":"2024-09-16T10:26:20.232791+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:20.232791+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9003,"log_dups_size":0,"ondisk_log_size":9003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36829068,"num_objects":8995,"num_object_clones":0,"num_object_copies":17990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9000,"num_write_kb":35971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4600},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":580993526,"num_objects":141860,"num_object_clones":0,"num_object_copies":283720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":141860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119489,"num_write_kb":477844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4572,"num_bytes_recovered":18645475,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1178292224,"data_stored":1178209125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141924,"ondisk_log_size":141924,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018926,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664368,"kb_used_data":420264,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93707472,"statfs":{"total":96636764160,"available":95956451328,"internally_reserved":0,"allocated":430350336,"data_stored":430096702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":249949872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.31},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.056}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":2.1560000000000001},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.44900000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":411696,"kb_used_data":235688,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93960144,"statfs":{"total":96636764160,"available":96215187456,"internally_reserved":0,"allocated":241344512,"data_stored":241018618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":180219562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":17.675000000000001},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.93999999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":6.9619999999999997},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":7.0119999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215154,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":795980,"kb_used_data":493700,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93575860,"statfs":{"total":96636764160,"available":95821680640,"internally_reserved":0,"allocated":505548800,"data_stored":505298780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":309522088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,3,3,1,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.5089999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":0.499},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.23200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":505683968,"data_stored":505680355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":241836032,"data_stored":241760159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":430772224,"data_stored":430768611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:23.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 191 20 143215 143195 2.92792 1.78906 0.0222459 0.0266746 2024-09-16T10:26:24.556 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 192 20 143696 143676 2.92246 1.87891 0.0250671 0.0267248 2024-09-16T10:26:25.559 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 193 20 144177 144157 2.91701 1.87891 0.0419944 0.0267751 2024-09-16T10:26:25.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:26.388 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:26.389 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:26.414 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-16T10:26:24.622833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":581480969,"num_objects":141981,"num_object_clones":0,"num_object_copies":283962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":141981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119610,"num_write_kb":478322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4573,"num_bytes_recovered":18645494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142046,"ondisk_log_size":142046,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1883088,"kb_used_data":1156600,"kb_used_omap":12,"kb_used_meta":726451,"kb_avail":281232432,"statfs":{"total":289910292480,"available":287982010368,"internally_reserved":0,"allocated":1184358400,"data_stored":1183528852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":743885826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,6,9,2,8,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19410944,"num_objects":4739,"num_object_clones":0,"num_object_copies":9478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4739,"num_write_kb":18956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":593920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004356"},"pg_stats":[{"pgid":"2.f","version":"44'8866","reported_seq":9643,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:20.241065+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:20.241065+0000","last_peered":"2024-09-16T10:26:20.241065+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:20.241065+0000","last_undegraded":"2024-09-16T10:26:20.241065+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8866,"log_dups_size":0,"ondisk_log_size":8866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36315136,"num_objects":8866,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":8866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8866,"num_write_kb":35464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5028},{"shards":"0,2","objects":3838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8842","reported_seq":8999,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.797110+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:20.797110+0000","last_peered":"2024-09-16T10:26:20.797110+0000","last_clean":"2024-09-16T10:26:20.797110+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:20.797110+0000","last_undegraded":"2024-09-16T10:26:20.797110+0000","last_fullsized":"2024-09-16T10:26:20.797110+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8842,"log_dups_size":0,"ondisk_log_size":8842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36192256,"num_objects":8836,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5055,"num_write_kb":20220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'8915","reported_seq":9099,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.860475+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:20.860475+0000","last_peered":"2024-09-16T10:26:20.860475+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:20.860475+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":8915,"log_dups_size":0,"ondisk_log_size":8915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36499456,"num_objects":8911,"num_object_clones":0,"num_object_copies":17822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4972,"num_write_kb":19888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4576}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'8910","reported_seq":8966,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.857858+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:20.857858+0000","last_peered":"2024-09-16T10:26:20.857858+0000","last_clean":"2024-09-16T10:26:20.857858+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:20.857858+0000","last_undegraded":"2024-09-16T10:26:20.857858+0000","last_fullsized":"2024-09-16T10:26:20.857858+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36495360,"num_objects":8910,"num_object_clones":0,"num_object_copies":17820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":35640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8843","reported_seq":8999,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.242686+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:20.242686+0000","last_peered":"2024-09-16T10:26:20.242686+0000","last_clean":"2024-09-16T10:26:20.242686+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:20.242686+0000","last_undegraded":"2024-09-16T10:26:20.242686+0000","last_fullsized":"2024-09-16T10:26:20.242686+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8843,"log_dups_size":0,"ondisk_log_size":8843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36192256,"num_objects":8836,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5485,"num_write_kb":21940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8743","reported_seq":8906,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.204886+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:20.204886+0000","last_peered":"2024-09-16T10:26:20.204886+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:20.204886+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8743,"log_dups_size":0,"ondisk_log_size":8743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35790848,"num_objects":8738,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5368,"num_write_kb":21472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4521},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9106","reported_seq":9279,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.140013+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:20.140013+0000","last_peered":"2024-09-16T10:26:20.140013+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:20.140013+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37273600,"num_objects":9100,"num_object_clones":0,"num_object_copies":18200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5348,"num_write_kb":21392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4697},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8840","reported_seq":9015,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:20.162037+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:20.162037+0000","last_peered":"2024-09-16T10:26:20.162037+0000","last_clean":"2024-09-16T10:26:20.162037+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:20.162037+0000","last_undegraded":"2024-09-16T10:26:20.162037+0000","last_fullsized":"2024-09-16T10:26:20.162037+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8840,"log_dups_size":0,"ondisk_log_size":8840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36200448,"num_objects":8838,"num_object_clones":0,"num_object_copies":17676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4643,"num_write_kb":18572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8914","reported_seq":17417,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:23.549973+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:23.549973+0000","last_peered":"2024-09-16T10:26:23.549973+0000","last_clean":"2024-09-16T10:26:23.549973+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:23.549973+0000","last_undegraded":"2024-09-16T10:26:23.549973+0000","last_fullsized":"2024-09-16T10:26:23.549973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8914,"log_dups_size":0,"ondisk_log_size":8914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36484038,"num_objects":8909,"num_object_clones":0,"num_object_copies":17818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":35632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9077","reported_seq":9155,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.206321+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:20.206321+0000","last_peered":"2024-09-16T10:26:20.206321+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:20.206321+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9077,"log_dups_size":0,"ondisk_log_size":9077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37147590,"num_objects":9071,"num_object_clones":0,"num_object_copies":18142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9075,"num_write_kb":36280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4637},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8824","reported_seq":8929,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.857627+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:20.857627+0000","last_peered":"2024-09-16T10:26:20.857627+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:20.857627+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8824,"log_dups_size":0,"ondisk_log_size":8824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36123590,"num_objects":8821,"num_object_clones":0,"num_object_copies":17642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8822,"num_write_kb":35279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8847","reported_seq":8933,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.820964+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:20.820964+0000","last_peered":"2024-09-16T10:26:20.820964+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:20.820964+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8847,"log_dups_size":0,"ondisk_log_size":8847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36213702,"num_objects":8843,"num_object_clones":0,"num_object_copies":17686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8845,"num_write_kb":35367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4651}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8646","reported_seq":8724,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.123458+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:20.123458+0000","last_peered":"2024-09-16T10:26:20.123458+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:20.123458+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8646,"log_dups_size":0,"ondisk_log_size":8646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35406307,"num_objects":8645,"num_object_clones":0,"num_object_copies":17290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8645,"num_write_kb":34577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4486},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'8994","reported_seq":9126,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.248607+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:20.248607+0000","last_peered":"2024-09-16T10:26:20.248607+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:20.248607+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36808105,"num_objects":8989,"num_object_clones":0,"num_object_copies":17978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8991,"num_write_kb":35948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4652},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8673","reported_seq":8753,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.195278+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:20.195278+0000","last_peered":"2024-09-16T10:26:20.195278+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:20.195278+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8673,"log_dups_size":0,"ondisk_log_size":8673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35509190,"num_objects":8671,"num_object_clones":0,"num_object_copies":17342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8671,"num_write_kb":34678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4504},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9003","reported_seq":9095,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:20.232791+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:20.232791+0000","last_peered":"2024-09-16T10:26:20.232791+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:20.232791+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9003,"log_dups_size":0,"ondisk_log_size":9003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36829068,"num_objects":8995,"num_object_clones":0,"num_object_copies":17990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9000,"num_write_kb":35971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4600},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":581480950,"num_objects":141979,"num_object_clones":0,"num_object_copies":283958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3838,"num_objects_unfound":0,"num_objects_dirty":141979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119608,"num_write_kb":478320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4572,"num_bytes_recovered":18645475,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1185722368,"data_stored":1185639269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142043,"ondisk_log_size":142043,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018926,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664368,"kb_used_data":420264,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93707472,"statfs":{"total":96636764160,"available":95956451328,"internally_reserved":0,"allocated":430350336,"data_stored":430096702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":249949872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.31},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.056}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":2.1560000000000001},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.44900000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422740,"kb_used_data":242636,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93949100,"statfs":{"total":96636764160,"available":96203878400,"internally_reserved":0,"allocated":248459264,"data_stored":248133370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":184413866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":0.20999999999999999},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.755},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":3.181}]}]},{"osd":0,"up_from":24,"seq":103079215154,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":795980,"kb_used_data":493700,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93575860,"statfs":{"total":96636764160,"available":95821680640,"internally_reserved":0,"allocated":505548800,"data_stored":505298780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":309522088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,3,3,1,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.5089999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":0.499},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.23200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":505683968,"data_stored":505680355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249266176,"data_stored":249190303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":430772224,"data_stored":430768611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:26.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:26.559 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 194 20 144604 144584 2.91057 1.66797 0.0439123 0.0268346 2024-09-16T10:26:26.951 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:26.951 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:26.974 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-16T10:26:26.623391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590176777,"num_objects":144104,"num_object_clones":0,"num_object_copies":288208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3765,"num_objects_unfound":0,"num_objects_dirty":144104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121733,"num_write_kb":486814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4646,"num_bytes_recovered":18944502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144169,"ondisk_log_size":144169,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1907368,"kb_used_data":1168592,"kb_used_omap":12,"kb_used_meta":738739,"kb_avail":281208152,"statfs":{"total":289910292480,"available":287957147648,"internally_reserved":0,"allocated":1196638208,"data_stored":1195808660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":756468738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,11,6,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27525120,"num_objects":6720,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-218,"num_objects_unfound":0,"num_objects_dirty":6720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6720,"num_write_kb":26880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":892928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004238"},"pg_stats":[{"pgid":"2.f","version":"44'8999","reported_seq":9922,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:25.320145+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:25.320145+0000","last_peered":"2024-09-16T10:26:25.320145+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:25.320145+0000","last_undegraded":"2024-09-16T10:26:25.320145+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8999,"log_dups_size":0,"ondisk_log_size":8999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36859904,"num_objects":8999,"num_object_clones":0,"num_object_copies":17998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3765,"num_objects_unfound":0,"num_objects_dirty":8999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8999,"num_write_kb":35996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":427,"num_bytes_recovered":1748992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5234},{"shards":"0,2","objects":3765}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8980","reported_seq":9137,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.882594+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:25.882594+0000","last_peered":"2024-09-16T10:26:25.882594+0000","last_clean":"2024-09-16T10:26:25.882594+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:25.882594+0000","last_undegraded":"2024-09-16T10:26:25.882594+0000","last_fullsized":"2024-09-16T10:26:25.882594+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36757504,"num_objects":8974,"num_object_clones":0,"num_object_copies":17948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":20772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9042","reported_seq":9227,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.818890+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:25.818890+0000","last_peered":"2024-09-16T10:26:25.818890+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:25.818890+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9042,"log_dups_size":0,"ondisk_log_size":9042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37019648,"num_objects":9038,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5099,"num_write_kb":20396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9057","reported_seq":9113,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.787753+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:25.787753+0000","last_peered":"2024-09-16T10:26:25.787753+0000","last_clean":"2024-09-16T10:26:25.787753+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:25.787753+0000","last_undegraded":"2024-09-16T10:26:25.787753+0000","last_fullsized":"2024-09-16T10:26:25.787753+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9057,"log_dups_size":0,"ondisk_log_size":9057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37097472,"num_objects":9057,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9057,"num_write_kb":36228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8984","reported_seq":9140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.181064+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:25.181064+0000","last_peered":"2024-09-16T10:26:25.181064+0000","last_clean":"2024-09-16T10:26:25.181064+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:25.181064+0000","last_undegraded":"2024-09-16T10:26:25.181064+0000","last_fullsized":"2024-09-16T10:26:25.181064+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8984,"log_dups_size":0,"ondisk_log_size":8984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36769792,"num_objects":8977,"num_object_clones":0,"num_object_copies":17954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5626,"num_write_kb":22504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8894","reported_seq":9056,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.272624+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:25.272624+0000","last_peered":"2024-09-16T10:26:25.272624+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:25.272624+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8894,"log_dups_size":0,"ondisk_log_size":8894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36409344,"num_objects":8889,"num_object_clones":0,"num_object_copies":17778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":22076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4672},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9237","reported_seq":9409,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.292574+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:25.292574+0000","last_peered":"2024-09-16T10:26:25.292574+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:25.292574+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9237,"log_dups_size":0,"ondisk_log_size":9237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37810176,"num_objects":9231,"num_object_clones":0,"num_object_copies":18462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":21916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4828},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8962","reported_seq":9137,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.289564+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:25.289564+0000","last_peered":"2024-09-16T10:26:25.289564+0000","last_clean":"2024-09-16T10:26:25.289564+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:25.289564+0000","last_undegraded":"2024-09-16T10:26:25.289564+0000","last_fullsized":"2024-09-16T10:26:25.289564+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36700160,"num_objects":8960,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4765,"num_write_kb":19060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8914","reported_seq":17417,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:23.549973+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:23.549973+0000","last_peered":"2024-09-16T10:26:23.549973+0000","last_clean":"2024-09-16T10:26:23.549973+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:23.549973+0000","last_undegraded":"2024-09-16T10:26:23.549973+0000","last_fullsized":"2024-09-16T10:26:23.549973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8914,"log_dups_size":0,"ondisk_log_size":8914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36484038,"num_objects":8909,"num_object_clones":0,"num_object_copies":17818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":35632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9225","reported_seq":9298,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.258067+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:25.258067+0000","last_peered":"2024-09-16T10:26:25.258067+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:25.258067+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9225,"log_dups_size":0,"ondisk_log_size":9225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37753798,"num_objects":9219,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9223,"num_write_kb":36872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4785},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8963","reported_seq":9064,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.877138+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:25.877138+0000","last_peered":"2024-09-16T10:26:25.877138+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:25.877138+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8963,"log_dups_size":0,"ondisk_log_size":8963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36692934,"num_objects":8960,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8961,"num_write_kb":35835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4683}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8982","reported_seq":9068,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.869528+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:25.869528+0000","last_peered":"2024-09-16T10:26:25.869528+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:25.869528+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8982,"log_dups_size":0,"ondisk_log_size":8982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36766662,"num_objects":8978,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":35907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4786}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8798","reported_seq":8877,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.309803+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:25.309803+0000","last_peered":"2024-09-16T10:26:25.309803+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:25.309803+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8798,"log_dups_size":0,"ondisk_log_size":8798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36028899,"num_objects":8797,"num_object_clones":0,"num_object_copies":17594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8797,"num_write_kb":35185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4638},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9136","reported_seq":9268,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.322787+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:25.322787+0000","last_peered":"2024-09-16T10:26:25.322787+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:25.322787+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9136,"log_dups_size":0,"ondisk_log_size":9136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37389737,"num_objects":9131,"num_object_clones":0,"num_object_copies":18262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9133,"num_write_kb":36516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4794},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8826","reported_seq":8906,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.277219+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:25.277219+0000","last_peered":"2024-09-16T10:26:25.277219+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:25.277219+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8826,"log_dups_size":0,"ondisk_log_size":8826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36135878,"num_objects":8824,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8824,"num_write_kb":35290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4657},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9167","reported_seq":9259,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.200804+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:25.200804+0000","last_peered":"2024-09-16T10:26:25.200804+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:25.200804+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37500812,"num_objects":9159,"num_object_clones":0,"num_object_copies":18318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9164,"num_write_kb":36627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4764},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":590176758,"num_objects":144102,"num_object_clones":0,"num_object_copies":288204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3765,"num_objects_unfound":0,"num_objects_dirty":144102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121731,"num_write_kb":486812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4645,"num_bytes_recovered":18944483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1197338624,"data_stored":1197255525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144166,"ondisk_log_size":144166,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018927,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677736,"kb_used_data":425440,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93694104,"statfs":{"total":96636764160,"available":95942762496,"internally_reserved":0,"allocated":435650560,"data_stored":435396926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.59599999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.46899999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422740,"kb_used_data":242636,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93949100,"statfs":{"total":96636764160,"available":96203878400,"internally_reserved":0,"allocated":248459264,"data_stored":248133370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":184413866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":0.20999999999999999},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.755},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":3.181}]}]},{"osd":0,"up_from":24,"seq":103079215155,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":806892,"kb_used_data":500516,"kb_used_omap":4,"kb_used_meta":306363,"kb_avail":93564948,"statfs":{"total":96636764160,"available":95810506752,"internally_reserved":0,"allocated":512528384,"data_stored":512278364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":313716392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.542},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":0.31}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":7.0410000000000004},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.47299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":512770048,"data_stored":512766435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249266176,"data_stored":249190303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":435302400,"data_stored":435298787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:26.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:26.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'8999", 'reported_seq': 9922, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:25.320145+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:25.320145+0000', 'last_peered': '2024-09-16T10:26:25.320145+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:25.320145+0000', 'last_undegraded': '2024-09-16T10:26:25.320145+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8999, 'log_dups_size': 0, 'ondisk_log_size': 8999, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36859904, 'num_objects': 8999, 'num_object_clones': 0, 'num_object_copies': 17998, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3765, 'num_objects_unfound': 0, 'num_objects_dirty': 8999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8999, 'num_write_kb': 35996, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 427, 'num_bytes_recovered': 1748992, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5234}, {'shards': '0,2', 'objects': 3765}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:26.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:26.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'9042", 'reported_seq': 9227, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.818890+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:25.818890+0000', 'last_peered': '2024-09-16T10:26:25.818890+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:25.818890+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 9042, 'log_dups_size': 0, 'ondisk_log_size': 9042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37019648, 'num_objects': 9038, 'num_object_clones': 0, 'num_object_copies': 18076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5099, 'num_write_kb': 20396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 4703}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:26.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:26.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'8894", 'reported_seq': 9056, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.272624+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:25.272624+0000', 'last_peered': '2024-09-16T10:26:25.272624+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:25.272624+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 8894, 'log_dups_size': 0, 'ondisk_log_size': 8894, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36409344, 'num_objects': 8889, 'num_object_clones': 0, 'num_object_copies': 17778, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5519, 'num_write_kb': 22076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4672}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:26.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:26.976 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'9237", 'reported_seq': 9409, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.292574+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:25.292574+0000', 'last_peered': '2024-09-16T10:26:25.292574+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:25.292574+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 9237, 'log_dups_size': 0, 'ondisk_log_size': 9237, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37810176, 'num_objects': 9231, 'num_object_clones': 0, 'num_object_copies': 18462, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9231, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5479, 'num_write_kb': 21916, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4828}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:26.977 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:26.977 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'9225", 'reported_seq': 9298, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.258067+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:25.258067+0000', 'last_peered': '2024-09-16T10:26:25.258067+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:25.258067+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9225, 'log_dups_size': 0, 'ondisk_log_size': 9225, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37753798, 'num_objects': 9219, 'num_object_clones': 0, 'num_object_copies': 18438, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9223, 'num_write_kb': 36872, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4785}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:26.977 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:26.977 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'8963", 'reported_seq': 9064, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.877138+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:25.877138+0000', 'last_peered': '2024-09-16T10:26:25.877138+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:25.877138+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 8963, 'log_dups_size': 0, 'ondisk_log_size': 8963, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36692934, 'num_objects': 8960, 'num_object_clones': 0, 'num_object_copies': 17920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8961, 'num_write_kb': 35835, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 4683}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:26.977 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:26.977 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'8982", 'reported_seq': 9068, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.869528+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:25.869528+0000', 'last_peered': '2024-09-16T10:26:25.869528+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:25.869528+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8982, 'log_dups_size': 0, 'ondisk_log_size': 8982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36766662, 'num_objects': 8978, 'num_object_clones': 0, 'num_object_copies': 17956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8980, 'num_write_kb': 35907, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 4786}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:26.977 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:26.977 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'8798", 'reported_seq': 8877, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.309803+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:25.309803+0000', 'last_peered': '2024-09-16T10:26:25.309803+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:25.309803+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8798, 'log_dups_size': 0, 'ondisk_log_size': 8798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36028899, 'num_objects': 8797, 'num_object_clones': 0, 'num_object_copies': 17594, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8797, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8797, 'num_write_kb': 35185, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4638}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:26.978 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:26.978 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'9136", 'reported_seq': 9268, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.322787+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:26:25.322787+0000', 'last_peered': '2024-09-16T10:26:25.322787+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:26:25.322787+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 9136, 'log_dups_size': 0, 'ondisk_log_size': 9136, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37389737, 'num_objects': 9131, 'num_object_clones': 0, 'num_object_copies': 18262, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9133, 'num_write_kb': 36516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4794}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:26.978 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:26.978 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'8826", 'reported_seq': 8906, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.277219+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:26:25.277219+0000', 'last_peered': '2024-09-16T10:26:25.277219+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:26:25.277219+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8826, 'log_dups_size': 0, 'ondisk_log_size': 8826, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36135878, 'num_objects': 8824, 'num_object_clones': 0, 'num_object_copies': 17648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8824, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8824, 'num_write_kb': 35290, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4657}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:26.978 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:26.978 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'9167", 'reported_seq': 9259, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.200804+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:25.200804+0000', 'last_peered': '2024-09-16T10:26:25.200804+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:25.200804+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9167, 'log_dups_size': 0, 'ondisk_log_size': 9167, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37500812, 'num_objects': 9159, 'num_object_clones': 0, 'num_object_copies': 18318, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9164, 'num_write_kb': 36627, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4764}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:26.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:27.559 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 195 20 145020 145000 2.90398 1.625 0.0251394 0.0268956 2024-09-16T10:26:27.654 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:27.654 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/288208 objects degraded (14.898%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":276,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":144104,"data_bytes":590176777,"bytes_used":1953144832,"bytes_avail":287957147648,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":288208,"degraded_ratio":0.14897573974351858,"misplaced_objects":3765,"misplaced_total":288208,"misplaced_ratio":0.013063481929717426,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":74384,"recovering_keys_per_sec":0,"num_objects_recovered":218,"num_bytes_recovered":892928,"num_keys_recovered":0,"write_bytes_sec":2292950,"read_op_per_sec":0,"write_op_per_sec":559},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:27.673 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:27.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:28.231 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:28.232 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:28.261 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-16T10:26:26.623391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590176777,"num_objects":144104,"num_object_clones":0,"num_object_copies":288208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3765,"num_objects_unfound":0,"num_objects_dirty":144104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121733,"num_write_kb":486814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4646,"num_bytes_recovered":18944502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144169,"ondisk_log_size":144169,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1907368,"kb_used_data":1168592,"kb_used_omap":12,"kb_used_meta":738739,"kb_avail":281208152,"statfs":{"total":289910292480,"available":287957147648,"internally_reserved":0,"allocated":1196638208,"data_stored":1195808660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":756468738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,6,11,6,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27525120,"num_objects":6720,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-218,"num_objects_unfound":0,"num_objects_dirty":6720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6720,"num_write_kb":26880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":892928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004238"},"pg_stats":[{"pgid":"2.f","version":"44'8999","reported_seq":9922,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:25.320145+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:25.320145+0000","last_peered":"2024-09-16T10:26:25.320145+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:25.320145+0000","last_undegraded":"2024-09-16T10:26:25.320145+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8999,"log_dups_size":0,"ondisk_log_size":8999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36859904,"num_objects":8999,"num_object_clones":0,"num_object_copies":17998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3765,"num_objects_unfound":0,"num_objects_dirty":8999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8999,"num_write_kb":35996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":427,"num_bytes_recovered":1748992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5234},{"shards":"0,2","objects":3765}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8980","reported_seq":9137,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.882594+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:25.882594+0000","last_peered":"2024-09-16T10:26:25.882594+0000","last_clean":"2024-09-16T10:26:25.882594+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:25.882594+0000","last_undegraded":"2024-09-16T10:26:25.882594+0000","last_fullsized":"2024-09-16T10:26:25.882594+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36757504,"num_objects":8974,"num_object_clones":0,"num_object_copies":17948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":20772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9042","reported_seq":9227,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.818890+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:25.818890+0000","last_peered":"2024-09-16T10:26:25.818890+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:25.818890+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9042,"log_dups_size":0,"ondisk_log_size":9042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37019648,"num_objects":9038,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5099,"num_write_kb":20396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9057","reported_seq":9113,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.787753+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:25.787753+0000","last_peered":"2024-09-16T10:26:25.787753+0000","last_clean":"2024-09-16T10:26:25.787753+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:25.787753+0000","last_undegraded":"2024-09-16T10:26:25.787753+0000","last_fullsized":"2024-09-16T10:26:25.787753+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9057,"log_dups_size":0,"ondisk_log_size":9057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37097472,"num_objects":9057,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9057,"num_write_kb":36228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'8984","reported_seq":9140,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.181064+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:25.181064+0000","last_peered":"2024-09-16T10:26:25.181064+0000","last_clean":"2024-09-16T10:26:25.181064+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:25.181064+0000","last_undegraded":"2024-09-16T10:26:25.181064+0000","last_fullsized":"2024-09-16T10:26:25.181064+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":8984,"log_dups_size":0,"ondisk_log_size":8984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36769792,"num_objects":8977,"num_object_clones":0,"num_object_copies":17954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5626,"num_write_kb":22504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'8894","reported_seq":9056,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.272624+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:25.272624+0000","last_peered":"2024-09-16T10:26:25.272624+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:25.272624+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":8894,"log_dups_size":0,"ondisk_log_size":8894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36409344,"num_objects":8889,"num_object_clones":0,"num_object_copies":17778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5519,"num_write_kb":22076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4672},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9237","reported_seq":9409,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.292574+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:25.292574+0000","last_peered":"2024-09-16T10:26:25.292574+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:25.292574+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9237,"log_dups_size":0,"ondisk_log_size":9237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37810176,"num_objects":9231,"num_object_clones":0,"num_object_copies":18462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5479,"num_write_kb":21916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4828},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'8962","reported_seq":9137,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.289564+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:25.289564+0000","last_peered":"2024-09-16T10:26:25.289564+0000","last_clean":"2024-09-16T10:26:25.289564+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:25.289564+0000","last_undegraded":"2024-09-16T10:26:25.289564+0000","last_fullsized":"2024-09-16T10:26:25.289564+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":8962,"log_dups_size":0,"ondisk_log_size":8962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36700160,"num_objects":8960,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4765,"num_write_kb":19060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'8914","reported_seq":17417,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:23.549973+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:23.549973+0000","last_peered":"2024-09-16T10:26:23.549973+0000","last_clean":"2024-09-16T10:26:23.549973+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:23.549973+0000","last_undegraded":"2024-09-16T10:26:23.549973+0000","last_fullsized":"2024-09-16T10:26:23.549973+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8914,"log_dups_size":0,"ondisk_log_size":8914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36484038,"num_objects":8909,"num_object_clones":0,"num_object_copies":17818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":35632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9225","reported_seq":9298,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.258067+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:25.258067+0000","last_peered":"2024-09-16T10:26:25.258067+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:25.258067+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9225,"log_dups_size":0,"ondisk_log_size":9225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37753798,"num_objects":9219,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9223,"num_write_kb":36872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4785},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8963","reported_seq":9064,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.877138+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:25.877138+0000","last_peered":"2024-09-16T10:26:25.877138+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:25.877138+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8963,"log_dups_size":0,"ondisk_log_size":8963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36692934,"num_objects":8960,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8961,"num_write_kb":35835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4683}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8982","reported_seq":9068,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.869528+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:25.869528+0000","last_peered":"2024-09-16T10:26:25.869528+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:25.869528+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8982,"log_dups_size":0,"ondisk_log_size":8982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36766662,"num_objects":8978,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":35907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4786}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8798","reported_seq":8877,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.309803+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:25.309803+0000","last_peered":"2024-09-16T10:26:25.309803+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:25.309803+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8798,"log_dups_size":0,"ondisk_log_size":8798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36028899,"num_objects":8797,"num_object_clones":0,"num_object_copies":17594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8797,"num_write_kb":35185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4638},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9136","reported_seq":9268,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.322787+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:25.322787+0000","last_peered":"2024-09-16T10:26:25.322787+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:25.322787+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9136,"log_dups_size":0,"ondisk_log_size":9136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37389737,"num_objects":9131,"num_object_clones":0,"num_object_copies":18262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9133,"num_write_kb":36516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4794},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8826","reported_seq":8906,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.277219+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:25.277219+0000","last_peered":"2024-09-16T10:26:25.277219+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:25.277219+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8826,"log_dups_size":0,"ondisk_log_size":8826,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36135878,"num_objects":8824,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8824,"num_write_kb":35290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4657},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9167","reported_seq":9259,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.200804+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:25.200804+0000","last_peered":"2024-09-16T10:26:25.200804+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:25.200804+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37500812,"num_objects":9159,"num_object_clones":0,"num_object_copies":18318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9164,"num_write_kb":36627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4764},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":590176758,"num_objects":144102,"num_object_clones":0,"num_object_copies":288204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3765,"num_objects_unfound":0,"num_objects_dirty":144102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121731,"num_write_kb":486812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4645,"num_bytes_recovered":18944483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1197338624,"data_stored":1197255525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144166,"ondisk_log_size":144166,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018927,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677736,"kb_used_data":425440,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93694104,"statfs":{"total":96636764160,"available":95942762496,"internally_reserved":0,"allocated":435650560,"data_stored":435396926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.59599999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.46899999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691871,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":422740,"kb_used_data":242636,"kb_used_omap":4,"kb_used_meta":180091,"kb_avail":93949100,"statfs":{"total":96636764160,"available":96203878400,"internally_reserved":0,"allocated":248459264,"data_stored":248133370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":184413866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,1,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":0.20999999999999999},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.65200000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.755},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":3.181}]}]},{"osd":0,"up_from":24,"seq":103079215155,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":806892,"kb_used_data":500516,"kb_used_omap":4,"kb_used_meta":306363,"kb_avail":93564948,"statfs":{"total":96636764160,"available":95810506752,"internally_reserved":0,"allocated":512528384,"data_stored":512278364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":313716392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.542},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":0.31}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":7.0410000000000004},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.47299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":512770048,"data_stored":512766435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":249266176,"data_stored":249190303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":435302400,"data_stored":435298787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:28.560 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 196 20 145449 145429 2.89771 1.67578 0.0226111 0.0269538 2024-09-16T10:26:29.560 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 197 20 145912 145892 2.89218 1.80859 0.063912 0.0270049 2024-09-16T10:26:30.560 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 198 20 146386 146366 2.88693 1.85156 0.0321302 0.0270552 2024-09-16T10:26:31.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:31.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 199 20 146817 146797 2.88086 1.68359 0.0681333 0.0271079 2024-09-16T10:26:31.797 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:31.800 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:31.826 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-16T10:26:30.625143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596488713,"num_objects":145645,"num_object_clones":0,"num_object_copies":291290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3682,"num_objects_unfound":0,"num_objects_dirty":145645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123274,"num_write_kb":492978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4729,"num_bytes_recovered":19284470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145710,"ondisk_log_size":145710,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1929080,"kb_used_data":1182108,"kb_used_omap":12,"kb_used_meta":746931,"kb_avail":281186440,"statfs":{"total":289910292480,"available":287934914560,"internally_reserved":0,"allocated":1210478592,"data_stored":1209649044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":764857346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,3,9,17,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24211456,"num_objects":5911,"num_object_clones":0,"num_object_copies":11822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-230,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5911,"num_write_kb":23644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004784"},"pg_stats":[{"pgid":"2.f","version":"44'9140","reported_seq":10228,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:30.333813+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:30.333813+0000","last_peered":"2024-09-16T10:26:30.333813+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:30.333813+0000","last_undegraded":"2024-09-16T10:26:30.333813+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9140,"log_dups_size":0,"ondisk_log_size":9140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37437440,"num_objects":9140,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3682,"num_objects_unfound":0,"num_objects_dirty":9140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":36560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":510,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5458},{"shards":"0,2","objects":3682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8980","reported_seq":9137,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.882594+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:25.882594+0000","last_peered":"2024-09-16T10:26:25.882594+0000","last_clean":"2024-09-16T10:26:25.882594+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:25.882594+0000","last_undegraded":"2024-09-16T10:26:25.882594+0000","last_fullsized":"2024-09-16T10:26:25.882594+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36757504,"num_objects":8974,"num_object_clones":0,"num_object_copies":17948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":20772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9042","reported_seq":9227,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.818890+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:25.818890+0000","last_peered":"2024-09-16T10:26:25.818890+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:25.818890+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9042,"log_dups_size":0,"ondisk_log_size":9042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37019648,"num_objects":9038,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5099,"num_write_kb":20396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9057","reported_seq":9113,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.787753+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:25.787753+0000","last_peered":"2024-09-16T10:26:25.787753+0000","last_clean":"2024-09-16T10:26:25.787753+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:25.787753+0000","last_undegraded":"2024-09-16T10:26:25.787753+0000","last_fullsized":"2024-09-16T10:26:25.787753+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9057,"log_dups_size":0,"ondisk_log_size":9057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37097472,"num_objects":9057,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9057,"num_write_kb":36228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9133","reported_seq":9289,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:30.308253+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:30.308253+0000","last_peered":"2024-09-16T10:26:30.308253+0000","last_clean":"2024-09-16T10:26:30.308253+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:30.308253+0000","last_undegraded":"2024-09-16T10:26:30.308253+0000","last_fullsized":"2024-09-16T10:26:30.308253+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9133,"log_dups_size":0,"ondisk_log_size":9133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37380096,"num_objects":9126,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5775,"num_write_kb":23100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9035","reported_seq":9198,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.354882+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:30.354882+0000","last_peered":"2024-09-16T10:26:30.354882+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:30.354882+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9035,"log_dups_size":0,"ondisk_log_size":9035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36986880,"num_objects":9030,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5660,"num_write_kb":22640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4813},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9364","reported_seq":9537,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.277692+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:30.277692+0000","last_peered":"2024-09-16T10:26:30.277692+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:30.277692+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38330368,"num_objects":9358,"num_object_clones":0,"num_object_copies":18716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":22424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4955},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9108","reported_seq":9283,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:30.350495+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:30.350495+0000","last_peered":"2024-09-16T10:26:30.350495+0000","last_clean":"2024-09-16T10:26:30.350495+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:30.350495+0000","last_undegraded":"2024-09-16T10:26:30.350495+0000","last_fullsized":"2024-09-16T10:26:30.350495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9108,"log_dups_size":0,"ondisk_log_size":9108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37298176,"num_objects":9106,"num_object_clones":0,"num_object_copies":18212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4911,"num_write_kb":19644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9063","reported_seq":17566,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:28.730878+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:28.730878+0000","last_peered":"2024-09-16T10:26:28.730878+0000","last_clean":"2024-09-16T10:26:28.730878+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:28.730878+0000","last_undegraded":"2024-09-16T10:26:28.730878+0000","last_fullsized":"2024-09-16T10:26:28.730878+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9063,"log_dups_size":0,"ondisk_log_size":9063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37094342,"num_objects":9058,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":36228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9352","reported_seq":9430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.354835+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:30.354835+0000","last_peered":"2024-09-16T10:26:30.354835+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:30.354835+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38273990,"num_objects":9346,"num_object_clones":0,"num_object_copies":18692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":37380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4912},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8963","reported_seq":9064,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.877138+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:25.877138+0000","last_peered":"2024-09-16T10:26:25.877138+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:25.877138+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8963,"log_dups_size":0,"ondisk_log_size":8963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36692934,"num_objects":8960,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8961,"num_write_kb":35835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4683}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8982","reported_seq":9068,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.869528+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:25.869528+0000","last_peered":"2024-09-16T10:26:25.869528+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:25.869528+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8982,"log_dups_size":0,"ondisk_log_size":8982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36766662,"num_objects":8978,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":35907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4786}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8913","reported_seq":8992,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.341092+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:30.341092+0000","last_peered":"2024-09-16T10:26:30.341092+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:30.341092+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36499939,"num_objects":8912,"num_object_clones":0,"num_object_copies":17824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":35645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4753},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9282","reported_seq":9415,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.271965+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:30.271965+0000","last_peered":"2024-09-16T10:26:30.271965+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:30.271965+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37987753,"num_objects":9277,"num_object_clones":0,"num_object_copies":18554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9279,"num_write_kb":37100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4940},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8980","reported_seq":9060,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.295199+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:30.295199+0000","last_peered":"2024-09-16T10:26:30.295199+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:30.295199+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36766662,"num_objects":8978,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8978,"num_write_kb":35906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4811},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9313","reported_seq":9404,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.308003+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:30.308003+0000","last_peered":"2024-09-16T10:26:30.308003+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:30.308003+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9313,"log_dups_size":0,"ondisk_log_size":9313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38098828,"num_objects":9305,"num_object_clones":0,"num_object_copies":18610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":37211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4910},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":596488694,"num_objects":145643,"num_object_clones":0,"num_object_copies":291286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3682,"num_objects_unfound":0,"num_objects_dirty":145643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123272,"num_write_kb":492976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4728,"num_bytes_recovered":19284451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1211305984,"data_stored":1211222885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145707,"ondisk_log_size":145707,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018927,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677736,"kb_used_data":425440,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93694104,"statfs":{"total":96636764160,"available":95942762496,"internally_reserved":0,"allocated":435650560,"data_stored":435396926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.59599999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.46899999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433720,"kb_used_data":249516,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93938120,"statfs":{"total":96636764160,"available":96192634880,"internally_reserved":0,"allocated":255504384,"data_stored":255178490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":188608170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":3.4569999999999999},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.488}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":1.603},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":1.647}]}]},{"osd":0,"up_from":24,"seq":103079215156,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":817624,"kb_used_data":507152,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93554216,"statfs":{"total":96636764160,"available":95799517184,"internally_reserved":0,"allocated":519323648,"data_stored":519073628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":317910696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":2.6040000000000001},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":2.3260000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":2.556},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":3.0739999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":519598080,"data_stored":519594467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256405504,"data_stored":256329631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":435302400,"data_stored":435298787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:31.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:32.408 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:32.410 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:32.441 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-16T10:26:30.625143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596488713,"num_objects":145645,"num_object_clones":0,"num_object_copies":291290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3682,"num_objects_unfound":0,"num_objects_dirty":145645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123274,"num_write_kb":492978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4729,"num_bytes_recovered":19284470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145710,"ondisk_log_size":145710,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1929080,"kb_used_data":1182108,"kb_used_omap":12,"kb_used_meta":746931,"kb_avail":281186440,"statfs":{"total":289910292480,"available":287934914560,"internally_reserved":0,"allocated":1210478592,"data_stored":1209649044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":764857346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,3,9,17,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24211456,"num_objects":5911,"num_object_clones":0,"num_object_copies":11822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-230,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5911,"num_write_kb":23644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004784"},"pg_stats":[{"pgid":"2.f","version":"44'9140","reported_seq":10228,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:30.333813+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:30.333813+0000","last_peered":"2024-09-16T10:26:30.333813+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:30.333813+0000","last_undegraded":"2024-09-16T10:26:30.333813+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9140,"log_dups_size":0,"ondisk_log_size":9140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37437440,"num_objects":9140,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3682,"num_objects_unfound":0,"num_objects_dirty":9140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":36560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":510,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5458},{"shards":"0,2","objects":3682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'8980","reported_seq":9137,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.882594+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:25.882594+0000","last_peered":"2024-09-16T10:26:25.882594+0000","last_clean":"2024-09-16T10:26:25.882594+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:25.882594+0000","last_undegraded":"2024-09-16T10:26:25.882594+0000","last_fullsized":"2024-09-16T10:26:25.882594+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36757504,"num_objects":8974,"num_object_clones":0,"num_object_copies":17948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":20772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9042","reported_seq":9227,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.818890+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:25.818890+0000","last_peered":"2024-09-16T10:26:25.818890+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:25.818890+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9042,"log_dups_size":0,"ondisk_log_size":9042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37019648,"num_objects":9038,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5099,"num_write_kb":20396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4703}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9057","reported_seq":9113,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:25.787753+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:25.787753+0000","last_peered":"2024-09-16T10:26:25.787753+0000","last_clean":"2024-09-16T10:26:25.787753+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:25.787753+0000","last_undegraded":"2024-09-16T10:26:25.787753+0000","last_fullsized":"2024-09-16T10:26:25.787753+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9057,"log_dups_size":0,"ondisk_log_size":9057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37097472,"num_objects":9057,"num_object_clones":0,"num_object_copies":18114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9057,"num_write_kb":36228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9133","reported_seq":9289,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:30.308253+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:30.308253+0000","last_peered":"2024-09-16T10:26:30.308253+0000","last_clean":"2024-09-16T10:26:30.308253+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:30.308253+0000","last_undegraded":"2024-09-16T10:26:30.308253+0000","last_fullsized":"2024-09-16T10:26:30.308253+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9133,"log_dups_size":0,"ondisk_log_size":9133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37380096,"num_objects":9126,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5775,"num_write_kb":23100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9035","reported_seq":9198,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.354882+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:30.354882+0000","last_peered":"2024-09-16T10:26:30.354882+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:30.354882+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9035,"log_dups_size":0,"ondisk_log_size":9035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36986880,"num_objects":9030,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5660,"num_write_kb":22640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4813},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9364","reported_seq":9537,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.277692+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:30.277692+0000","last_peered":"2024-09-16T10:26:30.277692+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:30.277692+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38330368,"num_objects":9358,"num_object_clones":0,"num_object_copies":18716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":22424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4955},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9108","reported_seq":9283,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:30.350495+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:30.350495+0000","last_peered":"2024-09-16T10:26:30.350495+0000","last_clean":"2024-09-16T10:26:30.350495+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:30.350495+0000","last_undegraded":"2024-09-16T10:26:30.350495+0000","last_fullsized":"2024-09-16T10:26:30.350495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9108,"log_dups_size":0,"ondisk_log_size":9108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37298176,"num_objects":9106,"num_object_clones":0,"num_object_copies":18212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4911,"num_write_kb":19644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9063","reported_seq":17566,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:28.730878+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:28.730878+0000","last_peered":"2024-09-16T10:26:28.730878+0000","last_clean":"2024-09-16T10:26:28.730878+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:28.730878+0000","last_undegraded":"2024-09-16T10:26:28.730878+0000","last_fullsized":"2024-09-16T10:26:28.730878+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9063,"log_dups_size":0,"ondisk_log_size":9063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37094342,"num_objects":9058,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":36228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9352","reported_seq":9430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.354835+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:30.354835+0000","last_peered":"2024-09-16T10:26:30.354835+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:30.354835+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38273990,"num_objects":9346,"num_object_clones":0,"num_object_copies":18692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":37380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4912},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'8963","reported_seq":9064,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.877138+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:25.877138+0000","last_peered":"2024-09-16T10:26:25.877138+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:25.877138+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":8963,"log_dups_size":0,"ondisk_log_size":8963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36692934,"num_objects":8960,"num_object_clones":0,"num_object_copies":17920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8961,"num_write_kb":35835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4683}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'8982","reported_seq":9068,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:25.869528+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:25.869528+0000","last_peered":"2024-09-16T10:26:25.869528+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:25.869528+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8982,"log_dups_size":0,"ondisk_log_size":8982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36766662,"num_objects":8978,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":35907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4786}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8913","reported_seq":8992,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.341092+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:30.341092+0000","last_peered":"2024-09-16T10:26:30.341092+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:30.341092+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36499939,"num_objects":8912,"num_object_clones":0,"num_object_copies":17824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":35645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4753},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9282","reported_seq":9415,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.271965+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:30.271965+0000","last_peered":"2024-09-16T10:26:30.271965+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:30.271965+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37987753,"num_objects":9277,"num_object_clones":0,"num_object_copies":18554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9279,"num_write_kb":37100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4940},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8980","reported_seq":9060,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.295199+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:30.295199+0000","last_peered":"2024-09-16T10:26:30.295199+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:30.295199+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36766662,"num_objects":8978,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8978,"num_write_kb":35906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4811},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9313","reported_seq":9404,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.308003+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:30.308003+0000","last_peered":"2024-09-16T10:26:30.308003+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:30.308003+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9313,"log_dups_size":0,"ondisk_log_size":9313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38098828,"num_objects":9305,"num_object_clones":0,"num_object_copies":18610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":37211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4910},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":596488694,"num_objects":145643,"num_object_clones":0,"num_object_copies":291286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3682,"num_objects_unfound":0,"num_objects_dirty":145643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123272,"num_write_kb":492976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4728,"num_bytes_recovered":19284451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1211305984,"data_stored":1211222885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145707,"ondisk_log_size":145707,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018927,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677736,"kb_used_data":425440,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93694104,"statfs":{"total":96636764160,"available":95942762496,"internally_reserved":0,"allocated":435650560,"data_stored":435396926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.59599999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.46899999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433720,"kb_used_data":249516,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93938120,"statfs":{"total":96636764160,"available":96192634880,"internally_reserved":0,"allocated":255504384,"data_stored":255178490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":188608170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":3.4569999999999999},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.488}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":1.603},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":1.647}]}]},{"osd":0,"up_from":24,"seq":103079215156,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":817624,"kb_used_data":507152,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93554216,"statfs":{"total":96636764160,"available":95799517184,"internally_reserved":0,"allocated":519323648,"data_stored":519073628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":317910696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":2.6040000000000001},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":2.3260000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":2.556},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":3.0739999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":519598080,"data_stored":519594467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256405504,"data_stored":256329631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":435302400,"data_stored":435298787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'9140", 'reported_seq': 10228, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:30.333813+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:30.333813+0000', 'last_peered': '2024-09-16T10:26:30.333813+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:30.333813+0000', 'last_undegraded': '2024-09-16T10:26:30.333813+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9140, 'log_dups_size': 0, 'ondisk_log_size': 9140, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37437440, 'num_objects': 9140, 'num_object_clones': 0, 'num_object_copies': 18280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3682, 'num_objects_unfound': 0, 'num_objects_dirty': 9140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9140, 'num_write_kb': 36560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 510, 'num_bytes_recovered': 2088960, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5458}, {'shards': '0,2', 'objects': 3682}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'9042", 'reported_seq': 9227, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.818890+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:25.818890+0000', 'last_peered': '2024-09-16T10:26:25.818890+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:25.818890+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 9042, 'log_dups_size': 0, 'ondisk_log_size': 9042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37019648, 'num_objects': 9038, 'num_object_clones': 0, 'num_object_copies': 18076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5099, 'num_write_kb': 20396, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 4703}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'9035", 'reported_seq': 9198, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:30.354882+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:30.354882+0000', 'last_peered': '2024-09-16T10:26:30.354882+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:30.354882+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 9035, 'log_dups_size': 0, 'ondisk_log_size': 9035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36986880, 'num_objects': 9030, 'num_object_clones': 0, 'num_object_copies': 18060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5660, 'num_write_kb': 22640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4813}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'9364", 'reported_seq': 9537, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:30.277692+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:30.277692+0000', 'last_peered': '2024-09-16T10:26:30.277692+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:30.277692+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 9364, 'log_dups_size': 0, 'ondisk_log_size': 9364, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38330368, 'num_objects': 9358, 'num_object_clones': 0, 'num_object_copies': 18716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5606, 'num_write_kb': 22424, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4955}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'9352", 'reported_seq': 9430, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:30.354835+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:30.354835+0000', 'last_peered': '2024-09-16T10:26:30.354835+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:30.354835+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9352, 'log_dups_size': 0, 'ondisk_log_size': 9352, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38273990, 'num_objects': 9346, 'num_object_clones': 0, 'num_object_copies': 18692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9350, 'num_write_kb': 37380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4912}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:32.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'8963", 'reported_seq': 9064, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.877138+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:25.877138+0000', 'last_peered': '2024-09-16T10:26:25.877138+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:25.877138+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 8963, 'log_dups_size': 0, 'ondisk_log_size': 8963, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36692934, 'num_objects': 8960, 'num_object_clones': 0, 'num_object_copies': 17920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8961, 'num_write_kb': 35835, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 4683}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:32.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:32.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'8982", 'reported_seq': 9068, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:25.869528+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:25.869528+0000', 'last_peered': '2024-09-16T10:26:25.869528+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:25.869528+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8982, 'log_dups_size': 0, 'ondisk_log_size': 8982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36766662, 'num_objects': 8978, 'num_object_clones': 0, 'num_object_copies': 17956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8980, 'num_write_kb': 35907, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 4786}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:32.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:32.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'8913", 'reported_seq': 8992, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:30.341092+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:30.341092+0000', 'last_peered': '2024-09-16T10:26:30.341092+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:30.341092+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8913, 'log_dups_size': 0, 'ondisk_log_size': 8913, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36499939, 'num_objects': 8912, 'num_object_clones': 0, 'num_object_copies': 17824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8912, 'num_write_kb': 35645, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4753}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:32.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:32.444 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'9282", 'reported_seq': 9415, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:30.271965+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:26:30.271965+0000', 'last_peered': '2024-09-16T10:26:30.271965+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:26:30.271965+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 9282, 'log_dups_size': 0, 'ondisk_log_size': 9282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37987753, 'num_objects': 9277, 'num_object_clones': 0, 'num_object_copies': 18554, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9277, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9279, 'num_write_kb': 37100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4940}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:32.445 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:32.445 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'8980", 'reported_seq': 9060, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:30.295199+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:26:30.295199+0000', 'last_peered': '2024-09-16T10:26:30.295199+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:26:30.295199+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 8980, 'log_dups_size': 0, 'ondisk_log_size': 8980, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36766662, 'num_objects': 8978, 'num_object_clones': 0, 'num_object_copies': 17956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8978, 'num_write_kb': 35906, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4811}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:32.446 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:32.446 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'9313", 'reported_seq': 9404, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:30.308003+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:30.308003+0000', 'last_peered': '2024-09-16T10:26:30.308003+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:30.308003+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9313, 'log_dups_size': 0, 'ondisk_log_size': 9313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38098828, 'num_objects': 9305, 'num_object_clones': 0, 'num_object_copies': 18610, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9310, 'num_write_kb': 37211, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4910}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:32.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:32.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:26:32.561147+0000 min lat: 0.00139595 max lat: 2.61294 avg lat: 0.0271703 2024-09-16T10:26:32.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:26:32.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 200 20 147228 147208 2.87448 1.60547 0.0390988 0.0271703 2024-09-16T10:26:33.116 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:33.116 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/291290 objects degraded (14.740%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":281,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":145645,"data_bytes":596488713,"bytes_used":1975377920,"bytes_avail":287934914560,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":291290,"degraded_ratio":0.14739949878128325,"misplaced_objects":3682,"misplaced_total":291290,"misplaced_ratio":0.012640324075663427,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":78475,"recovering_keys_per_sec":0,"num_objects_recovered":230,"num_bytes_recovered":942080,"num_keys_recovered":0,"write_bytes_sec":2016817,"read_op_per_sec":0,"write_op_per_sec":492},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:33.135 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:33.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:33.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 201 20 147638 147618 2.86815 1.60156 0.0488145 0.0272315 2024-09-16T10:26:33.655 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:33.657 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:33.681 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-16T10:26:32.626080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599310857,"num_objects":146334,"num_object_clones":0,"num_object_copies":292668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3682,"num_objects_unfound":0,"num_objects_dirty":146334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123963,"num_write_kb":495734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4729,"num_bytes_recovered":19284470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146399,"ondisk_log_size":146399,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1933468,"kb_used_data":1186496,"kb_used_omap":12,"kb_used_meta":746931,"kb_avail":281182052,"statfs":{"total":289910292480,"available":287930421248,"internally_reserved":0,"allocated":1214971904,"data_stored":1214142356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":764857346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,3,7,16,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21364736,"num_objects":5216,"num_object_clones":0,"num_object_copies":10432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-156,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":20864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004975"},"pg_stats":[{"pgid":"2.f","version":"44'9140","reported_seq":10228,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:30.333813+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:30.333813+0000","last_peered":"2024-09-16T10:26:30.333813+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:30.333813+0000","last_undegraded":"2024-09-16T10:26:30.333813+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9140,"log_dups_size":0,"ondisk_log_size":9140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37437440,"num_objects":9140,"num_object_clones":0,"num_object_copies":18280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3682,"num_objects_unfound":0,"num_objects_dirty":9140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":36560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":510,"num_bytes_recovered":2088960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5458},{"shards":"0,2","objects":3682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9139","reported_seq":9296,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:30.888067+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:30.888067+0000","last_peered":"2024-09-16T10:26:30.888067+0000","last_clean":"2024-09-16T10:26:30.888067+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:30.888067+0000","last_undegraded":"2024-09-16T10:26:30.888067+0000","last_fullsized":"2024-09-16T10:26:30.888067+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9139,"log_dups_size":0,"ondisk_log_size":9139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37408768,"num_objects":9133,"num_object_clones":0,"num_object_copies":18266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5352,"num_write_kb":21408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9177","reported_seq":9361,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.911411+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:30.911411+0000","last_peered":"2024-09-16T10:26:30.911411+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:30.911411+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9177,"log_dups_size":0,"ondisk_log_size":9177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37572608,"num_objects":9173,"num_object_clones":0,"num_object_copies":18346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":20936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4838}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9173","reported_seq":9229,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:30.934235+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:30.934235+0000","last_peered":"2024-09-16T10:26:30.934235+0000","last_clean":"2024-09-16T10:26:30.934235+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:30.934235+0000","last_undegraded":"2024-09-16T10:26:30.934235+0000","last_fullsized":"2024-09-16T10:26:30.934235+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9173,"log_dups_size":0,"ondisk_log_size":9173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37572608,"num_objects":9173,"num_object_clones":0,"num_object_copies":18346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9173,"num_write_kb":36692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9133","reported_seq":9289,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:30.308253+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:30.308253+0000","last_peered":"2024-09-16T10:26:30.308253+0000","last_clean":"2024-09-16T10:26:30.308253+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:30.308253+0000","last_undegraded":"2024-09-16T10:26:30.308253+0000","last_fullsized":"2024-09-16T10:26:30.308253+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9133,"log_dups_size":0,"ondisk_log_size":9133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37380096,"num_objects":9126,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5775,"num_write_kb":23100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9035","reported_seq":9198,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.354882+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:30.354882+0000","last_peered":"2024-09-16T10:26:30.354882+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:30.354882+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9035,"log_dups_size":0,"ondisk_log_size":9035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36986880,"num_objects":9030,"num_object_clones":0,"num_object_copies":18060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5660,"num_write_kb":22640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4813},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9364","reported_seq":9537,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.277692+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:30.277692+0000","last_peered":"2024-09-16T10:26:30.277692+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:30.277692+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9364,"log_dups_size":0,"ondisk_log_size":9364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38330368,"num_objects":9358,"num_object_clones":0,"num_object_copies":18716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":22424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4955},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9108","reported_seq":9283,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:30.350495+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:30.350495+0000","last_peered":"2024-09-16T10:26:30.350495+0000","last_clean":"2024-09-16T10:26:30.350495+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:30.350495+0000","last_undegraded":"2024-09-16T10:26:30.350495+0000","last_fullsized":"2024-09-16T10:26:30.350495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9108,"log_dups_size":0,"ondisk_log_size":9108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37298176,"num_objects":9106,"num_object_clones":0,"num_object_copies":18212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4911,"num_write_kb":19644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9063","reported_seq":17566,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:28.730878+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:28.730878+0000","last_peered":"2024-09-16T10:26:28.730878+0000","last_clean":"2024-09-16T10:26:28.730878+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:28.730878+0000","last_undegraded":"2024-09-16T10:26:28.730878+0000","last_fullsized":"2024-09-16T10:26:28.730878+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9063,"log_dups_size":0,"ondisk_log_size":9063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37094342,"num_objects":9058,"num_object_clones":0,"num_object_copies":18116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":36228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9352","reported_seq":9430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.354835+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:30.354835+0000","last_peered":"2024-09-16T10:26:30.354835+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:30.354835+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9352,"log_dups_size":0,"ondisk_log_size":9352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38273990,"num_objects":9346,"num_object_clones":0,"num_object_copies":18692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9350,"num_write_kb":37380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4912},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9101","reported_seq":9205,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.898604+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:30.898604+0000","last_peered":"2024-09-16T10:26:30.898604+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:30.898604+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9101,"log_dups_size":0,"ondisk_log_size":9101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37258182,"num_objects":9098,"num_object_clones":0,"num_object_copies":18196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9099,"num_write_kb":36387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4821}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9123","reported_seq":9209,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.939437+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:30.939437+0000","last_peered":"2024-09-16T10:26:30.939437+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:30.939437+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9123,"log_dups_size":0,"ondisk_log_size":9123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37344198,"num_objects":9119,"num_object_clones":0,"num_object_copies":18238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9121,"num_write_kb":36471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":4927}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'8913","reported_seq":8992,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.341092+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:30.341092+0000","last_peered":"2024-09-16T10:26:30.341092+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:30.341092+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8913,"log_dups_size":0,"ondisk_log_size":8913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36499939,"num_objects":8912,"num_object_clones":0,"num_object_copies":17824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8912,"num_write_kb":35645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4753},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9282","reported_seq":9415,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.271965+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:30.271965+0000","last_peered":"2024-09-16T10:26:30.271965+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:30.271965+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37987753,"num_objects":9277,"num_object_clones":0,"num_object_copies":18554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9279,"num_write_kb":37100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4940},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'8980","reported_seq":9060,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.295199+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:30.295199+0000","last_peered":"2024-09-16T10:26:30.295199+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:30.295199+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36766662,"num_objects":8978,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8978,"num_write_kb":35906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4811},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9313","reported_seq":9404,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:30.308003+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:30.308003+0000","last_peered":"2024-09-16T10:26:30.308003+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:30.308003+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9313,"log_dups_size":0,"ondisk_log_size":9313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38098828,"num_objects":9305,"num_object_clones":0,"num_object_copies":18610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":37211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4910},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":599310838,"num_objects":146332,"num_object_clones":0,"num_object_copies":292664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3682,"num_objects_unfound":0,"num_objects_dirty":146332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123961,"num_write_kb":495732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4728,"num_bytes_recovered":19284451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1215864832,"data_stored":1215781733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146396,"ondisk_log_size":146396,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018928,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682124,"kb_used_data":429828,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93689716,"statfs":{"total":96636764160,"available":95938269184,"internally_reserved":0,"allocated":440143872,"data_stored":439890238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.367},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.0469999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":1.1180000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691872,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":433720,"kb_used_data":249516,"kb_used_omap":4,"kb_used_meta":184187,"kb_avail":93938120,"statfs":{"total":96636764160,"available":96192634880,"internally_reserved":0,"allocated":255504384,"data_stored":255178490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":188608170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":3.4569999999999999},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.488}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":1.603},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":1.647}]}]},{"osd":0,"up_from":24,"seq":103079215156,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":817624,"kb_used_data":507152,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93554216,"statfs":{"total":96636764160,"available":95799517184,"internally_reserved":0,"allocated":519323648,"data_stored":519073628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":317910696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":2.6040000000000001},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":2.3260000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":2.556},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":3.0739999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":519598080,"data_stored":519594467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256405504,"data_stored":256329631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":439861248,"data_stored":439857635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:34.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 202 20 148094 148074 2.86277 1.78125 0.0114645 0.0272814 2024-09-16T10:26:35.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 203 20 148557 148537 2.85757 1.80859 0.0527037 0.0273328 2024-09-16T10:26:36.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 204 20 148932 148912 2.85075 1.46484 0.0245851 0.0273953 2024-09-16T10:26:36.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:37.234 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:37.237 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:37.256 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-16T10:26:36.627495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":608334345,"num_objects":148537,"num_object_clones":0,"num_object_copies":297074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":148537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126166,"num_write_kb":504546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4810,"num_bytes_recovered":19616246,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148602,"ondisk_log_size":148602,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1954336,"kb_used_data":1205704,"kb_used_omap":12,"kb_used_meta":748595,"kb_avail":281161184,"statfs":{"total":289910292480,"available":287909052416,"internally_reserved":0,"allocated":1234640896,"data_stored":1233811348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":766561282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,4,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26853376,"num_objects":6556,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-237,"num_objects_unfound":0,"num_objects_dirty":6556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":26224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":970752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004662"},"pg_stats":[{"pgid":"2.f","version":"44'9280","reported_seq":10531,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:35.362181+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:35.362181+0000","last_peered":"2024-09-16T10:26:35.362181+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:35.362181+0000","last_undegraded":"2024-09-16T10:26:35.362181+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9280,"log_dups_size":0,"ondisk_log_size":9280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38010880,"num_objects":9280,"num_object_clones":0,"num_object_copies":18560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":9280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":37120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":591,"num_bytes_recovered":2420736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5679},{"shards":"0,2","objects":3601}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9282","reported_seq":9439,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.917205+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:35.917205+0000","last_peered":"2024-09-16T10:26:35.917205+0000","last_clean":"2024-09-16T10:26:35.917205+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:35.917205+0000","last_undegraded":"2024-09-16T10:26:35.917205+0000","last_fullsized":"2024-09-16T10:26:35.917205+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37994496,"num_objects":9276,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9309","reported_seq":9494,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.944093+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:35.944093+0000","last_peered":"2024-09-16T10:26:35.944093+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:35.944093+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9309,"log_dups_size":0,"ondisk_log_size":9309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38113280,"num_objects":9305,"num_object_clones":0,"num_object_copies":18610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":21464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4970}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9309","reported_seq":9365,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.854681+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:35.854681+0000","last_peered":"2024-09-16T10:26:35.854681+0000","last_clean":"2024-09-16T10:26:35.854681+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:35.854681+0000","last_undegraded":"2024-09-16T10:26:35.854681+0000","last_fullsized":"2024-09-16T10:26:35.854681+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9309,"log_dups_size":0,"ondisk_log_size":9309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38129664,"num_objects":9309,"num_object_clones":0,"num_object_copies":18618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9309,"num_write_kb":37236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9260","reported_seq":9416,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.417872+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:35.417872+0000","last_peered":"2024-09-16T10:26:35.417872+0000","last_clean":"2024-09-16T10:26:35.417872+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:35.417872+0000","last_undegraded":"2024-09-16T10:26:35.417872+0000","last_fullsized":"2024-09-16T10:26:35.417872+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9260,"log_dups_size":0,"ondisk_log_size":9260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900288,"num_objects":9253,"num_object_clones":0,"num_object_copies":18506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5902,"num_write_kb":23608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9197","reported_seq":9360,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.262487+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:35.262487+0000","last_peered":"2024-09-16T10:26:35.262487+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:35.262487+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37650432,"num_objects":9192,"num_object_clones":0,"num_object_copies":18384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4975},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9499","reported_seq":9672,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.305884+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:35.305884+0000","last_peered":"2024-09-16T10:26:35.305884+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:35.305884+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9499,"log_dups_size":0,"ondisk_log_size":9499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38883328,"num_objects":9493,"num_object_clones":0,"num_object_copies":18986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5741,"num_write_kb":22964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5090},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9232","reported_seq":9407,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.329022+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:35.329022+0000","last_peered":"2024-09-16T10:26:35.329022+0000","last_clean":"2024-09-16T10:26:35.329022+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:35.329022+0000","last_undegraded":"2024-09-16T10:26:35.329022+0000","last_fullsized":"2024-09-16T10:26:35.329022+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9232,"log_dups_size":0,"ondisk_log_size":9232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37806080,"num_objects":9230,"num_object_clones":0,"num_object_copies":18460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5035,"num_write_kb":20140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9202","reported_seq":17705,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:33.681607+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:33.681607+0000","last_peered":"2024-09-16T10:26:33.681607+0000","last_clean":"2024-09-16T10:26:33.681607+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:33.681607+0000","last_undegraded":"2024-09-16T10:26:33.681607+0000","last_fullsized":"2024-09-16T10:26:33.681607+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9202,"log_dups_size":0,"ondisk_log_size":9202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37663686,"num_objects":9197,"num_object_clones":0,"num_object_copies":18394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9200,"num_write_kb":36784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9476","reported_seq":9554,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.389225+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:35.389225+0000","last_peered":"2024-09-16T10:26:35.389225+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:35.389225+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38781894,"num_objects":9470,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":37876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5036},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9241","reported_seq":9343,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.884323+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:35.884323+0000","last_peered":"2024-09-16T10:26:35.884323+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:35.884323+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9241,"log_dups_size":0,"ondisk_log_size":9241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37831622,"num_objects":9238,"num_object_clones":0,"num_object_copies":18476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9239,"num_write_kb":36947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9276","reported_seq":9362,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.943113+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:35.943113+0000","last_peered":"2024-09-16T10:26:35.943113+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:35.943113+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37970886,"num_objects":9272,"num_object_clones":0,"num_object_copies":18544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":37083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9022","reported_seq":9099,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.395620+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:35.395620+0000","last_peered":"2024-09-16T10:26:35.395620+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:35.395620+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36946403,"num_objects":9021,"num_object_clones":0,"num_object_copies":18042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9021,"num_write_kb":36081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4862},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9431","reported_seq":9564,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.400860+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:35.400860+0000","last_peered":"2024-09-16T10:26:35.400860+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:35.400860+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38598057,"num_objects":9426,"num_object_clones":0,"num_object_copies":18852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5089},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9120","reported_seq":9200,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.351573+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:35.351573+0000","last_peered":"2024-09-16T10:26:35.351573+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:35.351573+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9120,"log_dups_size":0,"ondisk_log_size":9120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37340102,"num_objects":9118,"num_object_clones":0,"num_object_copies":18236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9118,"num_write_kb":36466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4951},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9463","reported_seq":9555,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.390913+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:35.390913+0000","last_peered":"2024-09-16T10:26:35.390913+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:35.390913+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9463,"log_dups_size":0,"ondisk_log_size":9463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38713228,"num_objects":9455,"num_object_clones":0,"num_object_copies":18910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9460,"num_write_kb":37811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5060},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":608334326,"num_objects":148535,"num_object_clones":0,"num_object_copies":297070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":148535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126164,"num_write_kb":504544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4809,"num_bytes_recovered":19616227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1234194432,"data_stored":1234111333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148599,"ondisk_log_size":148599,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018929,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690568,"kb_used_data":434176,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93681272,"statfs":{"total":96636764160,"available":95929622528,"internally_reserved":0,"allocated":444596224,"data_stored":444342590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":262532784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.4740000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":2.2970000000000002},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":2.5209999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451768,"kb_used_data":257712,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93920072,"statfs":{"total":96636764160,"available":96174153728,"internally_reserved":0,"allocated":263897088,"data_stored":263571194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":198700714},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":0.54700000000000004},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.1429999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":0.58799999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215157,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":812000,"kb_used_data":513816,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93559840,"statfs":{"total":96636764160,"available":95805276160,"internally_reserved":0,"allocated":526147584,"data_stored":525897564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":305327784},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,2,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.895},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":0.311}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":3.855},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":1.9350000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":526336000,"data_stored":526332387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263561216,"data_stored":263485343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":444297216,"data_stored":444293603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:37.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:37.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 205 20 149363 149343 2.84505 1.68359 0.0480044 0.0274535 2024-09-16T10:26:37.811 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:37.813 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:37.840 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-16T10:26:36.627495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":608334345,"num_objects":148537,"num_object_clones":0,"num_object_copies":297074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":148537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126166,"num_write_kb":504546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4810,"num_bytes_recovered":19616246,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148602,"ondisk_log_size":148602,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1954336,"kb_used_data":1205704,"kb_used_omap":12,"kb_used_meta":748595,"kb_avail":281161184,"statfs":{"total":289910292480,"available":287909052416,"internally_reserved":0,"allocated":1234640896,"data_stored":1233811348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":766561282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,4,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26853376,"num_objects":6556,"num_object_clones":0,"num_object_copies":13112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-237,"num_objects_unfound":0,"num_objects_dirty":6556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6556,"num_write_kb":26224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":970752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004662"},"pg_stats":[{"pgid":"2.f","version":"44'9280","reported_seq":10531,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:35.362181+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:35.362181+0000","last_peered":"2024-09-16T10:26:35.362181+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:35.362181+0000","last_undegraded":"2024-09-16T10:26:35.362181+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9280,"log_dups_size":0,"ondisk_log_size":9280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38010880,"num_objects":9280,"num_object_clones":0,"num_object_copies":18560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":9280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":37120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":591,"num_bytes_recovered":2420736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5679},{"shards":"0,2","objects":3601}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9282","reported_seq":9439,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.917205+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:35.917205+0000","last_peered":"2024-09-16T10:26:35.917205+0000","last_clean":"2024-09-16T10:26:35.917205+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:35.917205+0000","last_undegraded":"2024-09-16T10:26:35.917205+0000","last_fullsized":"2024-09-16T10:26:35.917205+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37994496,"num_objects":9276,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9309","reported_seq":9494,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.944093+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:35.944093+0000","last_peered":"2024-09-16T10:26:35.944093+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:35.944093+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9309,"log_dups_size":0,"ondisk_log_size":9309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38113280,"num_objects":9305,"num_object_clones":0,"num_object_copies":18610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":21464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4970}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9309","reported_seq":9365,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.854681+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:35.854681+0000","last_peered":"2024-09-16T10:26:35.854681+0000","last_clean":"2024-09-16T10:26:35.854681+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:35.854681+0000","last_undegraded":"2024-09-16T10:26:35.854681+0000","last_fullsized":"2024-09-16T10:26:35.854681+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9309,"log_dups_size":0,"ondisk_log_size":9309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38129664,"num_objects":9309,"num_object_clones":0,"num_object_copies":18618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9309,"num_write_kb":37236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9260","reported_seq":9416,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.417872+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:35.417872+0000","last_peered":"2024-09-16T10:26:35.417872+0000","last_clean":"2024-09-16T10:26:35.417872+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:35.417872+0000","last_undegraded":"2024-09-16T10:26:35.417872+0000","last_fullsized":"2024-09-16T10:26:35.417872+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9260,"log_dups_size":0,"ondisk_log_size":9260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900288,"num_objects":9253,"num_object_clones":0,"num_object_copies":18506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5902,"num_write_kb":23608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9197","reported_seq":9360,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.262487+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:35.262487+0000","last_peered":"2024-09-16T10:26:35.262487+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:35.262487+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37650432,"num_objects":9192,"num_object_clones":0,"num_object_copies":18384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4975},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9499","reported_seq":9672,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.305884+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:35.305884+0000","last_peered":"2024-09-16T10:26:35.305884+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:35.305884+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9499,"log_dups_size":0,"ondisk_log_size":9499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38883328,"num_objects":9493,"num_object_clones":0,"num_object_copies":18986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5741,"num_write_kb":22964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5090},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9232","reported_seq":9407,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.329022+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:35.329022+0000","last_peered":"2024-09-16T10:26:35.329022+0000","last_clean":"2024-09-16T10:26:35.329022+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:35.329022+0000","last_undegraded":"2024-09-16T10:26:35.329022+0000","last_fullsized":"2024-09-16T10:26:35.329022+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9232,"log_dups_size":0,"ondisk_log_size":9232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37806080,"num_objects":9230,"num_object_clones":0,"num_object_copies":18460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5035,"num_write_kb":20140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9202","reported_seq":17705,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:33.681607+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:33.681607+0000","last_peered":"2024-09-16T10:26:33.681607+0000","last_clean":"2024-09-16T10:26:33.681607+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:33.681607+0000","last_undegraded":"2024-09-16T10:26:33.681607+0000","last_fullsized":"2024-09-16T10:26:33.681607+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9202,"log_dups_size":0,"ondisk_log_size":9202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37663686,"num_objects":9197,"num_object_clones":0,"num_object_copies":18394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9200,"num_write_kb":36784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9476","reported_seq":9554,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.389225+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:35.389225+0000","last_peered":"2024-09-16T10:26:35.389225+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:35.389225+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38781894,"num_objects":9470,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":37876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5036},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9241","reported_seq":9343,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.884323+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:35.884323+0000","last_peered":"2024-09-16T10:26:35.884323+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:35.884323+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9241,"log_dups_size":0,"ondisk_log_size":9241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37831622,"num_objects":9238,"num_object_clones":0,"num_object_copies":18476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9239,"num_write_kb":36947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9276","reported_seq":9362,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.943113+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:35.943113+0000","last_peered":"2024-09-16T10:26:35.943113+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:35.943113+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37970886,"num_objects":9272,"num_object_clones":0,"num_object_copies":18544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":37083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9022","reported_seq":9099,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.395620+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:35.395620+0000","last_peered":"2024-09-16T10:26:35.395620+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:35.395620+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36946403,"num_objects":9021,"num_object_clones":0,"num_object_copies":18042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9021,"num_write_kb":36081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4862},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9431","reported_seq":9564,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.400860+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:35.400860+0000","last_peered":"2024-09-16T10:26:35.400860+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:35.400860+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38598057,"num_objects":9426,"num_object_clones":0,"num_object_copies":18852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5089},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9120","reported_seq":9200,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.351573+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:35.351573+0000","last_peered":"2024-09-16T10:26:35.351573+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:35.351573+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9120,"log_dups_size":0,"ondisk_log_size":9120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37340102,"num_objects":9118,"num_object_clones":0,"num_object_copies":18236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9118,"num_write_kb":36466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4951},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9463","reported_seq":9555,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.390913+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:35.390913+0000","last_peered":"2024-09-16T10:26:35.390913+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:35.390913+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9463,"log_dups_size":0,"ondisk_log_size":9463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38713228,"num_objects":9455,"num_object_clones":0,"num_object_copies":18910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9460,"num_write_kb":37811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5060},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":608334326,"num_objects":148535,"num_object_clones":0,"num_object_copies":297070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":148535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126164,"num_write_kb":504544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4809,"num_bytes_recovered":19616227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1234194432,"data_stored":1234111333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148599,"ondisk_log_size":148599,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018929,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690568,"kb_used_data":434176,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93681272,"statfs":{"total":96636764160,"available":95929622528,"internally_reserved":0,"allocated":444596224,"data_stored":444342590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":262532784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.4740000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":2.2970000000000002},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":2.5209999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451768,"kb_used_data":257712,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93920072,"statfs":{"total":96636764160,"available":96174153728,"internally_reserved":0,"allocated":263897088,"data_stored":263571194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":198700714},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":0.54700000000000004},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.1429999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":0.58799999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215157,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":812000,"kb_used_data":513816,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93559840,"statfs":{"total":96636764160,"available":95805276160,"internally_reserved":0,"allocated":526147584,"data_stored":525897564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":305327784},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,2,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.895},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":0.311}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":3.855},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":1.9350000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":526336000,"data_stored":526332387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263561216,"data_stored":263485343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":444297216,"data_stored":444293603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:37.841 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:37.842 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'9280", 'reported_seq': 10531, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:35.362181+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:35.362181+0000', 'last_peered': '2024-09-16T10:26:35.362181+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:35.362181+0000', 'last_undegraded': '2024-09-16T10:26:35.362181+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9280, 'log_dups_size': 0, 'ondisk_log_size': 9280, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38010880, 'num_objects': 9280, 'num_object_clones': 0, 'num_object_copies': 18560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3601, 'num_objects_unfound': 0, 'num_objects_dirty': 9280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9280, 'num_write_kb': 37120, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 591, 'num_bytes_recovered': 2420736, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5679}, {'shards': '0,2', 'objects': 3601}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:37.842 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:37.842 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'9309", 'reported_seq': 9494, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:35.944093+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:35.944093+0000', 'last_peered': '2024-09-16T10:26:35.944093+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:35.944093+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 9309, 'log_dups_size': 0, 'ondisk_log_size': 9309, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38113280, 'num_objects': 9305, 'num_object_clones': 0, 'num_object_copies': 18610, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5366, 'num_write_kb': 21464, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 4970}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:37.842 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:37.842 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'9197", 'reported_seq': 9360, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:35.262487+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:35.262487+0000', 'last_peered': '2024-09-16T10:26:35.262487+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:35.262487+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 9197, 'log_dups_size': 0, 'ondisk_log_size': 9197, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37650432, 'num_objects': 9192, 'num_object_clones': 0, 'num_object_copies': 18384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5822, 'num_write_kb': 23288, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4975}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:37.842 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:37.842 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'9499", 'reported_seq': 9672, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:35.305884+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:35.305884+0000', 'last_peered': '2024-09-16T10:26:35.305884+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:35.305884+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 9499, 'log_dups_size': 0, 'ondisk_log_size': 9499, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38883328, 'num_objects': 9493, 'num_object_clones': 0, 'num_object_copies': 18986, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5741, 'num_write_kb': 22964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5090}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:37.842 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:37.842 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'9476", 'reported_seq': 9554, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:35.389225+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:35.389225+0000', 'last_peered': '2024-09-16T10:26:35.389225+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:35.389225+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9476, 'log_dups_size': 0, 'ondisk_log_size': 9476, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38781894, 'num_objects': 9470, 'num_object_clones': 0, 'num_object_copies': 18940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9474, 'num_write_kb': 37876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5036}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:37.843 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:37.843 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'9241", 'reported_seq': 9343, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:35.884323+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:35.884323+0000', 'last_peered': '2024-09-16T10:26:35.884323+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:35.884323+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 9241, 'log_dups_size': 0, 'ondisk_log_size': 9241, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37831622, 'num_objects': 9238, 'num_object_clones': 0, 'num_object_copies': 18476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9239, 'num_write_kb': 36947, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 4961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:37.843 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:37.843 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'9276", 'reported_seq': 9362, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:35.943113+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:35.943113+0000', 'last_peered': '2024-09-16T10:26:35.943113+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:35.943113+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9276, 'log_dups_size': 0, 'ondisk_log_size': 9276, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37970886, 'num_objects': 9272, 'num_object_clones': 0, 'num_object_copies': 18544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9274, 'num_write_kb': 37083, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 5080}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:37.843 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:37.843 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'9022", 'reported_seq': 9099, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:35.395620+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:35.395620+0000', 'last_peered': '2024-09-16T10:26:35.395620+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:35.395620+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9022, 'log_dups_size': 0, 'ondisk_log_size': 9022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36946403, 'num_objects': 9021, 'num_object_clones': 0, 'num_object_copies': 18042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9021, 'num_write_kb': 36081, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4862}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:37.843 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:37.843 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'9431", 'reported_seq': 9564, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:35.400860+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:26:35.400860+0000', 'last_peered': '2024-09-16T10:26:35.400860+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:26:35.400860+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 9431, 'log_dups_size': 0, 'ondisk_log_size': 9431, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38598057, 'num_objects': 9426, 'num_object_clones': 0, 'num_object_copies': 18852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9426, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9428, 'num_write_kb': 37696, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5089}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:37.844 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:37.844 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'9120", 'reported_seq': 9200, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:35.351573+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:26:35.351573+0000', 'last_peered': '2024-09-16T10:26:35.351573+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:26:35.351573+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9120, 'log_dups_size': 0, 'ondisk_log_size': 9120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37340102, 'num_objects': 9118, 'num_object_clones': 0, 'num_object_copies': 18236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9118, 'num_write_kb': 36466, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4951}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:37.844 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:37.844 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'9463", 'reported_seq': 9555, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:35.390913+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:35.390913+0000', 'last_peered': '2024-09-16T10:26:35.390913+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:35.390913+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9463, 'log_dups_size': 0, 'ondisk_log_size': 9463, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38713228, 'num_objects': 9455, 'num_object_clones': 0, 'num_object_copies': 18910, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9460, 'num_write_kb': 37811, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5060}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:37.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:38.522 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:38.522 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/297074 objects degraded (14.453%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":287,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":148537,"data_bytes":608334345,"bytes_used":2001240064,"bytes_avail":287909052416,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":297074,"degraded_ratio":0.14452964581215455,"misplaced_objects":3601,"misplaced_total":297074,"misplaced_ratio":0.01212155893817702,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":80864,"recovering_keys_per_sec":0,"num_objects_recovered":237,"num_bytes_recovered":970752,"num_keys_recovered":0,"write_bytes_sec":2236912,"read_op_per_sec":0,"write_op_per_sec":546},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:38.544 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:38.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:38.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 206 20 149763 149743 2.83883 1.5625 0.0123625 0.0275116 2024-09-16T10:26:39.086 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:39.088 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:39.112 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-16T10:26:38.628149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":608334345,"num_objects":148537,"num_object_clones":0,"num_object_copies":297074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":148537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126166,"num_write_kb":504546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4810,"num_bytes_recovered":19616246,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148602,"ondisk_log_size":148602,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1954336,"kb_used_data":1205704,"kb_used_omap":12,"kb_used_meta":748595,"kb_avail":281161184,"statfs":{"total":289910292480,"available":287909052416,"internally_reserved":0,"allocated":1234640896,"data_stored":1233811348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":766561282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,4,9,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18157568,"num_objects":4433,"num_object_clones":0,"num_object_copies":8866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-164,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4433,"num_write_kb":17732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":164,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004757"},"pg_stats":[{"pgid":"2.f","version":"44'9280","reported_seq":10531,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:35.362181+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:35.362181+0000","last_peered":"2024-09-16T10:26:35.362181+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:35.362181+0000","last_undegraded":"2024-09-16T10:26:35.362181+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9280,"log_dups_size":0,"ondisk_log_size":9280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38010880,"num_objects":9280,"num_object_clones":0,"num_object_copies":18560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":9280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":37120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":591,"num_bytes_recovered":2420736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5679},{"shards":"0,2","objects":3601}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9282","reported_seq":9439,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.917205+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:35.917205+0000","last_peered":"2024-09-16T10:26:35.917205+0000","last_clean":"2024-09-16T10:26:35.917205+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:35.917205+0000","last_undegraded":"2024-09-16T10:26:35.917205+0000","last_fullsized":"2024-09-16T10:26:35.917205+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9282,"log_dups_size":0,"ondisk_log_size":9282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37994496,"num_objects":9276,"num_object_clones":0,"num_object_copies":18552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9309","reported_seq":9494,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.944093+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:35.944093+0000","last_peered":"2024-09-16T10:26:35.944093+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:35.944093+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9309,"log_dups_size":0,"ondisk_log_size":9309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38113280,"num_objects":9305,"num_object_clones":0,"num_object_copies":18610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":21464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":4970}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9309","reported_seq":9365,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.854681+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:35.854681+0000","last_peered":"2024-09-16T10:26:35.854681+0000","last_clean":"2024-09-16T10:26:35.854681+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:35.854681+0000","last_undegraded":"2024-09-16T10:26:35.854681+0000","last_fullsized":"2024-09-16T10:26:35.854681+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9309,"log_dups_size":0,"ondisk_log_size":9309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38129664,"num_objects":9309,"num_object_clones":0,"num_object_copies":18618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9309,"num_write_kb":37236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9260","reported_seq":9416,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.417872+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:35.417872+0000","last_peered":"2024-09-16T10:26:35.417872+0000","last_clean":"2024-09-16T10:26:35.417872+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:35.417872+0000","last_undegraded":"2024-09-16T10:26:35.417872+0000","last_fullsized":"2024-09-16T10:26:35.417872+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9260,"log_dups_size":0,"ondisk_log_size":9260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900288,"num_objects":9253,"num_object_clones":0,"num_object_copies":18506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5902,"num_write_kb":23608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9197","reported_seq":9360,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.262487+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:35.262487+0000","last_peered":"2024-09-16T10:26:35.262487+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:35.262487+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9197,"log_dups_size":0,"ondisk_log_size":9197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37650432,"num_objects":9192,"num_object_clones":0,"num_object_copies":18384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4975},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9499","reported_seq":9672,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.305884+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:35.305884+0000","last_peered":"2024-09-16T10:26:35.305884+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:35.305884+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9499,"log_dups_size":0,"ondisk_log_size":9499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38883328,"num_objects":9493,"num_object_clones":0,"num_object_copies":18986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5741,"num_write_kb":22964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5090},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9232","reported_seq":9407,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:35.329022+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:35.329022+0000","last_peered":"2024-09-16T10:26:35.329022+0000","last_clean":"2024-09-16T10:26:35.329022+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:35.329022+0000","last_undegraded":"2024-09-16T10:26:35.329022+0000","last_fullsized":"2024-09-16T10:26:35.329022+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9232,"log_dups_size":0,"ondisk_log_size":9232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37806080,"num_objects":9230,"num_object_clones":0,"num_object_copies":18460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5035,"num_write_kb":20140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9202","reported_seq":17705,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:33.681607+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:33.681607+0000","last_peered":"2024-09-16T10:26:33.681607+0000","last_clean":"2024-09-16T10:26:33.681607+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:33.681607+0000","last_undegraded":"2024-09-16T10:26:33.681607+0000","last_fullsized":"2024-09-16T10:26:33.681607+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9202,"log_dups_size":0,"ondisk_log_size":9202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37663686,"num_objects":9197,"num_object_clones":0,"num_object_copies":18394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9200,"num_write_kb":36784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9476","reported_seq":9554,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.389225+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:35.389225+0000","last_peered":"2024-09-16T10:26:35.389225+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:35.389225+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38781894,"num_objects":9470,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":37876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5036},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9241","reported_seq":9343,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.884323+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:35.884323+0000","last_peered":"2024-09-16T10:26:35.884323+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:35.884323+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9241,"log_dups_size":0,"ondisk_log_size":9241,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37831622,"num_objects":9238,"num_object_clones":0,"num_object_copies":18476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9239,"num_write_kb":36947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":4961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9276","reported_seq":9362,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.943113+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:35.943113+0000","last_peered":"2024-09-16T10:26:35.943113+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:35.943113+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9276,"log_dups_size":0,"ondisk_log_size":9276,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37970886,"num_objects":9272,"num_object_clones":0,"num_object_copies":18544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":37083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5080}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9022","reported_seq":9099,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.395620+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:35.395620+0000","last_peered":"2024-09-16T10:26:35.395620+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:35.395620+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36946403,"num_objects":9021,"num_object_clones":0,"num_object_copies":18042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9021,"num_write_kb":36081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4862},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9431","reported_seq":9564,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.400860+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:35.400860+0000","last_peered":"2024-09-16T10:26:35.400860+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:35.400860+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9431,"log_dups_size":0,"ondisk_log_size":9431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38598057,"num_objects":9426,"num_object_clones":0,"num_object_copies":18852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5089},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9120","reported_seq":9200,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.351573+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:35.351573+0000","last_peered":"2024-09-16T10:26:35.351573+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:35.351573+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9120,"log_dups_size":0,"ondisk_log_size":9120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37340102,"num_objects":9118,"num_object_clones":0,"num_object_copies":18236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9118,"num_write_kb":36466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4951},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9463","reported_seq":9555,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:35.390913+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:35.390913+0000","last_peered":"2024-09-16T10:26:35.390913+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:35.390913+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9463,"log_dups_size":0,"ondisk_log_size":9463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38713228,"num_objects":9455,"num_object_clones":0,"num_object_copies":18910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9460,"num_write_kb":37811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5060},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":608334326,"num_objects":148535,"num_object_clones":0,"num_object_copies":297070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3601,"num_objects_unfound":0,"num_objects_dirty":148535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126164,"num_write_kb":504544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4809,"num_bytes_recovered":19616227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1234194432,"data_stored":1234111333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148599,"ondisk_log_size":148599,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018929,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690568,"kb_used_data":434176,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93681272,"statfs":{"total":96636764160,"available":95929622528,"internally_reserved":0,"allocated":444596224,"data_stored":444342590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":262532784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.4740000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":2.2970000000000002},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":2.5209999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691873,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451768,"kb_used_data":257712,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93920072,"statfs":{"total":96636764160,"available":96174153728,"internally_reserved":0,"allocated":263897088,"data_stored":263571194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":198700714},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":0.54700000000000004},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.1429999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":0.58799999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215157,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":812000,"kb_used_data":513816,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93559840,"statfs":{"total":96636764160,"available":95805276160,"internally_reserved":0,"allocated":526147584,"data_stored":525897564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":305327784},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,2,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":1.895},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":0.311}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":3.855},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":1.9350000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":526336000,"data_stored":526332387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263561216,"data_stored":263485343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":444297216,"data_stored":444293603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:39.563 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 207 20 150201 150181 2.83338 1.71094 0.0293543 0.0275653 2024-09-16T10:26:40.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 208 20 150667 150647 2.82851 1.82031 0.056281 0.0276137 2024-09-16T10:26:41.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 209 20 151139 151119 2.8238 1.84375 0.0196657 0.02766 2024-09-16T10:26:42.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:42.562 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 210 20 151569 151549 2.81835 1.67969 0.0544533 0.0277119 2024-09-16T10:26:42.641 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:42.643 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:42.665 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-16T10:26:42.629678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617058825,"num_objects":150667,"num_object_clones":0,"num_object_copies":301334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":150667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128296,"num_write_kb":513066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4901,"num_bytes_recovered":19988982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150732,"ondisk_log_size":150732,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1967540,"kb_used_data":1223004,"kb_used_omap":12,"kb_used_meta":744499,"kb_avail":281147980,"statfs":{"total":289910292480,"available":287895531520,"internally_reserved":0,"allocated":1252356096,"data_stored":1251526548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":762366978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,4,5,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20570112,"num_objects":5022,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-172,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5022,"num_write_kb":20088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004534"},"pg_stats":[{"pgid":"2.f","version":"44'9416","reported_seq":10849,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:40.466154+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:40.466154+0000","last_peered":"2024-09-16T10:26:40.466154+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:40.466154+0000","last_undegraded":"2024-09-16T10:26:40.466154+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38567936,"num_objects":9416,"num_object_clones":0,"num_object_copies":18832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":9416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":37664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5906},{"shards":"0,2","objects":3510}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9423","reported_seq":9580,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.939447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:40.939447+0000","last_peered":"2024-09-16T10:26:40.939447+0000","last_clean":"2024-09-16T10:26:40.939447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:40.939447+0000","last_undegraded":"2024-09-16T10:26:40.939447+0000","last_fullsized":"2024-09-16T10:26:40.939447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38572032,"num_objects":9417,"num_object_clones":0,"num_object_copies":18834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5636,"num_write_kb":22544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9438","reported_seq":9623,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.923583+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:40.923583+0000","last_peered":"2024-09-16T10:26:40.923583+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:40.923583+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":9434,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5099}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9436","reported_seq":9492,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.944630+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:40.944630+0000","last_peered":"2024-09-16T10:26:40.944630+0000","last_clean":"2024-09-16T10:26:40.944630+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:40.944630+0000","last_undegraded":"2024-09-16T10:26:40.944630+0000","last_fullsized":"2024-09-16T10:26:40.944630+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9436,"log_dups_size":0,"ondisk_log_size":9436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38649856,"num_objects":9436,"num_object_clones":0,"num_object_copies":18872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9436,"num_write_kb":37744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9398","reported_seq":9553,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.452958+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:40.452958+0000","last_peered":"2024-09-16T10:26:40.452958+0000","last_clean":"2024-09-16T10:26:40.452958+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:40.452958+0000","last_undegraded":"2024-09-16T10:26:40.452958+0000","last_fullsized":"2024-09-16T10:26:40.452958+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9398,"log_dups_size":0,"ondisk_log_size":9398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38465536,"num_objects":9391,"num_object_clones":0,"num_object_copies":18782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":24160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9323","reported_seq":9486,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.361324+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:40.361324+0000","last_peered":"2024-09-16T10:26:40.361324+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:40.361324+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9323,"log_dups_size":0,"ondisk_log_size":9323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38166528,"num_objects":9318,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5948,"num_write_kb":23792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5101},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9623","reported_seq":9796,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.437261+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:40.437261+0000","last_peered":"2024-09-16T10:26:40.437261+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:40.437261+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39391232,"num_objects":9617,"num_object_clones":0,"num_object_copies":19234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5865,"num_write_kb":23460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5214},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9354","reported_seq":9529,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.375022+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:40.375022+0000","last_peered":"2024-09-16T10:26:40.375022+0000","last_clean":"2024-09-16T10:26:40.375022+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:40.375022+0000","last_undegraded":"2024-09-16T10:26:40.375022+0000","last_fullsized":"2024-09-16T10:26:40.375022+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38305792,"num_objects":9352,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":20628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9333","reported_seq":17836,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:38.753123+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:38.753123+0000","last_peered":"2024-09-16T10:26:38.753123+0000","last_clean":"2024-09-16T10:26:38.753123+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:38.753123+0000","last_undegraded":"2024-09-16T10:26:38.753123+0000","last_fullsized":"2024-09-16T10:26:38.753123+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9333,"log_dups_size":0,"ondisk_log_size":9333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38200262,"num_objects":9328,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9331,"num_write_kb":37308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9621","reported_seq":9698,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.398254+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:40.398254+0000","last_peered":"2024-09-16T10:26:40.398254+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:40.398254+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9621,"log_dups_size":0,"ondisk_log_size":9621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39375814,"num_objects":9615,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9619,"num_write_kb":38456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5181},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9362","reported_seq":9466,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.975149+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:40.975149+0000","last_peered":"2024-09-16T10:26:40.975149+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:40.975149+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9362,"log_dups_size":0,"ondisk_log_size":9362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38327238,"num_objects":9359,"num_object_clones":0,"num_object_copies":18718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9360,"num_write_kb":37431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5082}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9396","reported_seq":9480,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.926997+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:40.926997+0000","last_peered":"2024-09-16T10:26:40.926997+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:40.926997+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38462406,"num_objects":9392,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9394,"num_write_kb":37563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9156","reported_seq":9234,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.163170+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:40.163170+0000","last_peered":"2024-09-16T10:26:40.163170+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:40.163170+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9156,"log_dups_size":0,"ondisk_log_size":9156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37495267,"num_objects":9155,"num_object_clones":0,"num_object_copies":18310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9155,"num_write_kb":36617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4996},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9576","reported_seq":9709,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.264780+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:40.264780+0000","last_peered":"2024-09-16T10:26:40.264780+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:40.264780+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9576,"log_dups_size":0,"ondisk_log_size":9576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39191977,"num_objects":9571,"num_object_clones":0,"num_object_copies":19142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9573,"num_write_kb":38276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5234},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9251","reported_seq":9331,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.380449+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:40.380449+0000","last_peered":"2024-09-16T10:26:40.380449+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:40.380449+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876678,"num_objects":9249,"num_object_clones":0,"num_object_copies":18498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":36990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5082},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9623","reported_seq":9715,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.437741+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:40.437741+0000","last_peered":"2024-09-16T10:26:40.437741+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:40.437741+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39368588,"num_objects":9615,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9620,"num_write_kb":38451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5220},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617058806,"num_objects":150665,"num_object_clones":0,"num_object_copies":301330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":150665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128294,"num_write_kb":513064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4900,"num_bytes_recovered":19988963,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1252077568,"data_stored":1251994469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150729,"ondisk_log_size":150729,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018930,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682372,"kb_used_data":438268,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93689468,"statfs":{"total":96636764160,"available":95938015232,"internally_reserved":0,"allocated":448786432,"data_stored":448532798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":249949872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.593}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":1.548}]}]},{"osd":1,"up_from":40,"seq":171798691874,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462612,"kb_used_data":264460,"kb_used_omap":4,"kb_used_meta":198139,"kb_avail":93909228,"statfs":{"total":96636764160,"available":96163049472,"internally_reserved":0,"allocated":270807040,"data_stored":270481146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":202895018},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":3.1429999999999998},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.1099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":5.133}]}]},{"osd":0,"up_from":24,"seq":103079215158,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":822556,"kb_used_data":520276,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93549284,"statfs":{"total":96636764160,"available":95794466816,"internally_reserved":0,"allocated":532762624,"data_stored":532512604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":309522088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,1,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":2.734},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.3759999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":3.109},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.80400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532963328,"data_stored":532959715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270594048,"data_stored":270518175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":448520192,"data_stored":448516579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:42.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:43.225 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:43.227 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:43.248 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-16T10:26:42.629678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617058825,"num_objects":150667,"num_object_clones":0,"num_object_copies":301334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":150667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128296,"num_write_kb":513066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4901,"num_bytes_recovered":19988982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150732,"ondisk_log_size":150732,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1967540,"kb_used_data":1223004,"kb_used_omap":12,"kb_used_meta":744499,"kb_avail":281147980,"statfs":{"total":289910292480,"available":287895531520,"internally_reserved":0,"allocated":1252356096,"data_stored":1251526548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":762366978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,4,5,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20570112,"num_objects":5022,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-172,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5022,"num_write_kb":20088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004534"},"pg_stats":[{"pgid":"2.f","version":"44'9416","reported_seq":10849,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:40.466154+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:40.466154+0000","last_peered":"2024-09-16T10:26:40.466154+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:40.466154+0000","last_undegraded":"2024-09-16T10:26:40.466154+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38567936,"num_objects":9416,"num_object_clones":0,"num_object_copies":18832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":9416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":37664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5906},{"shards":"0,2","objects":3510}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9423","reported_seq":9580,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.939447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:40.939447+0000","last_peered":"2024-09-16T10:26:40.939447+0000","last_clean":"2024-09-16T10:26:40.939447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:40.939447+0000","last_undegraded":"2024-09-16T10:26:40.939447+0000","last_fullsized":"2024-09-16T10:26:40.939447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38572032,"num_objects":9417,"num_object_clones":0,"num_object_copies":18834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5636,"num_write_kb":22544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9438","reported_seq":9623,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.923583+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:40.923583+0000","last_peered":"2024-09-16T10:26:40.923583+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:40.923583+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":9434,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5099}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9436","reported_seq":9492,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.944630+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:40.944630+0000","last_peered":"2024-09-16T10:26:40.944630+0000","last_clean":"2024-09-16T10:26:40.944630+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:40.944630+0000","last_undegraded":"2024-09-16T10:26:40.944630+0000","last_fullsized":"2024-09-16T10:26:40.944630+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9436,"log_dups_size":0,"ondisk_log_size":9436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38649856,"num_objects":9436,"num_object_clones":0,"num_object_copies":18872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9436,"num_write_kb":37744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9398","reported_seq":9553,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.452958+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:40.452958+0000","last_peered":"2024-09-16T10:26:40.452958+0000","last_clean":"2024-09-16T10:26:40.452958+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:40.452958+0000","last_undegraded":"2024-09-16T10:26:40.452958+0000","last_fullsized":"2024-09-16T10:26:40.452958+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9398,"log_dups_size":0,"ondisk_log_size":9398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38465536,"num_objects":9391,"num_object_clones":0,"num_object_copies":18782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":24160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9323","reported_seq":9486,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.361324+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:40.361324+0000","last_peered":"2024-09-16T10:26:40.361324+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:40.361324+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9323,"log_dups_size":0,"ondisk_log_size":9323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38166528,"num_objects":9318,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5948,"num_write_kb":23792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5101},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9623","reported_seq":9796,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.437261+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:40.437261+0000","last_peered":"2024-09-16T10:26:40.437261+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:40.437261+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39391232,"num_objects":9617,"num_object_clones":0,"num_object_copies":19234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5865,"num_write_kb":23460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5214},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9354","reported_seq":9529,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.375022+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:40.375022+0000","last_peered":"2024-09-16T10:26:40.375022+0000","last_clean":"2024-09-16T10:26:40.375022+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:40.375022+0000","last_undegraded":"2024-09-16T10:26:40.375022+0000","last_fullsized":"2024-09-16T10:26:40.375022+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38305792,"num_objects":9352,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":20628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9333","reported_seq":17836,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:38.753123+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:38.753123+0000","last_peered":"2024-09-16T10:26:38.753123+0000","last_clean":"2024-09-16T10:26:38.753123+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:38.753123+0000","last_undegraded":"2024-09-16T10:26:38.753123+0000","last_fullsized":"2024-09-16T10:26:38.753123+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9333,"log_dups_size":0,"ondisk_log_size":9333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38200262,"num_objects":9328,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9331,"num_write_kb":37308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9621","reported_seq":9698,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.398254+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:40.398254+0000","last_peered":"2024-09-16T10:26:40.398254+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:40.398254+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9621,"log_dups_size":0,"ondisk_log_size":9621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39375814,"num_objects":9615,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9619,"num_write_kb":38456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5181},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9362","reported_seq":9466,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.975149+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:40.975149+0000","last_peered":"2024-09-16T10:26:40.975149+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:40.975149+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9362,"log_dups_size":0,"ondisk_log_size":9362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38327238,"num_objects":9359,"num_object_clones":0,"num_object_copies":18718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9360,"num_write_kb":37431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5082}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9396","reported_seq":9480,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.926997+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:40.926997+0000","last_peered":"2024-09-16T10:26:40.926997+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:40.926997+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38462406,"num_objects":9392,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9394,"num_write_kb":37563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9156","reported_seq":9234,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.163170+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:40.163170+0000","last_peered":"2024-09-16T10:26:40.163170+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:40.163170+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9156,"log_dups_size":0,"ondisk_log_size":9156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37495267,"num_objects":9155,"num_object_clones":0,"num_object_copies":18310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9155,"num_write_kb":36617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4996},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9576","reported_seq":9709,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.264780+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:40.264780+0000","last_peered":"2024-09-16T10:26:40.264780+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:40.264780+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9576,"log_dups_size":0,"ondisk_log_size":9576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39191977,"num_objects":9571,"num_object_clones":0,"num_object_copies":19142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9573,"num_write_kb":38276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5234},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9251","reported_seq":9331,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.380449+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:40.380449+0000","last_peered":"2024-09-16T10:26:40.380449+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:40.380449+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876678,"num_objects":9249,"num_object_clones":0,"num_object_copies":18498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":36990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5082},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9623","reported_seq":9715,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.437741+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:40.437741+0000","last_peered":"2024-09-16T10:26:40.437741+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:40.437741+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39368588,"num_objects":9615,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9620,"num_write_kb":38451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5220},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617058806,"num_objects":150665,"num_object_clones":0,"num_object_copies":301330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":150665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128294,"num_write_kb":513064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4900,"num_bytes_recovered":19988963,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1252077568,"data_stored":1251994469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150729,"ondisk_log_size":150729,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018930,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682372,"kb_used_data":438268,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93689468,"statfs":{"total":96636764160,"available":95938015232,"internally_reserved":0,"allocated":448786432,"data_stored":448532798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":249949872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.593}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":1.548}]}]},{"osd":1,"up_from":40,"seq":171798691874,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462612,"kb_used_data":264460,"kb_used_omap":4,"kb_used_meta":198139,"kb_avail":93909228,"statfs":{"total":96636764160,"available":96163049472,"internally_reserved":0,"allocated":270807040,"data_stored":270481146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":202895018},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":3.1429999999999998},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.1099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":5.133}]}]},{"osd":0,"up_from":24,"seq":103079215158,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":822556,"kb_used_data":520276,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93549284,"statfs":{"total":96636764160,"available":95794466816,"internally_reserved":0,"allocated":532762624,"data_stored":532512604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":309522088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,1,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":2.734},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.3759999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":3.109},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.80400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532963328,"data_stored":532959715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270594048,"data_stored":270518175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":448520192,"data_stored":448516579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:43.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:43.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'9416", 'reported_seq': 10849, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:40.466154+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:40.466154+0000', 'last_peered': '2024-09-16T10:26:40.466154+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:40.466154+0000', 'last_undegraded': '2024-09-16T10:26:40.466154+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9416, 'log_dups_size': 0, 'ondisk_log_size': 9416, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38567936, 'num_objects': 9416, 'num_object_clones': 0, 'num_object_copies': 18832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3510, 'num_objects_unfound': 0, 'num_objects_dirty': 9416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9416, 'num_write_kb': 37664, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 682, 'num_bytes_recovered': 2793472, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5906}, {'shards': '0,2', 'objects': 3510}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:43.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:43.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'9438", 'reported_seq': 9623, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:40.923583+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:40.923583+0000', 'last_peered': '2024-09-16T10:26:40.923583+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:40.923583+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 9438, 'log_dups_size': 0, 'ondisk_log_size': 9438, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38641664, 'num_objects': 9434, 'num_object_clones': 0, 'num_object_copies': 18868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5495, 'num_write_kb': 21980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 5099}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:43.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:43.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'9323", 'reported_seq': 9486, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:40.361324+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:40.361324+0000', 'last_peered': '2024-09-16T10:26:40.361324+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:40.361324+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 9323, 'log_dups_size': 0, 'ondisk_log_size': 9323, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38166528, 'num_objects': 9318, 'num_object_clones': 0, 'num_object_copies': 18636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5948, 'num_write_kb': 23792, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5101}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:43.250 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:43.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'9623", 'reported_seq': 9796, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:40.437261+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:40.437261+0000', 'last_peered': '2024-09-16T10:26:40.437261+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:40.437261+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 9623, 'log_dups_size': 0, 'ondisk_log_size': 9623, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39391232, 'num_objects': 9617, 'num_object_clones': 0, 'num_object_copies': 19234, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5865, 'num_write_kb': 23460, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5214}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:43.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:43.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'9621", 'reported_seq': 9698, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:40.398254+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:40.398254+0000', 'last_peered': '2024-09-16T10:26:40.398254+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:40.398254+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9621, 'log_dups_size': 0, 'ondisk_log_size': 9621, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39375814, 'num_objects': 9615, 'num_object_clones': 0, 'num_object_copies': 19230, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9619, 'num_write_kb': 38456, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5181}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:43.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:43.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'9362", 'reported_seq': 9466, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:40.975149+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:40.975149+0000', 'last_peered': '2024-09-16T10:26:40.975149+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:40.975149+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 9362, 'log_dups_size': 0, 'ondisk_log_size': 9362, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38327238, 'num_objects': 9359, 'num_object_clones': 0, 'num_object_copies': 18718, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9360, 'num_write_kb': 37431, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 5082}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:43.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:43.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'9396", 'reported_seq': 9480, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:40.926997+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:40.926997+0000', 'last_peered': '2024-09-16T10:26:40.926997+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:40.926997+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9396, 'log_dups_size': 0, 'ondisk_log_size': 9396, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38462406, 'num_objects': 9392, 'num_object_clones': 0, 'num_object_copies': 18784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9394, 'num_write_kb': 37563, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 5200}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:43.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:43.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'9156", 'reported_seq': 9234, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:40.163170+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:40.163170+0000', 'last_peered': '2024-09-16T10:26:40.163170+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:40.163170+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9156, 'log_dups_size': 0, 'ondisk_log_size': 9156, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37495267, 'num_objects': 9155, 'num_object_clones': 0, 'num_object_copies': 18310, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9155, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9155, 'num_write_kb': 36617, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4996}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:43.251 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:43.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'9576", 'reported_seq': 9709, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:40.264780+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:26:40.264780+0000', 'last_peered': '2024-09-16T10:26:40.264780+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:26:40.264780+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 9576, 'log_dups_size': 0, 'ondisk_log_size': 9576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39191977, 'num_objects': 9571, 'num_object_clones': 0, 'num_object_copies': 19142, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9571, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9573, 'num_write_kb': 38276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5234}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:43.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:43.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'9251", 'reported_seq': 9331, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:40.380449+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:26:40.380449+0000', 'last_peered': '2024-09-16T10:26:40.380449+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:26:40.380449+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9251, 'log_dups_size': 0, 'ondisk_log_size': 9251, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37876678, 'num_objects': 9249, 'num_object_clones': 0, 'num_object_copies': 18498, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9249, 'num_write_kb': 36990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5082}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:43.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:43.252 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'9623", 'reported_seq': 9715, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:40.437741+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:40.437741+0000', 'last_peered': '2024-09-16T10:26:40.437741+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:40.437741+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9623, 'log_dups_size': 0, 'ondisk_log_size': 9623, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39368588, 'num_objects': 9615, 'num_object_clones': 0, 'num_object_copies': 19230, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9615, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9620, 'num_write_kb': 38451, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5220}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:43.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:43.563 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 211 20 151985 151965 2.81269 1.625 0.0441659 0.0277693 2024-09-16T10:26:43.948 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:43.948 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/301334 objects degraded (14.249%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":292,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":150667,"data_bytes":617058825,"bytes_used":2014760960,"bytes_avail":287895531520,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":301334,"degraded_ratio":0.14248641042829552,"misplaced_objects":3510,"misplaced_total":301334,"misplaced_ratio":0.01164820431813204,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":58687,"recovering_keys_per_sec":0,"num_objects_recovered":172,"num_bytes_recovered":704512,"num_keys_recovered":0,"write_bytes_sec":1713528,"read_op_per_sec":0,"write_op_per_sec":418},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:43.973 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:43.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:44.548 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:44.549 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:44.563 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 212 20 152388 152368 2.80685 1.57422 0.0276361 0.0278259 2024-09-16T10:26:44.572 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-16T10:26:42.629678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617058825,"num_objects":150667,"num_object_clones":0,"num_object_copies":301334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":150667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128296,"num_write_kb":513066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4901,"num_bytes_recovered":19988982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150732,"ondisk_log_size":150732,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1967540,"kb_used_data":1223004,"kb_used_omap":12,"kb_used_meta":744499,"kb_avail":281147980,"statfs":{"total":289910292480,"available":287895531520,"internally_reserved":0,"allocated":1252356096,"data_stored":1251526548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":762366978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,4,5,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20570112,"num_objects":5022,"num_object_clones":0,"num_object_copies":10044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-172,"num_objects_unfound":0,"num_objects_dirty":5022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5022,"num_write_kb":20088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004534"},"pg_stats":[{"pgid":"2.f","version":"44'9416","reported_seq":10849,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:40.466154+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:40.466154+0000","last_peered":"2024-09-16T10:26:40.466154+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:40.466154+0000","last_undegraded":"2024-09-16T10:26:40.466154+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9416,"log_dups_size":0,"ondisk_log_size":9416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38567936,"num_objects":9416,"num_object_clones":0,"num_object_copies":18832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":9416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9416,"num_write_kb":37664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":2793472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5906},{"shards":"0,2","objects":3510}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9423","reported_seq":9580,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.939447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:40.939447+0000","last_peered":"2024-09-16T10:26:40.939447+0000","last_clean":"2024-09-16T10:26:40.939447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:40.939447+0000","last_undegraded":"2024-09-16T10:26:40.939447+0000","last_fullsized":"2024-09-16T10:26:40.939447+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38572032,"num_objects":9417,"num_object_clones":0,"num_object_copies":18834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5636,"num_write_kb":22544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9438","reported_seq":9623,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.923583+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:40.923583+0000","last_peered":"2024-09-16T10:26:40.923583+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:40.923583+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9438,"log_dups_size":0,"ondisk_log_size":9438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38641664,"num_objects":9434,"num_object_clones":0,"num_object_copies":18868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":21980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5099}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9436","reported_seq":9492,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.944630+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:40.944630+0000","last_peered":"2024-09-16T10:26:40.944630+0000","last_clean":"2024-09-16T10:26:40.944630+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:40.944630+0000","last_undegraded":"2024-09-16T10:26:40.944630+0000","last_fullsized":"2024-09-16T10:26:40.944630+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9436,"log_dups_size":0,"ondisk_log_size":9436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38649856,"num_objects":9436,"num_object_clones":0,"num_object_copies":18872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9436,"num_write_kb":37744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9398","reported_seq":9553,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.452958+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:40.452958+0000","last_peered":"2024-09-16T10:26:40.452958+0000","last_clean":"2024-09-16T10:26:40.452958+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:40.452958+0000","last_undegraded":"2024-09-16T10:26:40.452958+0000","last_fullsized":"2024-09-16T10:26:40.452958+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9398,"log_dups_size":0,"ondisk_log_size":9398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38465536,"num_objects":9391,"num_object_clones":0,"num_object_copies":18782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6040,"num_write_kb":24160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9323","reported_seq":9486,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.361324+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:40.361324+0000","last_peered":"2024-09-16T10:26:40.361324+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:40.361324+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9323,"log_dups_size":0,"ondisk_log_size":9323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38166528,"num_objects":9318,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5948,"num_write_kb":23792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5101},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9623","reported_seq":9796,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.437261+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:40.437261+0000","last_peered":"2024-09-16T10:26:40.437261+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:40.437261+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39391232,"num_objects":9617,"num_object_clones":0,"num_object_copies":19234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5865,"num_write_kb":23460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5214},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9354","reported_seq":9529,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:40.375022+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:40.375022+0000","last_peered":"2024-09-16T10:26:40.375022+0000","last_clean":"2024-09-16T10:26:40.375022+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:40.375022+0000","last_undegraded":"2024-09-16T10:26:40.375022+0000","last_fullsized":"2024-09-16T10:26:40.375022+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38305792,"num_objects":9352,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5157,"num_write_kb":20628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9333","reported_seq":17836,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:38.753123+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:38.753123+0000","last_peered":"2024-09-16T10:26:38.753123+0000","last_clean":"2024-09-16T10:26:38.753123+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:38.753123+0000","last_undegraded":"2024-09-16T10:26:38.753123+0000","last_fullsized":"2024-09-16T10:26:38.753123+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9333,"log_dups_size":0,"ondisk_log_size":9333,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38200262,"num_objects":9328,"num_object_clones":0,"num_object_copies":18656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9331,"num_write_kb":37308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9621","reported_seq":9698,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.398254+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:40.398254+0000","last_peered":"2024-09-16T10:26:40.398254+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:40.398254+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9621,"log_dups_size":0,"ondisk_log_size":9621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39375814,"num_objects":9615,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9619,"num_write_kb":38456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5181},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":87,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.975194+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:00.975194+0000","last_peered":"2024-09-16T10:26:00.975194+0000","last_clean":"2024-09-16T10:26:00.975194+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:00.975194+0000","last_undegraded":"2024-09-16T10:26:00.975194+0000","last_fullsized":"2024-09-16T10:26:00.975194+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9362","reported_seq":9466,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.975149+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:40.975149+0000","last_peered":"2024-09-16T10:26:40.975149+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:40.975149+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9362,"log_dups_size":0,"ondisk_log_size":9362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38327238,"num_objects":9359,"num_object_clones":0,"num_object_copies":18718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9360,"num_write_kb":37431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5082}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9396","reported_seq":9480,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.926997+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:40.926997+0000","last_peered":"2024-09-16T10:26:40.926997+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:40.926997+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9396,"log_dups_size":0,"ondisk_log_size":9396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38462406,"num_objects":9392,"num_object_clones":0,"num_object_copies":18784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9394,"num_write_kb":37563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9156","reported_seq":9234,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.163170+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:40.163170+0000","last_peered":"2024-09-16T10:26:40.163170+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:40.163170+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9156,"log_dups_size":0,"ondisk_log_size":9156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37495267,"num_objects":9155,"num_object_clones":0,"num_object_copies":18310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9155,"num_write_kb":36617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4996},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9576","reported_seq":9709,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.264780+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:40.264780+0000","last_peered":"2024-09-16T10:26:40.264780+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:40.264780+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9576,"log_dups_size":0,"ondisk_log_size":9576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39191977,"num_objects":9571,"num_object_clones":0,"num_object_copies":19142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9573,"num_write_kb":38276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5234},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9251","reported_seq":9331,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.380449+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:40.380449+0000","last_peered":"2024-09-16T10:26:40.380449+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:40.380449+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9251,"log_dups_size":0,"ondisk_log_size":9251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37876678,"num_objects":9249,"num_object_clones":0,"num_object_copies":18498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":36990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5082},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9623","reported_seq":9715,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:40.437741+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:40.437741+0000","last_peered":"2024-09-16T10:26:40.437741+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:40.437741+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9623,"log_dups_size":0,"ondisk_log_size":9623,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39368588,"num_objects":9615,"num_object_clones":0,"num_object_copies":19230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9620,"num_write_kb":38451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5220},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":617058806,"num_objects":150665,"num_object_clones":0,"num_object_copies":301330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":150665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":128294,"num_write_kb":513064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4900,"num_bytes_recovered":19988963,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1252077568,"data_stored":1251994469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150729,"ondisk_log_size":150729,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018930,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":682372,"kb_used_data":438268,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93689468,"statfs":{"total":96636764160,"available":95938015232,"internally_reserved":0,"allocated":448786432,"data_stored":448532798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":249949872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.593}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":1.548}]}]},{"osd":1,"up_from":40,"seq":171798691874,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":462612,"kb_used_data":264460,"kb_used_omap":4,"kb_used_meta":198139,"kb_avail":93909228,"statfs":{"total":96636764160,"available":96163049472,"internally_reserved":0,"allocated":270807040,"data_stored":270481146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":202895018},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":3.1429999999999998},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.1099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.52000000000000002},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":5.133}]}]},{"osd":0,"up_from":24,"seq":103079215158,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":822556,"kb_used_data":520276,"kb_used_omap":4,"kb_used_meta":302267,"kb_avail":93549284,"statfs":{"total":96636764160,"available":95794466816,"internally_reserved":0,"allocated":532762624,"data_stored":532512604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":309522088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,0,1,4,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":2.734},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":1.3759999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":3.109},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.80400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532963328,"data_stored":532959715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":270594048,"data_stored":270518175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":448520192,"data_stored":448516579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:45.563 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 213 20 152849 152829 2.80212 1.80078 0.0128418 0.0278717 2024-09-16T10:26:46.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 214 20 153304 153284 2.79733 1.77734 0.0124853 0.027921 2024-09-16T10:26:47.563 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 215 20 153759 153739 2.79259 1.77734 0.0491036 0.0279689 2024-09-16T10:26:47.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:48.122 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:48.124 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:48.146 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-16T10:26:46.631057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":626016777,"num_objects":152854,"num_object_clones":0,"num_object_copies":305708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3436,"num_objects_unfound":0,"num_objects_dirty":152854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130483,"num_write_kb":521814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20292086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152919,"ondisk_log_size":152919,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2001740,"kb_used_data":1240820,"kb_used_omap":12,"kb_used_meta":760883,"kb_avail":281113780,"statfs":{"total":289910292480,"available":287860510720,"internally_reserved":0,"allocated":1270599680,"data_stored":1269770132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":779144194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,4,11,5,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26136576,"num_objects":6381,"num_object_clones":0,"num_object_copies":12762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-246,"num_objects_unfound":0,"num_objects_dirty":6381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6381,"num_write_kb":25524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004290"},"pg_stats":[{"pgid":"2.f","version":"44'9552","reported_seq":11131,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:45.532176+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:45.532176+0000","last_peered":"2024-09-16T10:26:45.532176+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:45.532176+0000","last_undegraded":"2024-09-16T10:26:45.532176+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9552,"log_dups_size":0,"ondisk_log_size":9552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39124992,"num_objects":9552,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3436,"num_objects_unfound":0,"num_objects_dirty":9552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9552,"num_write_kb":38208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":3096576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6116},{"shards":"0,2","objects":3436}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9560","reported_seq":9717,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.981981+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:45.981981+0000","last_peered":"2024-09-16T10:26:45.981981+0000","last_clean":"2024-09-16T10:26:45.981981+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:45.981981+0000","last_undegraded":"2024-09-16T10:26:45.981981+0000","last_fullsized":"2024-09-16T10:26:45.981981+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9560,"log_dups_size":0,"ondisk_log_size":9560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39133184,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5773,"num_write_kb":23092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9591","reported_seq":9776,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.952884+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:45.952884+0000","last_peered":"2024-09-16T10:26:45.952884+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:45.952884+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9591,"log_dups_size":0,"ondisk_log_size":9591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39268352,"num_objects":9587,"num_object_clones":0,"num_object_copies":19174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":22592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9571","reported_seq":9627,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.808576+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:45.808576+0000","last_peered":"2024-09-16T10:26:45.808576+0000","last_clean":"2024-09-16T10:26:45.808576+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:45.808576+0000","last_undegraded":"2024-09-16T10:26:45.808576+0000","last_fullsized":"2024-09-16T10:26:45.808576+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9571,"log_dups_size":0,"ondisk_log_size":9571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39202816,"num_objects":9571,"num_object_clones":0,"num_object_copies":19142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":38284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9537","reported_seq":9693,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.454124+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:45.454124+0000","last_peered":"2024-09-16T10:26:45.454124+0000","last_clean":"2024-09-16T10:26:45.454124+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:45.454124+0000","last_undegraded":"2024-09-16T10:26:45.454124+0000","last_fullsized":"2024-09-16T10:26:45.454124+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":9530,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6179,"num_write_kb":24716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9454","reported_seq":9616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.395577+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:45.395577+0000","last_peered":"2024-09-16T10:26:45.395577+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:45.395577+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38703104,"num_objects":9449,"num_object_clones":0,"num_object_copies":18898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":24316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5232},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9762","reported_seq":9935,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.489897+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:45.489897+0000","last_peered":"2024-09-16T10:26:45.489897+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:45.489897+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39960576,"num_objects":9756,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6004,"num_write_kb":24016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5353},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9485","reported_seq":9660,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.481719+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:45.481719+0000","last_peered":"2024-09-16T10:26:45.481719+0000","last_clean":"2024-09-16T10:26:45.481719+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:45.481719+0000","last_undegraded":"2024-09-16T10:26:45.481719+0000","last_fullsized":"2024-09-16T10:26:45.481719+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9485,"log_dups_size":0,"ondisk_log_size":9485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842368,"num_objects":9483,"num_object_clones":0,"num_object_copies":18966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":21152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9464","reported_seq":17967,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:43.837864+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:43.837864+0000","last_peered":"2024-09-16T10:26:43.837864+0000","last_clean":"2024-09-16T10:26:43.837864+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:43.837864+0000","last_undegraded":"2024-09-16T10:26:43.837864+0000","last_fullsized":"2024-09-16T10:26:43.837864+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9464,"log_dups_size":0,"ondisk_log_size":9464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38736838,"num_objects":9459,"num_object_clones":0,"num_object_copies":18918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9462,"num_write_kb":37832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9765","reported_seq":9843,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.506693+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:45.506693+0000","last_peered":"2024-09-16T10:26:45.506693+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:45.506693+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965638,"num_objects":9759,"num_object_clones":0,"num_object_copies":19518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5325},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9502","reported_seq":9607,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.952754+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:45.952754+0000","last_peered":"2024-09-16T10:26:45.952754+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:45.952754+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900678,"num_objects":9499,"num_object_clones":0,"num_object_copies":18998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":37991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5222}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9535","reported_seq":9621,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.813724+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:45.813724+0000","last_peered":"2024-09-16T10:26:45.813724+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:45.813724+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9535,"log_dups_size":0,"ondisk_log_size":9535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39031750,"num_objects":9531,"num_object_clones":0,"num_object_copies":19062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9533,"num_write_kb":38119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9293","reported_seq":9372,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.458167+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:45.458167+0000","last_peered":"2024-09-16T10:26:45.458167+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:45.458167+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9293,"log_dups_size":0,"ondisk_log_size":9293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38056419,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":37165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5133},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9716","reported_seq":9849,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.452474+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:45.452474+0000","last_peered":"2024-09-16T10:26:45.452474+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:45.452474+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39765417,"num_objects":9711,"num_object_clones":0,"num_object_copies":19422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9713,"num_write_kb":38836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5374},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9382","reported_seq":9460,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.506830+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:45.506830+0000","last_peered":"2024-09-16T10:26:45.506830+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:45.506830+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38413254,"num_objects":9380,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":37514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5213},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9747","reported_seq":9839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.425245+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:45.425245+0000","last_peered":"2024-09-16T10:26:45.425245+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:45.425245+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39876492,"num_objects":9739,"num_object_clones":0,"num_object_copies":19478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":38947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5344},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":626016758,"num_objects":152852,"num_object_clones":0,"num_object_copies":305704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3436,"num_objects_unfound":0,"num_objects_dirty":152852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130481,"num_write_kb":521812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4974,"num_bytes_recovered":20292067,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1270554624,"data_stored":1270471525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152916,"ondisk_log_size":152916,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018931,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690908,"kb_used_data":442708,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93680932,"statfs":{"total":96636764160,"available":95929274368,"internally_reserved":0,"allocated":453332992,"data_stored":453079358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":254144176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.54800000000000004}]}]},{"osd":1,"up_from":40,"seq":171798691875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473588,"kb_used_data":271340,"kb_used_omap":4,"kb_used_meta":202235,"kb_avail":93898252,"statfs":{"total":96636764160,"available":96151810048,"internally_reserved":0,"allocated":277852160,"data_stored":277526266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":207089322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":12.569000000000001},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":8.6590000000000007}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":4.5670000000000002},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":3.258}]}]},{"osd":0,"up_from":24,"seq":103079215159,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":837244,"kb_used_data":526772,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93534596,"statfs":{"total":96636764160,"available":95779426304,"internally_reserved":0,"allocated":539414528,"data_stored":539164508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":317910696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,8,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":5.8040000000000003},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":5.2569999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":5.2270000000000003},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.867}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":539705344,"data_stored":539701731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277782528,"data_stored":277706655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":453066752,"data_stored":453063139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:48.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:48.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 216 20 154165 154145 2.787 1.58594 0.0360953 0.0280231 2024-09-16T10:26:48.728 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:48.728 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:48.753 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-16T10:26:48.631586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":626016777,"num_objects":152854,"num_object_clones":0,"num_object_copies":305708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3436,"num_objects_unfound":0,"num_objects_dirty":152854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130483,"num_write_kb":521814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20292086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152919,"ondisk_log_size":152919,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2001740,"kb_used_data":1240820,"kb_used_omap":12,"kb_used_meta":760883,"kb_avail":281113780,"statfs":{"total":289910292480,"available":287860510720,"internally_reserved":0,"allocated":1270599680,"data_stored":1269770132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":779144194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,4,11,5,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17682432,"num_objects":4317,"num_object_clones":0,"num_object_copies":8634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-165,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4317,"num_write_kb":17268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":675840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004090"},"pg_stats":[{"pgid":"2.f","version":"44'9552","reported_seq":11131,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:45.532176+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:45.532176+0000","last_peered":"2024-09-16T10:26:45.532176+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:45.532176+0000","last_undegraded":"2024-09-16T10:26:45.532176+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9552,"log_dups_size":0,"ondisk_log_size":9552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39124992,"num_objects":9552,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3436,"num_objects_unfound":0,"num_objects_dirty":9552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9552,"num_write_kb":38208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":3096576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6116},{"shards":"0,2","objects":3436}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9560","reported_seq":9717,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.981981+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:45.981981+0000","last_peered":"2024-09-16T10:26:45.981981+0000","last_clean":"2024-09-16T10:26:45.981981+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:45.981981+0000","last_undegraded":"2024-09-16T10:26:45.981981+0000","last_fullsized":"2024-09-16T10:26:45.981981+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9560,"log_dups_size":0,"ondisk_log_size":9560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39133184,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5773,"num_write_kb":23092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9591","reported_seq":9776,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.952884+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:45.952884+0000","last_peered":"2024-09-16T10:26:45.952884+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:45.952884+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9591,"log_dups_size":0,"ondisk_log_size":9591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39268352,"num_objects":9587,"num_object_clones":0,"num_object_copies":19174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":22592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9571","reported_seq":9627,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.808576+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:45.808576+0000","last_peered":"2024-09-16T10:26:45.808576+0000","last_clean":"2024-09-16T10:26:45.808576+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:45.808576+0000","last_undegraded":"2024-09-16T10:26:45.808576+0000","last_fullsized":"2024-09-16T10:26:45.808576+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9571,"log_dups_size":0,"ondisk_log_size":9571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39202816,"num_objects":9571,"num_object_clones":0,"num_object_copies":19142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":38284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9537","reported_seq":9693,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.454124+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:45.454124+0000","last_peered":"2024-09-16T10:26:45.454124+0000","last_clean":"2024-09-16T10:26:45.454124+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:45.454124+0000","last_undegraded":"2024-09-16T10:26:45.454124+0000","last_fullsized":"2024-09-16T10:26:45.454124+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":9530,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6179,"num_write_kb":24716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9454","reported_seq":9616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.395577+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:45.395577+0000","last_peered":"2024-09-16T10:26:45.395577+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:45.395577+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38703104,"num_objects":9449,"num_object_clones":0,"num_object_copies":18898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":24316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5232},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9762","reported_seq":9935,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.489897+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:45.489897+0000","last_peered":"2024-09-16T10:26:45.489897+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:45.489897+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39960576,"num_objects":9756,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6004,"num_write_kb":24016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5353},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9485","reported_seq":9660,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.481719+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:45.481719+0000","last_peered":"2024-09-16T10:26:45.481719+0000","last_clean":"2024-09-16T10:26:45.481719+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:45.481719+0000","last_undegraded":"2024-09-16T10:26:45.481719+0000","last_fullsized":"2024-09-16T10:26:45.481719+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9485,"log_dups_size":0,"ondisk_log_size":9485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842368,"num_objects":9483,"num_object_clones":0,"num_object_copies":18966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":21152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9464","reported_seq":17967,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:43.837864+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:43.837864+0000","last_peered":"2024-09-16T10:26:43.837864+0000","last_clean":"2024-09-16T10:26:43.837864+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:43.837864+0000","last_undegraded":"2024-09-16T10:26:43.837864+0000","last_fullsized":"2024-09-16T10:26:43.837864+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9464,"log_dups_size":0,"ondisk_log_size":9464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38736838,"num_objects":9459,"num_object_clones":0,"num_object_copies":18918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9462,"num_write_kb":37832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9765","reported_seq":9843,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.506693+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:45.506693+0000","last_peered":"2024-09-16T10:26:45.506693+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:45.506693+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965638,"num_objects":9759,"num_object_clones":0,"num_object_copies":19518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5325},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9502","reported_seq":9607,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.952754+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:45.952754+0000","last_peered":"2024-09-16T10:26:45.952754+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:45.952754+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900678,"num_objects":9499,"num_object_clones":0,"num_object_copies":18998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":37991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5222}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9535","reported_seq":9621,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.813724+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:45.813724+0000","last_peered":"2024-09-16T10:26:45.813724+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:45.813724+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9535,"log_dups_size":0,"ondisk_log_size":9535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39031750,"num_objects":9531,"num_object_clones":0,"num_object_copies":19062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9533,"num_write_kb":38119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9293","reported_seq":9372,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.458167+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:45.458167+0000","last_peered":"2024-09-16T10:26:45.458167+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:45.458167+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9293,"log_dups_size":0,"ondisk_log_size":9293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38056419,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":37165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5133},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9716","reported_seq":9849,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.452474+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:45.452474+0000","last_peered":"2024-09-16T10:26:45.452474+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:45.452474+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39765417,"num_objects":9711,"num_object_clones":0,"num_object_copies":19422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9713,"num_write_kb":38836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5374},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9382","reported_seq":9460,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.506830+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:45.506830+0000","last_peered":"2024-09-16T10:26:45.506830+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:45.506830+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38413254,"num_objects":9380,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":37514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5213},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9747","reported_seq":9839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.425245+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:45.425245+0000","last_peered":"2024-09-16T10:26:45.425245+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:45.425245+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39876492,"num_objects":9739,"num_object_clones":0,"num_object_copies":19478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":38947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5344},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":626016758,"num_objects":152852,"num_object_clones":0,"num_object_copies":305704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3436,"num_objects_unfound":0,"num_objects_dirty":152852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130481,"num_write_kb":521812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4974,"num_bytes_recovered":20292067,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1270554624,"data_stored":1270471525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152916,"ondisk_log_size":152916,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018931,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690908,"kb_used_data":442708,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93680932,"statfs":{"total":96636764160,"available":95929274368,"internally_reserved":0,"allocated":453332992,"data_stored":453079358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":254144176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.54800000000000004}]}]},{"osd":1,"up_from":40,"seq":171798691875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473588,"kb_used_data":271340,"kb_used_omap":4,"kb_used_meta":202235,"kb_avail":93898252,"statfs":{"total":96636764160,"available":96151810048,"internally_reserved":0,"allocated":277852160,"data_stored":277526266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":207089322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":12.569000000000001},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":8.6590000000000007}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":4.5670000000000002},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":3.258}]}]},{"osd":0,"up_from":24,"seq":103079215159,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":837244,"kb_used_data":526772,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93534596,"statfs":{"total":96636764160,"available":95779426304,"internally_reserved":0,"allocated":539414528,"data_stored":539164508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":317910696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,8,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":5.8040000000000003},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":5.2569999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":5.2270000000000003},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.867}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":539705344,"data_stored":539701731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277782528,"data_stored":277706655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":453066752,"data_stored":453063139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:48.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:48.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'9552", 'reported_seq': 11131, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:45.532176+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:45.532176+0000', 'last_peered': '2024-09-16T10:26:45.532176+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:45.532176+0000', 'last_undegraded': '2024-09-16T10:26:45.532176+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9552, 'log_dups_size': 0, 'ondisk_log_size': 9552, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39124992, 'num_objects': 9552, 'num_object_clones': 0, 'num_object_copies': 19104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3436, 'num_objects_unfound': 0, 'num_objects_dirty': 9552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9552, 'num_write_kb': 38208, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 756, 'num_bytes_recovered': 3096576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6116}, {'shards': '0,2', 'objects': 3436}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:48.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:48.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'9591", 'reported_seq': 9776, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:45.952884+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:45.952884+0000', 'last_peered': '2024-09-16T10:26:45.952884+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:45.952884+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 9591, 'log_dups_size': 0, 'ondisk_log_size': 9591, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39268352, 'num_objects': 9587, 'num_object_clones': 0, 'num_object_copies': 19174, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5648, 'num_write_kb': 22592, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 5252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:48.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:48.756 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'9454", 'reported_seq': 9616, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:45.395577+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:45.395577+0000', 'last_peered': '2024-09-16T10:26:45.395577+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:45.395577+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 9454, 'log_dups_size': 0, 'ondisk_log_size': 9454, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38703104, 'num_objects': 9449, 'num_object_clones': 0, 'num_object_copies': 18898, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6079, 'num_write_kb': 24316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5232}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:48.756 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:48.756 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'9762", 'reported_seq': 9935, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:45.489897+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:45.489897+0000', 'last_peered': '2024-09-16T10:26:45.489897+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:45.489897+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 9762, 'log_dups_size': 0, 'ondisk_log_size': 9762, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39960576, 'num_objects': 9756, 'num_object_clones': 0, 'num_object_copies': 19512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6004, 'num_write_kb': 24016, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5353}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:48.756 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:48.756 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'9765", 'reported_seq': 9843, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:45.506693+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:45.506693+0000', 'last_peered': '2024-09-16T10:26:45.506693+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:45.506693+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9765, 'log_dups_size': 0, 'ondisk_log_size': 9765, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39965638, 'num_objects': 9759, 'num_object_clones': 0, 'num_object_copies': 19518, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9763, 'num_write_kb': 39032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5325}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:48.756 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:48.756 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'9502", 'reported_seq': 9607, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:45.952754+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:45.952754+0000', 'last_peered': '2024-09-16T10:26:45.952754+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:45.952754+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 9502, 'log_dups_size': 0, 'ondisk_log_size': 9502, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38900678, 'num_objects': 9499, 'num_object_clones': 0, 'num_object_copies': 18998, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9500, 'num_write_kb': 37991, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 5222}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:48.756 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:48.756 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'9535", 'reported_seq': 9621, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:45.813724+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:45.813724+0000', 'last_peered': '2024-09-16T10:26:45.813724+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:45.813724+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9535, 'log_dups_size': 0, 'ondisk_log_size': 9535, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39031750, 'num_objects': 9531, 'num_object_clones': 0, 'num_object_copies': 19062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9533, 'num_write_kb': 38119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 5339}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:48.757 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:48.757 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'9293", 'reported_seq': 9372, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:45.458167+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:45.458167+0000', 'last_peered': '2024-09-16T10:26:45.458167+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:45.458167+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9293, 'log_dups_size': 0, 'ondisk_log_size': 9293, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38056419, 'num_objects': 9292, 'num_object_clones': 0, 'num_object_copies': 18584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9292, 'num_write_kb': 37165, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5133}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:48.757 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:48.757 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'9716", 'reported_seq': 9849, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:45.452474+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:26:45.452474+0000', 'last_peered': '2024-09-16T10:26:45.452474+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:26:45.452474+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 9716, 'log_dups_size': 0, 'ondisk_log_size': 9716, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39765417, 'num_objects': 9711, 'num_object_clones': 0, 'num_object_copies': 19422, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9711, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9713, 'num_write_kb': 38836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5374}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:48.757 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:48.757 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'9382", 'reported_seq': 9460, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:45.506830+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:26:45.506830+0000', 'last_peered': '2024-09-16T10:26:45.506830+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:26:45.506830+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9382, 'log_dups_size': 0, 'ondisk_log_size': 9382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38413254, 'num_objects': 9380, 'num_object_clones': 0, 'num_object_copies': 18760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9380, 'num_write_kb': 37514, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5213}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:48.757 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:48.757 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'9747", 'reported_seq': 9839, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:45.425245+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:45.425245+0000', 'last_peered': '2024-09-16T10:26:45.425245+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:45.425245+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9747, 'log_dups_size': 0, 'ondisk_log_size': 9747, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39876492, 'num_objects': 9739, 'num_object_clones': 0, 'num_object_copies': 19478, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9739, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9744, 'num_write_kb': 38947, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5344}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:48.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:49.422 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:49.422 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/305708 objects degraded (14.045%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":298,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":152854,"data_bytes":626016777,"bytes_used":2049781760,"bytes_avail":287860510720,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":305708,"degraded_ratio":0.14044774752378086,"misplaced_objects":3436,"misplaced_total":305708,"misplaced_ratio":0.011239483428631243,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":56300,"recovering_keys_per_sec":0,"num_objects_recovered":165,"num_bytes_recovered":675840,"num_keys_recovered":0,"write_bytes_sec":1473033,"read_op_per_sec":0,"write_op_per_sec":359},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:49.445 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:49.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:49.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 217 19 154563 154544 2.78134 1.55859 0.0363295 0.0280799 2024-09-16T10:26:50.004 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:50.006 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:50.029 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-16T10:26:48.631586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":626016777,"num_objects":152854,"num_object_clones":0,"num_object_copies":305708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3436,"num_objects_unfound":0,"num_objects_dirty":152854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130483,"num_write_kb":521814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20292086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152919,"ondisk_log_size":152919,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2001740,"kb_used_data":1240820,"kb_used_omap":12,"kb_used_meta":760883,"kb_avail":281113780,"statfs":{"total":289910292480,"available":287860510720,"internally_reserved":0,"allocated":1270599680,"data_stored":1269770132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":779144194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,4,11,5,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17682432,"num_objects":4317,"num_object_clones":0,"num_object_copies":8634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-165,"num_objects_unfound":0,"num_objects_dirty":4317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4317,"num_write_kb":17268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":675840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004090"},"pg_stats":[{"pgid":"2.f","version":"44'9552","reported_seq":11131,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:45.532176+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:45.532176+0000","last_peered":"2024-09-16T10:26:45.532176+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:45.532176+0000","last_undegraded":"2024-09-16T10:26:45.532176+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9552,"log_dups_size":0,"ondisk_log_size":9552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39124992,"num_objects":9552,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3436,"num_objects_unfound":0,"num_objects_dirty":9552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9552,"num_write_kb":38208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":3096576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6116},{"shards":"0,2","objects":3436}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9560","reported_seq":9717,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.981981+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:45.981981+0000","last_peered":"2024-09-16T10:26:45.981981+0000","last_clean":"2024-09-16T10:26:45.981981+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:45.981981+0000","last_undegraded":"2024-09-16T10:26:45.981981+0000","last_fullsized":"2024-09-16T10:26:45.981981+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9560,"log_dups_size":0,"ondisk_log_size":9560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39133184,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5773,"num_write_kb":23092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9591","reported_seq":9776,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.952884+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:45.952884+0000","last_peered":"2024-09-16T10:26:45.952884+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:45.952884+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9591,"log_dups_size":0,"ondisk_log_size":9591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39268352,"num_objects":9587,"num_object_clones":0,"num_object_copies":19174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5648,"num_write_kb":22592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9571","reported_seq":9627,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.808576+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:45.808576+0000","last_peered":"2024-09-16T10:26:45.808576+0000","last_clean":"2024-09-16T10:26:45.808576+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:45.808576+0000","last_undegraded":"2024-09-16T10:26:45.808576+0000","last_fullsized":"2024-09-16T10:26:45.808576+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9571,"log_dups_size":0,"ondisk_log_size":9571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39202816,"num_objects":9571,"num_object_clones":0,"num_object_copies":19142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":38284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9537","reported_seq":9693,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.454124+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:45.454124+0000","last_peered":"2024-09-16T10:26:45.454124+0000","last_clean":"2024-09-16T10:26:45.454124+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:45.454124+0000","last_undegraded":"2024-09-16T10:26:45.454124+0000","last_fullsized":"2024-09-16T10:26:45.454124+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":9530,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6179,"num_write_kb":24716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9454","reported_seq":9616,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.395577+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:45.395577+0000","last_peered":"2024-09-16T10:26:45.395577+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:45.395577+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38703104,"num_objects":9449,"num_object_clones":0,"num_object_copies":18898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6079,"num_write_kb":24316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5232},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9762","reported_seq":9935,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.489897+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:45.489897+0000","last_peered":"2024-09-16T10:26:45.489897+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:45.489897+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9762,"log_dups_size":0,"ondisk_log_size":9762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39960576,"num_objects":9756,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6004,"num_write_kb":24016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5353},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9485","reported_seq":9660,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.481719+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:45.481719+0000","last_peered":"2024-09-16T10:26:45.481719+0000","last_clean":"2024-09-16T10:26:45.481719+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:45.481719+0000","last_undegraded":"2024-09-16T10:26:45.481719+0000","last_fullsized":"2024-09-16T10:26:45.481719+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9485,"log_dups_size":0,"ondisk_log_size":9485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38842368,"num_objects":9483,"num_object_clones":0,"num_object_copies":18966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":21152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9464","reported_seq":17967,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:43.837864+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:43.837864+0000","last_peered":"2024-09-16T10:26:43.837864+0000","last_clean":"2024-09-16T10:26:43.837864+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:43.837864+0000","last_undegraded":"2024-09-16T10:26:43.837864+0000","last_fullsized":"2024-09-16T10:26:43.837864+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9464,"log_dups_size":0,"ondisk_log_size":9464,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38736838,"num_objects":9459,"num_object_clones":0,"num_object_copies":18918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9462,"num_write_kb":37832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9765","reported_seq":9843,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.506693+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:45.506693+0000","last_peered":"2024-09-16T10:26:45.506693+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:45.506693+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9765,"log_dups_size":0,"ondisk_log_size":9765,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965638,"num_objects":9759,"num_object_clones":0,"num_object_copies":19518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9763,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5325},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9502","reported_seq":9607,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.952754+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:45.952754+0000","last_peered":"2024-09-16T10:26:45.952754+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:45.952754+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9502,"log_dups_size":0,"ondisk_log_size":9502,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900678,"num_objects":9499,"num_object_clones":0,"num_object_copies":18998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9500,"num_write_kb":37991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5222}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9535","reported_seq":9621,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.813724+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:45.813724+0000","last_peered":"2024-09-16T10:26:45.813724+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:45.813724+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9535,"log_dups_size":0,"ondisk_log_size":9535,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39031750,"num_objects":9531,"num_object_clones":0,"num_object_copies":19062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9533,"num_write_kb":38119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5339}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9293","reported_seq":9372,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.458167+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:45.458167+0000","last_peered":"2024-09-16T10:26:45.458167+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:45.458167+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9293,"log_dups_size":0,"ondisk_log_size":9293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38056419,"num_objects":9292,"num_object_clones":0,"num_object_copies":18584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":37165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5133},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9716","reported_seq":9849,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.452474+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:45.452474+0000","last_peered":"2024-09-16T10:26:45.452474+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:45.452474+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9716,"log_dups_size":0,"ondisk_log_size":9716,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39765417,"num_objects":9711,"num_object_clones":0,"num_object_copies":19422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9713,"num_write_kb":38836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5374},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9382","reported_seq":9460,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.506830+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:45.506830+0000","last_peered":"2024-09-16T10:26:45.506830+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:45.506830+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9382,"log_dups_size":0,"ondisk_log_size":9382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38413254,"num_objects":9380,"num_object_clones":0,"num_object_copies":18760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9380,"num_write_kb":37514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5213},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9747","reported_seq":9839,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:45.425245+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:45.425245+0000","last_peered":"2024-09-16T10:26:45.425245+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:45.425245+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39876492,"num_objects":9739,"num_object_clones":0,"num_object_copies":19478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":38947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5344},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":626016758,"num_objects":152852,"num_object_clones":0,"num_object_copies":305704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3436,"num_objects_unfound":0,"num_objects_dirty":152852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":130481,"num_write_kb":521812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4974,"num_bytes_recovered":20292067,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1270554624,"data_stored":1270471525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152916,"ondisk_log_size":152916,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018931,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690908,"kb_used_data":442708,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93680932,"statfs":{"total":96636764160,"available":95929274368,"internally_reserved":0,"allocated":453332992,"data_stored":453079358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":254144176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.54800000000000004}]}]},{"osd":1,"up_from":40,"seq":171798691875,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473588,"kb_used_data":271340,"kb_used_omap":4,"kb_used_meta":202235,"kb_avail":93898252,"statfs":{"total":96636764160,"available":96151810048,"internally_reserved":0,"allocated":277852160,"data_stored":277526266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":207089322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":12.569000000000001},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":8.6590000000000007}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":4.5670000000000002},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":3.258}]}]},{"osd":0,"up_from":24,"seq":103079215159,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":837244,"kb_used_data":526772,"kb_used_omap":4,"kb_used_meta":310459,"kb_avail":93534596,"statfs":{"total":96636764160,"available":95779426304,"internally_reserved":0,"allocated":539414528,"data_stored":539164508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":317910696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,8,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":5.8040000000000003},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":5.2569999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":5.2270000000000003},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":5.867}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":539705344,"data_stored":539701731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":277782528,"data_stored":277706655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":453066752,"data_stored":453063139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:50.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 218 20 154998 154978 2.77636 1.69531 0.0250575 0.0281311 2024-09-16T10:26:51.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 219 20 155453 155433 2.7718 1.77734 0.0242071 0.028179 2024-09-16T10:26:52.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:2024-09-16T10:26:52.563562+0000 min lat: 0.00139595 max lat: 2.61294 avg lat: 0.0282241 2024-09-16T10:26:52.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:26:52.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 220 20 155917 155897 2.76744 1.8125 0.0939278 0.0282241 2024-09-16T10:26:53.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:53.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 221 20 156339 156319 2.76237 1.64844 0.0347742 0.0282731 2024-09-16T10:26:53.596 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:53.598 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:53.623 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-16T10:26:52.632849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634929673,"num_objects":155030,"num_object_clones":0,"num_object_copies":310060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3359,"num_objects_unfound":0,"num_objects_dirty":155030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132659,"num_write_kb":530518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5052,"num_bytes_recovered":20607478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155095,"ondisk_log_size":155095,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2050356,"kb_used_data":1257436,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281065164,"statfs":{"total":289910292480,"available":287810727936,"internally_reserved":0,"allocated":1287614464,"data_stored":1286784916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":811912194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,4,10,13,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20484096,"num_objects":5001,"num_object_clones":0,"num_object_copies":10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-151,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":20004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":618496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003919"},"pg_stats":[{"pgid":"2.f","version":"44'9663","reported_seq":11396,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:50.599224+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:50.599224+0000","last_peered":"2024-09-16T10:26:50.599224+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:50.599224+0000","last_undegraded":"2024-09-16T10:26:50.599224+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9663,"log_dups_size":0,"ondisk_log_size":9663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39579648,"num_objects":9663,"num_object_clones":0,"num_object_copies":19326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3359,"num_objects_unfound":0,"num_objects_dirty":9663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9663,"num_write_kb":38652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":3411968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6304},{"shards":"0,2","objects":3359}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9685","reported_seq":9842,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.923651+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:50.923651+0000","last_peered":"2024-09-16T10:26:50.923651+0000","last_clean":"2024-09-16T10:26:50.923651+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:50.923651+0000","last_undegraded":"2024-09-16T10:26:50.923651+0000","last_fullsized":"2024-09-16T10:26:50.923651+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39645184,"num_objects":9679,"num_object_clones":0,"num_object_copies":19358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5898,"num_write_kb":23592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9754","reported_seq":9939,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.983790+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:50.983790+0000","last_peered":"2024-09-16T10:26:50.983790+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:50.983790+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39936000,"num_objects":9750,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5811,"num_write_kb":23244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9703","reported_seq":9758,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.866553+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:50.866553+0000","last_peered":"2024-09-16T10:26:50.866553+0000","last_clean":"2024-09-16T10:26:50.866553+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:50.866553+0000","last_undegraded":"2024-09-16T10:26:50.866553+0000","last_fullsized":"2024-09-16T10:26:50.866553+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39743488,"num_objects":9703,"num_object_clones":0,"num_object_copies":19406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9703,"num_write_kb":38812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9664","reported_seq":9820,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.506943+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:50.506943+0000","last_peered":"2024-09-16T10:26:50.506943+0000","last_clean":"2024-09-16T10:26:50.506943+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:50.506943+0000","last_undegraded":"2024-09-16T10:26:50.506943+0000","last_fullsized":"2024-09-16T10:26:50.506943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39555072,"num_objects":9657,"num_object_clones":0,"num_object_copies":19314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6306,"num_write_kb":25224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9584","reported_seq":9747,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.506911+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:50.506911+0000","last_peered":"2024-09-16T10:26:50.506911+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:50.506911+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39235584,"num_objects":9579,"num_object_clones":0,"num_object_copies":19158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6209,"num_write_kb":24836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5362},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9890","reported_seq":10063,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.532964+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:50.532964+0000","last_peered":"2024-09-16T10:26:50.532964+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:50.532964+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9890,"log_dups_size":0,"ondisk_log_size":9890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40484864,"num_objects":9884,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6132,"num_write_kb":24528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5481},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9621","reported_seq":9796,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.573368+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:50.573368+0000","last_peered":"2024-09-16T10:26:50.573368+0000","last_clean":"2024-09-16T10:26:50.573368+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:50.573368+0000","last_undegraded":"2024-09-16T10:26:50.573368+0000","last_fullsized":"2024-09-16T10:26:50.573368+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9621,"log_dups_size":0,"ondisk_log_size":9621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39399424,"num_objects":9619,"num_object_clones":0,"num_object_copies":19238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9602","reported_seq":18105,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:48.924113+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:48.924113+0000","last_peered":"2024-09-16T10:26:48.924113+0000","last_clean":"2024-09-16T10:26:48.924113+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:48.924113+0000","last_undegraded":"2024-09-16T10:26:48.924113+0000","last_fullsized":"2024-09-16T10:26:48.924113+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39302086,"num_objects":9597,"num_object_clones":0,"num_object_copies":19194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9600,"num_write_kb":38384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9897","reported_seq":9976,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.581466+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:50.581466+0000","last_peered":"2024-09-16T10:26:50.581466+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:50.581466+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40506310,"num_objects":9891,"num_object_clones":0,"num_object_copies":19782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":39560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5457},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9652","reported_seq":9757,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:51.018991+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:51.018991+0000","last_peered":"2024-09-16T10:26:51.018991+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:51.018991+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39515078,"num_objects":9649,"num_object_clones":0,"num_object_copies":19298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":38591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9692","reported_seq":9778,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.993582+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:50.993582+0000","last_peered":"2024-09-16T10:26:50.993582+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:50.993582+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39674822,"num_objects":9688,"num_object_clones":0,"num_object_copies":19376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":38747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9443","reported_seq":9522,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.470891+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:50.470891+0000","last_peered":"2024-09-16T10:26:50.470891+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:50.470891+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9443,"log_dups_size":0,"ondisk_log_size":9443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38670819,"num_objects":9442,"num_object_clones":0,"num_object_copies":18884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9442,"num_write_kb":37765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5283},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9865","reported_seq":9997,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.566282+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:50.566282+0000","last_peered":"2024-09-16T10:26:50.566282+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:50.566282+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40375721,"num_objects":9860,"num_object_clones":0,"num_object_copies":19720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":39432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5523},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9501","reported_seq":9581,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.563334+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:50.563334+0000","last_peered":"2024-09-16T10:26:50.563334+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:50.563334+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9501,"log_dups_size":0,"ondisk_log_size":9501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900678,"num_objects":9499,"num_object_clones":0,"num_object_copies":18998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9499,"num_write_kb":37990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5332},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9876","reported_seq":9967,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.566349+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:50.566349+0000","last_peered":"2024-09-16T10:26:50.566349+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:50.566349+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9876,"log_dups_size":0,"ondisk_log_size":9876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40404876,"num_objects":9868,"num_object_clones":0,"num_object_copies":19736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9873,"num_write_kb":39463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5473},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":634929654,"num_objects":155028,"num_object_clones":0,"num_object_copies":310056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3359,"num_objects_unfound":0,"num_objects_dirty":155028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132657,"num_write_kb":530516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5051,"num_bytes_recovered":20607459,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1288646656,"data_stored":1288563557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155092,"ondisk_log_size":155092,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018932,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698672,"kb_used_data":446376,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93673168,"statfs":{"total":96636764160,"available":95921324032,"internally_reserved":0,"allocated":457089024,"data_stored":456835390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":6.274},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":7.4560000000000004}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":13.807}]}]},{"osd":1,"up_from":40,"seq":171798691876,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486104,"kb_used_data":278096,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93885736,"statfs":{"total":96636764160,"available":96138993664,"internally_reserved":0,"allocated":284770304,"data_stored":284444410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":212987562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":1.2809999999999999},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.7519999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.93700000000000006},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":0.78000000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215160,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":865580,"kb_used_data":532964,"kb_used_omap":4,"kb_used_meta":332603,"kb_avail":93506260,"statfs":{"total":96636764160,"available":95750410240,"internally_reserved":0,"allocated":545755136,"data_stored":545505116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":340586152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,3,7,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":0.184},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":4.9489999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":1.2270000000000001},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":546091008,"data_stored":546087395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":284909568,"data_stored":284833695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":457646080,"data_stored":457642467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:53.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:54.179 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:54.180 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:54.206 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-16T10:26:52.632849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634929673,"num_objects":155030,"num_object_clones":0,"num_object_copies":310060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3359,"num_objects_unfound":0,"num_objects_dirty":155030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132659,"num_write_kb":530518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5052,"num_bytes_recovered":20607478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155095,"ondisk_log_size":155095,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2050356,"kb_used_data":1257436,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281065164,"statfs":{"total":289910292480,"available":287810727936,"internally_reserved":0,"allocated":1287614464,"data_stored":1286784916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":811912194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,4,10,13,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20484096,"num_objects":5001,"num_object_clones":0,"num_object_copies":10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-151,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5001,"num_write_kb":20004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":618496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003919"},"pg_stats":[{"pgid":"2.f","version":"44'9663","reported_seq":11396,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:50.599224+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:50.599224+0000","last_peered":"2024-09-16T10:26:50.599224+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:50.599224+0000","last_undegraded":"2024-09-16T10:26:50.599224+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9663,"log_dups_size":0,"ondisk_log_size":9663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39579648,"num_objects":9663,"num_object_clones":0,"num_object_copies":19326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3359,"num_objects_unfound":0,"num_objects_dirty":9663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9663,"num_write_kb":38652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":3411968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6304},{"shards":"0,2","objects":3359}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9685","reported_seq":9842,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.923651+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:50.923651+0000","last_peered":"2024-09-16T10:26:50.923651+0000","last_clean":"2024-09-16T10:26:50.923651+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:50.923651+0000","last_undegraded":"2024-09-16T10:26:50.923651+0000","last_fullsized":"2024-09-16T10:26:50.923651+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39645184,"num_objects":9679,"num_object_clones":0,"num_object_copies":19358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5898,"num_write_kb":23592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9754","reported_seq":9939,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.983790+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:50.983790+0000","last_peered":"2024-09-16T10:26:50.983790+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:50.983790+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39936000,"num_objects":9750,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5811,"num_write_kb":23244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9703","reported_seq":9758,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.866553+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:50.866553+0000","last_peered":"2024-09-16T10:26:50.866553+0000","last_clean":"2024-09-16T10:26:50.866553+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:50.866553+0000","last_undegraded":"2024-09-16T10:26:50.866553+0000","last_fullsized":"2024-09-16T10:26:50.866553+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39743488,"num_objects":9703,"num_object_clones":0,"num_object_copies":19406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9703,"num_write_kb":38812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9664","reported_seq":9820,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.506943+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:50.506943+0000","last_peered":"2024-09-16T10:26:50.506943+0000","last_clean":"2024-09-16T10:26:50.506943+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:50.506943+0000","last_undegraded":"2024-09-16T10:26:50.506943+0000","last_fullsized":"2024-09-16T10:26:50.506943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39555072,"num_objects":9657,"num_object_clones":0,"num_object_copies":19314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6306,"num_write_kb":25224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9584","reported_seq":9747,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.506911+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:50.506911+0000","last_peered":"2024-09-16T10:26:50.506911+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:50.506911+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39235584,"num_objects":9579,"num_object_clones":0,"num_object_copies":19158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6209,"num_write_kb":24836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5362},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9890","reported_seq":10063,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.532964+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:50.532964+0000","last_peered":"2024-09-16T10:26:50.532964+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:50.532964+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9890,"log_dups_size":0,"ondisk_log_size":9890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40484864,"num_objects":9884,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6132,"num_write_kb":24528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5481},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9621","reported_seq":9796,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.573368+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:50.573368+0000","last_peered":"2024-09-16T10:26:50.573368+0000","last_clean":"2024-09-16T10:26:50.573368+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:50.573368+0000","last_undegraded":"2024-09-16T10:26:50.573368+0000","last_fullsized":"2024-09-16T10:26:50.573368+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9621,"log_dups_size":0,"ondisk_log_size":9621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39399424,"num_objects":9619,"num_object_clones":0,"num_object_copies":19238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9602","reported_seq":18105,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:48.924113+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:48.924113+0000","last_peered":"2024-09-16T10:26:48.924113+0000","last_clean":"2024-09-16T10:26:48.924113+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:48.924113+0000","last_undegraded":"2024-09-16T10:26:48.924113+0000","last_fullsized":"2024-09-16T10:26:48.924113+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39302086,"num_objects":9597,"num_object_clones":0,"num_object_copies":19194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9600,"num_write_kb":38384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9897","reported_seq":9976,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.581466+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:50.581466+0000","last_peered":"2024-09-16T10:26:50.581466+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:50.581466+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40506310,"num_objects":9891,"num_object_clones":0,"num_object_copies":19782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":39560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5457},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9652","reported_seq":9757,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:51.018991+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:51.018991+0000","last_peered":"2024-09-16T10:26:51.018991+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:51.018991+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39515078,"num_objects":9649,"num_object_clones":0,"num_object_copies":19298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":38591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9692","reported_seq":9778,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.993582+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:50.993582+0000","last_peered":"2024-09-16T10:26:50.993582+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:50.993582+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39674822,"num_objects":9688,"num_object_clones":0,"num_object_copies":19376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":38747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9443","reported_seq":9522,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.470891+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:50.470891+0000","last_peered":"2024-09-16T10:26:50.470891+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:50.470891+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9443,"log_dups_size":0,"ondisk_log_size":9443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38670819,"num_objects":9442,"num_object_clones":0,"num_object_copies":18884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9442,"num_write_kb":37765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5283},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9865","reported_seq":9997,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.566282+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:50.566282+0000","last_peered":"2024-09-16T10:26:50.566282+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:50.566282+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40375721,"num_objects":9860,"num_object_clones":0,"num_object_copies":19720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":39432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5523},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9501","reported_seq":9581,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.563334+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:50.563334+0000","last_peered":"2024-09-16T10:26:50.563334+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:50.563334+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9501,"log_dups_size":0,"ondisk_log_size":9501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900678,"num_objects":9499,"num_object_clones":0,"num_object_copies":18998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9499,"num_write_kb":37990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5332},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9876","reported_seq":9967,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.566349+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:50.566349+0000","last_peered":"2024-09-16T10:26:50.566349+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:50.566349+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9876,"log_dups_size":0,"ondisk_log_size":9876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40404876,"num_objects":9868,"num_object_clones":0,"num_object_copies":19736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9873,"num_write_kb":39463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5473},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":634929654,"num_objects":155028,"num_object_clones":0,"num_object_copies":310056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3359,"num_objects_unfound":0,"num_objects_dirty":155028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132657,"num_write_kb":530516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5051,"num_bytes_recovered":20607459,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1288646656,"data_stored":1288563557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155092,"ondisk_log_size":155092,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018932,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698672,"kb_used_data":446376,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93673168,"statfs":{"total":96636764160,"available":95921324032,"internally_reserved":0,"allocated":457089024,"data_stored":456835390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":6.274},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":7.4560000000000004}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":13.807}]}]},{"osd":1,"up_from":40,"seq":171798691876,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":486104,"kb_used_data":278096,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93885736,"statfs":{"total":96636764160,"available":96138993664,"internally_reserved":0,"allocated":284770304,"data_stored":284444410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":212987562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,7,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":1.2809999999999999},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.7519999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.93700000000000006},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":0.78000000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215160,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":865580,"kb_used_data":532964,"kb_used_omap":4,"kb_used_meta":332603,"kb_avail":93506260,"statfs":{"total":96636764160,"available":95750410240,"internally_reserved":0,"allocated":545755136,"data_stored":545505116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":340586152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,3,7,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":0.184},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":4.9489999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":1.2270000000000001},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":546091008,"data_stored":546087395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":284909568,"data_stored":284833695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":457646080,"data_stored":457642467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:54.208 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:54.208 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'9663", 'reported_seq': 11396, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:50.599224+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:50.599224+0000', 'last_peered': '2024-09-16T10:26:50.599224+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:50.599224+0000', 'last_undegraded': '2024-09-16T10:26:50.599224+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9663, 'log_dups_size': 0, 'ondisk_log_size': 9663, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39579648, 'num_objects': 9663, 'num_object_clones': 0, 'num_object_copies': 19326, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3359, 'num_objects_unfound': 0, 'num_objects_dirty': 9663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9663, 'num_write_kb': 38652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 833, 'num_bytes_recovered': 3411968, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6304}, {'shards': '0,2', 'objects': 3359}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:54.208 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:54.208 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'9754", 'reported_seq': 9939, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:50.983790+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:50.983790+0000', 'last_peered': '2024-09-16T10:26:50.983790+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:50.983790+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 9754, 'log_dups_size': 0, 'ondisk_log_size': 9754, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39936000, 'num_objects': 9750, 'num_object_clones': 0, 'num_object_copies': 19500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5811, 'num_write_kb': 23244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 5415}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:54.208 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:54.208 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'9584", 'reported_seq': 9747, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:50.506911+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:50.506911+0000', 'last_peered': '2024-09-16T10:26:50.506911+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:50.506911+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 9584, 'log_dups_size': 0, 'ondisk_log_size': 9584, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39235584, 'num_objects': 9579, 'num_object_clones': 0, 'num_object_copies': 19158, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6209, 'num_write_kb': 24836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5362}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:54.208 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:54.208 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'9890", 'reported_seq': 10063, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:50.532964+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:50.532964+0000', 'last_peered': '2024-09-16T10:26:50.532964+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:50.532964+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 9890, 'log_dups_size': 0, 'ondisk_log_size': 9890, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40484864, 'num_objects': 9884, 'num_object_clones': 0, 'num_object_copies': 19768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6132, 'num_write_kb': 24528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5481}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'9897", 'reported_seq': 9976, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:50.581466+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:50.581466+0000', 'last_peered': '2024-09-16T10:26:50.581466+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:50.581466+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9897, 'log_dups_size': 0, 'ondisk_log_size': 9897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40506310, 'num_objects': 9891, 'num_object_clones': 0, 'num_object_copies': 19782, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9895, 'num_write_kb': 39560, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5457}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'9652", 'reported_seq': 9757, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:51.018991+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:51.018991+0000', 'last_peered': '2024-09-16T10:26:51.018991+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:51.018991+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 9652, 'log_dups_size': 0, 'ondisk_log_size': 9652, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39515078, 'num_objects': 9649, 'num_object_clones': 0, 'num_object_copies': 19298, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9650, 'num_write_kb': 38591, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 5372}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'9692", 'reported_seq': 9778, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:50.993582+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:50.993582+0000', 'last_peered': '2024-09-16T10:26:50.993582+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:50.993582+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9692, 'log_dups_size': 0, 'ondisk_log_size': 9692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39674822, 'num_objects': 9688, 'num_object_clones': 0, 'num_object_copies': 19376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9690, 'num_write_kb': 38747, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 5496}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'9443", 'reported_seq': 9522, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:50.470891+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:50.470891+0000', 'last_peered': '2024-09-16T10:26:50.470891+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:50.470891+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9443, 'log_dups_size': 0, 'ondisk_log_size': 9443, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38670819, 'num_objects': 9442, 'num_object_clones': 0, 'num_object_copies': 18884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9442, 'num_write_kb': 37765, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5283}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'9865", 'reported_seq': 9997, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:50.566282+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:26:50.566282+0000', 'last_peered': '2024-09-16T10:26:50.566282+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:26:50.566282+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 9865, 'log_dups_size': 0, 'ondisk_log_size': 9865, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40375721, 'num_objects': 9860, 'num_object_clones': 0, 'num_object_copies': 19720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9862, 'num_write_kb': 39432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5523}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:54.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:54.210 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'9501", 'reported_seq': 9581, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:50.563334+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:26:50.563334+0000', 'last_peered': '2024-09-16T10:26:50.563334+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:26:50.563334+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9501, 'log_dups_size': 0, 'ondisk_log_size': 9501, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38900678, 'num_objects': 9499, 'num_object_clones': 0, 'num_object_copies': 18998, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9499, 'num_write_kb': 37990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5332}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:54.210 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:54.210 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'9876", 'reported_seq': 9967, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:50.566349+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:50.566349+0000', 'last_peered': '2024-09-16T10:26:50.566349+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:50.566349+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9876, 'log_dups_size': 0, 'ondisk_log_size': 9876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40404876, 'num_objects': 9868, 'num_object_clones': 0, 'num_object_copies': 19736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9873, 'num_write_kb': 39463, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5473}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:54.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:26:54.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 222 20 156746 156726 2.75709 1.58984 0.0284877 0.0283285 2024-09-16T10:26:54.917 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:54.918 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/310348 objects degraded (13.835%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":303,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":155174,"data_bytes":635519497,"bytes_used":2115186688,"bytes_avail":287795105792,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":310348,"degraded_ratio":0.1383479191101602,"misplaced_objects":3359,"misplaced_total":310348,"misplaced_ratio":0.010823333805921093,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":51523,"recovering_keys_per_sec":0,"num_objects_recovered":151,"num_bytes_recovered":618496,"num_keys_recovered":0,"write_bytes_sec":1537859,"read_op_per_sec":0,"write_op_per_sec":375},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:26:54.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:26:54.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:55.518 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:55.520 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:55.553 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-16T10:26:54.633810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635519497,"num_objects":155174,"num_object_clones":0,"num_object_copies":310348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3359,"num_objects_unfound":0,"num_objects_dirty":155174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132803,"num_write_kb":531094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5052,"num_bytes_recovered":20607478,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155239,"ondisk_log_size":155239,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2065612,"kb_used_data":1264500,"kb_used_omap":12,"kb_used_meta":801075,"kb_avail":281049908,"statfs":{"total":289910292480,"available":287795105792,"internally_reserved":0,"allocated":1294848000,"data_stored":1294018452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":820300802},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,3,2,9,12,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18460672,"num_objects":4507,"num_object_clones":0,"num_object_copies":9014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-151,"num_objects_unfound":0,"num_objects_dirty":4507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4507,"num_write_kb":18028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":618496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004131"},"pg_stats":[{"pgid":"2.f","version":"44'9663","reported_seq":11396,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:50.599224+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:50.599224+0000","last_peered":"2024-09-16T10:26:50.599224+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:50.599224+0000","last_undegraded":"2024-09-16T10:26:50.599224+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9663,"log_dups_size":0,"ondisk_log_size":9663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39579648,"num_objects":9663,"num_object_clones":0,"num_object_copies":19326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3359,"num_objects_unfound":0,"num_objects_dirty":9663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9663,"num_write_kb":38652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":833,"num_bytes_recovered":3411968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6304},{"shards":"0,2","objects":3359}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9685","reported_seq":9842,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.923651+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:50.923651+0000","last_peered":"2024-09-16T10:26:50.923651+0000","last_clean":"2024-09-16T10:26:50.923651+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:50.923651+0000","last_undegraded":"2024-09-16T10:26:50.923651+0000","last_fullsized":"2024-09-16T10:26:50.923651+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9685,"log_dups_size":0,"ondisk_log_size":9685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39645184,"num_objects":9679,"num_object_clones":0,"num_object_copies":19358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5898,"num_write_kb":23592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9754","reported_seq":9939,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.983790+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:50.983790+0000","last_peered":"2024-09-16T10:26:50.983790+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:50.983790+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9754,"log_dups_size":0,"ondisk_log_size":9754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39936000,"num_objects":9750,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5811,"num_write_kb":23244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5415}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9703","reported_seq":9758,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.866553+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:50.866553+0000","last_peered":"2024-09-16T10:26:50.866553+0000","last_clean":"2024-09-16T10:26:50.866553+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:50.866553+0000","last_undegraded":"2024-09-16T10:26:50.866553+0000","last_fullsized":"2024-09-16T10:26:50.866553+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9703,"log_dups_size":0,"ondisk_log_size":9703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39743488,"num_objects":9703,"num_object_clones":0,"num_object_copies":19406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9703,"num_write_kb":38812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9664","reported_seq":9820,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.506943+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:50.506943+0000","last_peered":"2024-09-16T10:26:50.506943+0000","last_clean":"2024-09-16T10:26:50.506943+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:50.506943+0000","last_undegraded":"2024-09-16T10:26:50.506943+0000","last_fullsized":"2024-09-16T10:26:50.506943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9664,"log_dups_size":0,"ondisk_log_size":9664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39555072,"num_objects":9657,"num_object_clones":0,"num_object_copies":19314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6306,"num_write_kb":25224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9584","reported_seq":9747,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.506911+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:50.506911+0000","last_peered":"2024-09-16T10:26:50.506911+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:50.506911+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9584,"log_dups_size":0,"ondisk_log_size":9584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39235584,"num_objects":9579,"num_object_clones":0,"num_object_copies":19158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6209,"num_write_kb":24836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5362},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'9890","reported_seq":10063,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.532964+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:50.532964+0000","last_peered":"2024-09-16T10:26:50.532964+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:50.532964+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":9890,"log_dups_size":0,"ondisk_log_size":9890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40484864,"num_objects":9884,"num_object_clones":0,"num_object_copies":19768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6132,"num_write_kb":24528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5481},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9621","reported_seq":9796,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:50.573368+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:50.573368+0000","last_peered":"2024-09-16T10:26:50.573368+0000","last_clean":"2024-09-16T10:26:50.573368+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:50.573368+0000","last_undegraded":"2024-09-16T10:26:50.573368+0000","last_fullsized":"2024-09-16T10:26:50.573368+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9621,"log_dups_size":0,"ondisk_log_size":9621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39399424,"num_objects":9619,"num_object_clones":0,"num_object_copies":19238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9746","reported_seq":18249,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:53.942756+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:53.942756+0000","last_peered":"2024-09-16T10:26:53.942756+0000","last_clean":"2024-09-16T10:26:53.942756+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:53.942756+0000","last_undegraded":"2024-09-16T10:26:53.942756+0000","last_fullsized":"2024-09-16T10:26:53.942756+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39891910,"num_objects":9741,"num_object_clones":0,"num_object_copies":19482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":38960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'9897","reported_seq":9976,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.581466+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:50.581466+0000","last_peered":"2024-09-16T10:26:50.581466+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:50.581466+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40506310,"num_objects":9891,"num_object_clones":0,"num_object_copies":19782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9895,"num_write_kb":39560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5457},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9652","reported_seq":9757,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:51.018991+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:51.018991+0000","last_peered":"2024-09-16T10:26:51.018991+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:51.018991+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39515078,"num_objects":9649,"num_object_clones":0,"num_object_copies":19298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9650,"num_write_kb":38591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5372}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9692","reported_seq":9778,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.993582+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:50.993582+0000","last_peered":"2024-09-16T10:26:50.993582+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:50.993582+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9692,"log_dups_size":0,"ondisk_log_size":9692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39674822,"num_objects":9688,"num_object_clones":0,"num_object_copies":19376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":38747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9443","reported_seq":9522,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.470891+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:50.470891+0000","last_peered":"2024-09-16T10:26:50.470891+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:50.470891+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9443,"log_dups_size":0,"ondisk_log_size":9443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38670819,"num_objects":9442,"num_object_clones":0,"num_object_copies":18884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9442,"num_write_kb":37765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5283},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'9865","reported_seq":9997,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.566282+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:50.566282+0000","last_peered":"2024-09-16T10:26:50.566282+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:50.566282+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":9865,"log_dups_size":0,"ondisk_log_size":9865,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40375721,"num_objects":9860,"num_object_clones":0,"num_object_copies":19720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":39432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5523},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9501","reported_seq":9581,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.563334+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:50.563334+0000","last_peered":"2024-09-16T10:26:50.563334+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:50.563334+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9501,"log_dups_size":0,"ondisk_log_size":9501,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38900678,"num_objects":9499,"num_object_clones":0,"num_object_copies":18998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9499,"num_write_kb":37990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5332},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'9876","reported_seq":9967,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:50.566349+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:50.566349+0000","last_peered":"2024-09-16T10:26:50.566349+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:50.566349+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9876,"log_dups_size":0,"ondisk_log_size":9876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40404876,"num_objects":9868,"num_object_clones":0,"num_object_copies":19736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9873,"num_write_kb":39463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5473},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":635519478,"num_objects":155172,"num_object_clones":0,"num_object_copies":310344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3359,"num_objects_unfound":0,"num_objects_dirty":155172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132801,"num_write_kb":531092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5051,"num_bytes_recovered":20607459,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1295757312,"data_stored":1295674213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155236,"ondisk_log_size":155236,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018932,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698672,"kb_used_data":446376,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93673168,"statfs":{"total":96636764160,"available":95921324032,"internally_reserved":0,"allocated":457089024,"data_stored":456835390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":6.274},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":7.4560000000000004}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.32000000000000001},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":13.807}]}]},{"osd":1,"up_from":40,"seq":171798691877,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501360,"kb_used_data":285160,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93870480,"statfs":{"total":96636764160,"available":96123371520,"internally_reserved":0,"allocated":292003840,"data_stored":291677946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":221376170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":13.659000000000001},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":15.837999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":2.8879999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215160,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":865580,"kb_used_data":532964,"kb_used_omap":4,"kb_used_meta":332603,"kb_avail":93506260,"statfs":{"total":96636764160,"available":95750410240,"internally_reserved":0,"allocated":545755136,"data_stored":545505116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":340586152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,3,7,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":0.184},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":4.9489999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":1.2270000000000001},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":546091008,"data_stored":546087395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292020224,"data_stored":291944351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":457646080,"data_stored":457642467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:55.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 223 20 157141 157121 2.75164 1.54297 0.123619 0.0283844 2024-09-16T10:26:56.565 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 224 20 157587 157567 2.74714 1.74219 0.0674952 0.0284317 2024-09-16T10:26:57.565 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 225 20 158038 158018 2.74276 1.76172 0.075999 0.0284761 2024-09-16T10:26:58.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:58.564 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 226 20 158499 158479 2.73859 1.80078 0.022975 0.02852 2024-09-16T10:26:59.096 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:59.099 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:59.124 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-16T10:26:58.635354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":643817993,"num_objects":157200,"num_object_clones":0,"num_object_copies":314400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3280,"num_objects_unfound":0,"num_objects_dirty":157200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134829,"num_write_kb":539198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5131,"num_bytes_recovered":20931062,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157265,"ondisk_log_size":157265,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2069096,"kb_used_data":1276176,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281046424,"statfs":{"total":289910292480,"available":287791538176,"internally_reserved":0,"allocated":1306804224,"data_stored":1305974676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":811912194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,4,12,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17801216,"num_objects":4346,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-156,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4346,"num_write_kb":17384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004296"},"pg_stats":[{"pgid":"2.f","version":"44'9788","reported_seq":11681,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:55.633526+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:55.633526+0000","last_peered":"2024-09-16T10:26:55.633526+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:55.633526+0000","last_undegraded":"2024-09-16T10:26:55.633526+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9788,"log_dups_size":0,"ondisk_log_size":9788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40091648,"num_objects":9788,"num_object_clones":0,"num_object_copies":19576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3280,"num_objects_unfound":0,"num_objects_dirty":9788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9788,"num_write_kb":39152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":3735552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6508},{"shards":"0,2","objects":3280}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9808","reported_seq":9965,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.998267+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:55.998267+0000","last_peered":"2024-09-16T10:26:55.998267+0000","last_clean":"2024-09-16T10:26:55.998267+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:55.998267+0000","last_undegraded":"2024-09-16T10:26:55.998267+0000","last_fullsized":"2024-09-16T10:26:55.998267+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9808,"log_dups_size":0,"ondisk_log_size":9808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40148992,"num_objects":9802,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6021,"num_write_kb":24084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9897","reported_seq":10081,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:56.060067+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:56.060067+0000","last_peered":"2024-09-16T10:26:56.060067+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:56.060067+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40521728,"num_objects":9893,"num_object_clones":0,"num_object_copies":19786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":23816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5558}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9823","reported_seq":9879,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.974939+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:55.974939+0000","last_peered":"2024-09-16T10:26:55.974939+0000","last_clean":"2024-09-16T10:26:55.974939+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:55.974939+0000","last_undegraded":"2024-09-16T10:26:55.974939+0000","last_fullsized":"2024-09-16T10:26:55.974939+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9823,"log_dups_size":0,"ondisk_log_size":9823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40235008,"num_objects":9823,"num_object_clones":0,"num_object_copies":19646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9823,"num_write_kb":39292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9811","reported_seq":9967,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.572636+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:55.572636+0000","last_peered":"2024-09-16T10:26:55.572636+0000","last_clean":"2024-09-16T10:26:55.572636+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:55.572636+0000","last_undegraded":"2024-09-16T10:26:55.572636+0000","last_fullsized":"2024-09-16T10:26:55.572636+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40157184,"num_objects":9804,"num_object_clones":0,"num_object_copies":19608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":25812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9710","reported_seq":9873,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.539549+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:55.539549+0000","last_peered":"2024-09-16T10:26:55.539549+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:55.539549+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39751680,"num_objects":9705,"num_object_clones":0,"num_object_copies":19410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6335,"num_write_kb":25340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5488},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10026","reported_seq":10199,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.602431+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:55.602431+0000","last_peered":"2024-09-16T10:26:55.602431+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:55.602431+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41041920,"num_objects":10020,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6268,"num_write_kb":25072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5617},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9758","reported_seq":9933,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.618419+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:55.618419+0000","last_peered":"2024-09-16T10:26:55.618419+0000","last_clean":"2024-09-16T10:26:55.618419+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:55.618419+0000","last_undegraded":"2024-09-16T10:26:55.618419+0000","last_fullsized":"2024-09-16T10:26:55.618419+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9758,"log_dups_size":0,"ondisk_log_size":9758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39960576,"num_objects":9756,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5561,"num_write_kb":22244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9746","reported_seq":18249,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:53.942756+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:53.942756+0000","last_peered":"2024-09-16T10:26:53.942756+0000","last_clean":"2024-09-16T10:26:53.942756+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:53.942756+0000","last_undegraded":"2024-09-16T10:26:53.942756+0000","last_fullsized":"2024-09-16T10:26:53.942756+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39891910,"num_objects":9741,"num_object_clones":0,"num_object_copies":19482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":38960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10052","reported_seq":10131,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.568520+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:55.568520+0000","last_peered":"2024-09-16T10:26:55.568520+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:55.568520+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41141190,"num_objects":10046,"num_object_clones":0,"num_object_copies":20092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":40180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5612},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9783","reported_seq":9888,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:56.063519+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:56.063519+0000","last_peered":"2024-09-16T10:26:56.063519+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:56.063519+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9783,"log_dups_size":0,"ondisk_log_size":9783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40051654,"num_objects":9780,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9781,"num_write_kb":39115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5503}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9843","reported_seq":9929,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:56.063429+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:56.063429+0000","last_peered":"2024-09-16T10:26:56.063429+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:56.063429+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9843,"log_dups_size":0,"ondisk_log_size":9843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40293318,"num_objects":9839,"num_object_clones":0,"num_object_copies":19678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9841,"num_write_kb":39351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5647}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9570","reported_seq":9649,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.597689+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:55.597689+0000","last_peered":"2024-09-16T10:26:55.597689+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:55.597689+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39191011,"num_objects":9569,"num_object_clones":0,"num_object_copies":19138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9569,"num_write_kb":38273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5410},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10005","reported_seq":10138,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.651807+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:55.651807+0000","last_peered":"2024-09-16T10:26:55.651807+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:55.651807+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40949161,"num_objects":10000,"num_object_clones":0,"num_object_copies":20000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10002,"num_write_kb":39992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5663},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9635","reported_seq":9715,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.645715+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:55.645715+0000","last_peered":"2024-09-16T10:26:55.645715+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:55.645715+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9635,"log_dups_size":0,"ondisk_log_size":9635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39449542,"num_objects":9633,"num_object_clones":0,"num_object_copies":19266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9633,"num_write_kb":38526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5466},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10007","reported_seq":10099,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.539641+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:55.539641+0000","last_peered":"2024-09-16T10:26:55.539641+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:55.539641+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40941452,"num_objects":9999,"num_object_clones":0,"num_object_copies":19998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10004,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5604},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":643817974,"num_objects":157198,"num_object_clones":0,"num_object_copies":314396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3280,"num_objects_unfound":0,"num_objects_dirty":157198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134827,"num_write_kb":539196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5130,"num_bytes_recovered":20931043,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1306796032,"data_stored":1306712933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157262,"ondisk_log_size":157262,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018933,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707968,"kb_used_data":451576,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93663872,"statfs":{"total":96636764160,"available":95911804928,"internally_reserved":0,"allocated":462413824,"data_stored":462160190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":262532784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.1779999999999999},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.81899999999999995}]}]},{"osd":1,"up_from":40,"seq":171798691877,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501360,"kb_used_data":285160,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93870480,"statfs":{"total":96636764160,"available":96123371520,"internally_reserved":0,"allocated":292003840,"data_stored":291677946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":221376170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":13.659000000000001},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":15.837999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":2.8879999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215161,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859768,"kb_used_data":539440,"kb_used_omap":4,"kb_used_meta":320315,"kb_avail":93512072,"statfs":{"total":96636764160,"available":95756361728,"internally_reserved":0,"allocated":552386560,"data_stored":552136540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":328003240},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":0.51600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":3.6629999999999998},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.38800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":552710144,"data_stored":552706531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292020224,"data_stored":291944351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462065664,"data_stored":462062051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:59.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:26:59.565 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 227 20 158898 158878 2.73339 1.55859 0.0908767 0.0285748 2024-09-16T10:26:59.719 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:26:59.720 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:26:59.744 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-16T10:26:58.635354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":643817993,"num_objects":157200,"num_object_clones":0,"num_object_copies":314400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3280,"num_objects_unfound":0,"num_objects_dirty":157200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134829,"num_write_kb":539198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5131,"num_bytes_recovered":20931062,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157265,"ondisk_log_size":157265,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2069096,"kb_used_data":1276176,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281046424,"statfs":{"total":289910292480,"available":287791538176,"internally_reserved":0,"allocated":1306804224,"data_stored":1305974676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":811912194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,4,12,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17801216,"num_objects":4346,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-156,"num_objects_unfound":0,"num_objects_dirty":4346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4346,"num_write_kb":17384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004296"},"pg_stats":[{"pgid":"2.f","version":"44'9788","reported_seq":11681,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:55.633526+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:55.633526+0000","last_peered":"2024-09-16T10:26:55.633526+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:55.633526+0000","last_undegraded":"2024-09-16T10:26:55.633526+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9788,"log_dups_size":0,"ondisk_log_size":9788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40091648,"num_objects":9788,"num_object_clones":0,"num_object_copies":19576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3280,"num_objects_unfound":0,"num_objects_dirty":9788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9788,"num_write_kb":39152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":3735552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6508},{"shards":"0,2","objects":3280}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9808","reported_seq":9965,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.998267+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:55.998267+0000","last_peered":"2024-09-16T10:26:55.998267+0000","last_clean":"2024-09-16T10:26:55.998267+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:55.998267+0000","last_undegraded":"2024-09-16T10:26:55.998267+0000","last_fullsized":"2024-09-16T10:26:55.998267+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9808,"log_dups_size":0,"ondisk_log_size":9808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40148992,"num_objects":9802,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6021,"num_write_kb":24084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9897","reported_seq":10081,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:56.060067+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:56.060067+0000","last_peered":"2024-09-16T10:26:56.060067+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:56.060067+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40521728,"num_objects":9893,"num_object_clones":0,"num_object_copies":19786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":23816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5558}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9823","reported_seq":9879,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.974939+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:55.974939+0000","last_peered":"2024-09-16T10:26:55.974939+0000","last_clean":"2024-09-16T10:26:55.974939+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:55.974939+0000","last_undegraded":"2024-09-16T10:26:55.974939+0000","last_fullsized":"2024-09-16T10:26:55.974939+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9823,"log_dups_size":0,"ondisk_log_size":9823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40235008,"num_objects":9823,"num_object_clones":0,"num_object_copies":19646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9823,"num_write_kb":39292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9811","reported_seq":9967,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.572636+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:55.572636+0000","last_peered":"2024-09-16T10:26:55.572636+0000","last_clean":"2024-09-16T10:26:55.572636+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:55.572636+0000","last_undegraded":"2024-09-16T10:26:55.572636+0000","last_fullsized":"2024-09-16T10:26:55.572636+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40157184,"num_objects":9804,"num_object_clones":0,"num_object_copies":19608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":25812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9710","reported_seq":9873,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.539549+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:55.539549+0000","last_peered":"2024-09-16T10:26:55.539549+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:55.539549+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39751680,"num_objects":9705,"num_object_clones":0,"num_object_copies":19410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6335,"num_write_kb":25340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5488},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10026","reported_seq":10199,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.602431+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:55.602431+0000","last_peered":"2024-09-16T10:26:55.602431+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:55.602431+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41041920,"num_objects":10020,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6268,"num_write_kb":25072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5617},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9758","reported_seq":9933,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.618419+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:55.618419+0000","last_peered":"2024-09-16T10:26:55.618419+0000","last_clean":"2024-09-16T10:26:55.618419+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:55.618419+0000","last_undegraded":"2024-09-16T10:26:55.618419+0000","last_fullsized":"2024-09-16T10:26:55.618419+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9758,"log_dups_size":0,"ondisk_log_size":9758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39960576,"num_objects":9756,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5561,"num_write_kb":22244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9746","reported_seq":18249,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:53.942756+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:53.942756+0000","last_peered":"2024-09-16T10:26:53.942756+0000","last_clean":"2024-09-16T10:26:53.942756+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:53.942756+0000","last_undegraded":"2024-09-16T10:26:53.942756+0000","last_fullsized":"2024-09-16T10:26:53.942756+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9746,"log_dups_size":0,"ondisk_log_size":9746,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39891910,"num_objects":9741,"num_object_clones":0,"num_object_copies":19482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":38960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10052","reported_seq":10131,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.568520+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:55.568520+0000","last_peered":"2024-09-16T10:26:55.568520+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:55.568520+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41141190,"num_objects":10046,"num_object_clones":0,"num_object_copies":20092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":40180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5612},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9783","reported_seq":9888,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:56.063519+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:56.063519+0000","last_peered":"2024-09-16T10:26:56.063519+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:56.063519+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9783,"log_dups_size":0,"ondisk_log_size":9783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40051654,"num_objects":9780,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9781,"num_write_kb":39115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5503}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9843","reported_seq":9929,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:56.063429+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:56.063429+0000","last_peered":"2024-09-16T10:26:56.063429+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:56.063429+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9843,"log_dups_size":0,"ondisk_log_size":9843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40293318,"num_objects":9839,"num_object_clones":0,"num_object_copies":19678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9841,"num_write_kb":39351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5647}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9570","reported_seq":9649,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.597689+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:55.597689+0000","last_peered":"2024-09-16T10:26:55.597689+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:55.597689+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39191011,"num_objects":9569,"num_object_clones":0,"num_object_copies":19138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9569,"num_write_kb":38273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5410},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10005","reported_seq":10138,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.651807+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:55.651807+0000","last_peered":"2024-09-16T10:26:55.651807+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:55.651807+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40949161,"num_objects":10000,"num_object_clones":0,"num_object_copies":20000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10002,"num_write_kb":39992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5663},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9635","reported_seq":9715,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.645715+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:55.645715+0000","last_peered":"2024-09-16T10:26:55.645715+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:55.645715+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9635,"log_dups_size":0,"ondisk_log_size":9635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39449542,"num_objects":9633,"num_object_clones":0,"num_object_copies":19266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9633,"num_write_kb":38526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5466},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10007","reported_seq":10099,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.539641+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:55.539641+0000","last_peered":"2024-09-16T10:26:55.539641+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:55.539641+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40941452,"num_objects":9999,"num_object_clones":0,"num_object_copies":19998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10004,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5604},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":643817974,"num_objects":157198,"num_object_clones":0,"num_object_copies":314396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3280,"num_objects_unfound":0,"num_objects_dirty":157198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134827,"num_write_kb":539196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5130,"num_bytes_recovered":20931043,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1306796032,"data_stored":1306712933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157262,"ondisk_log_size":157262,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018933,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707968,"kb_used_data":451576,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93663872,"statfs":{"total":96636764160,"available":95911804928,"internally_reserved":0,"allocated":462413824,"data_stored":462160190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":262532784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.1779999999999999},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.81899999999999995}]}]},{"osd":1,"up_from":40,"seq":171798691877,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501360,"kb_used_data":285160,"kb_used_omap":4,"kb_used_meta":216187,"kb_avail":93870480,"statfs":{"total":96636764160,"available":96123371520,"internally_reserved":0,"allocated":292003840,"data_stored":291677946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":221376170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":13.659000000000001},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":15.837999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":2.8879999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215161,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859768,"kb_used_data":539440,"kb_used_omap":4,"kb_used_meta":320315,"kb_avail":93512072,"statfs":{"total":96636764160,"available":95756361728,"internally_reserved":0,"allocated":552386560,"data_stored":552136540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":328003240},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":0.51600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":3.6629999999999998},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.38800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":552710144,"data_stored":552706531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292020224,"data_stored":291944351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462065664,"data_stored":462062051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:26:59.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:26:59.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'9788", 'reported_seq': 11681, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:26:55.633526+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:26:55.633526+0000', 'last_peered': '2024-09-16T10:26:55.633526+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:26:55.633526+0000', 'last_undegraded': '2024-09-16T10:26:55.633526+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9788, 'log_dups_size': 0, 'ondisk_log_size': 9788, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40091648, 'num_objects': 9788, 'num_object_clones': 0, 'num_object_copies': 19576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3280, 'num_objects_unfound': 0, 'num_objects_dirty': 9788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9788, 'num_write_kb': 39152, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 912, 'num_bytes_recovered': 3735552, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6508}, {'shards': '0,2', 'objects': 3280}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:59.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:26:59.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'9897", 'reported_seq': 10081, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:56.060067+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:26:56.060067+0000', 'last_peered': '2024-09-16T10:26:56.060067+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:26:56.060067+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 9897, 'log_dups_size': 0, 'ondisk_log_size': 9897, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40521728, 'num_objects': 9893, 'num_object_clones': 0, 'num_object_copies': 19786, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9893, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5954, 'num_write_kb': 23816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 5558}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:59.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:26:59.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'9710", 'reported_seq': 9873, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:55.539549+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:26:55.539549+0000', 'last_peered': '2024-09-16T10:26:55.539549+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:26:55.539549+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 9710, 'log_dups_size': 0, 'ondisk_log_size': 9710, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39751680, 'num_objects': 9705, 'num_object_clones': 0, 'num_object_copies': 19410, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6335, 'num_write_kb': 25340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5488}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:59.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:26:59.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'10026", 'reported_seq': 10199, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:55.602431+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:26:55.602431+0000', 'last_peered': '2024-09-16T10:26:55.602431+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:26:55.602431+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41041920, 'num_objects': 10020, 'num_object_clones': 0, 'num_object_copies': 20040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6268, 'num_write_kb': 25072, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5617}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:59.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:26:59.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'10052", 'reported_seq': 10131, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:55.568520+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:26:55.568520+0000', 'last_peered': '2024-09-16T10:26:55.568520+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:26:55.568520+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41141190, 'num_objects': 10046, 'num_object_clones': 0, 'num_object_copies': 20092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10050, 'num_write_kb': 40180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5612}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:59.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:26:59.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'9783", 'reported_seq': 9888, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:56.063519+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:26:56.063519+0000', 'last_peered': '2024-09-16T10:26:56.063519+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:26:56.063519+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 9783, 'log_dups_size': 0, 'ondisk_log_size': 9783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40051654, 'num_objects': 9780, 'num_object_clones': 0, 'num_object_copies': 19560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9781, 'num_write_kb': 39115, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 5503}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:59.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:26:59.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'9843", 'reported_seq': 9929, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:56.063429+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:26:56.063429+0000', 'last_peered': '2024-09-16T10:26:56.063429+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:26:56.063429+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9843, 'log_dups_size': 0, 'ondisk_log_size': 9843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40293318, 'num_objects': 9839, 'num_object_clones': 0, 'num_object_copies': 19678, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9841, 'num_write_kb': 39351, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 5647}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:26:59.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:26:59.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'9570", 'reported_seq': 9649, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:55.597689+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:26:55.597689+0000', 'last_peered': '2024-09-16T10:26:55.597689+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:26:55.597689+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9570, 'log_dups_size': 0, 'ondisk_log_size': 9570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39191011, 'num_objects': 9569, 'num_object_clones': 0, 'num_object_copies': 19138, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9569, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9569, 'num_write_kb': 38273, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5410}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:59.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:26:59.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'10005", 'reported_seq': 10138, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:55.651807+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:26:55.651807+0000', 'last_peered': '2024-09-16T10:26:55.651807+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:26:55.651807+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40949161, 'num_objects': 10000, 'num_object_clones': 0, 'num_object_copies': 20000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10002, 'num_write_kb': 39992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5663}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:59.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:26:59.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'9635", 'reported_seq': 9715, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:55.645715+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:26:55.645715+0000', 'last_peered': '2024-09-16T10:26:55.645715+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:26:55.645715+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9635, 'log_dups_size': 0, 'ondisk_log_size': 9635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39449542, 'num_objects': 9633, 'num_object_clones': 0, 'num_object_copies': 19266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9633, 'num_write_kb': 38526, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5466}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:59.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:26:59.750 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'10007", 'reported_seq': 10099, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:26:55.539641+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:26:55.539641+0000', 'last_peered': '2024-09-16T10:26:55.539641+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:26:55.539641+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40941452, 'num_objects': 9999, 'num_object_clones': 0, 'num_object_copies': 19998, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9999, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10004, 'num_write_kb': 39987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5604}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:26:59.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:00.435 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:00.435 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/314400 objects degraded (13.656%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":309,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":157200,"data_bytes":643817993,"bytes_used":2118754304,"bytes_avail":287791538176,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":314400,"degraded_ratio":0.1365648854961832,"misplaced_objects":3280,"misplaced_total":314400,"misplaced_ratio":0.010432569974554707,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":53228,"recovering_keys_per_sec":0,"num_objects_recovered":156,"num_bytes_recovered":638976,"num_keys_recovered":0,"write_bytes_sec":1482903,"read_op_per_sec":0,"write_op_per_sec":362},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:00.459 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:00.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:00.565 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 228 20 159292 159272 2.72815 1.53906 0.0462855 0.0286285 2024-09-16T10:27:01.034 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:01.034 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:01.067 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-16T10:27:00.636232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":644362761,"num_objects":157333,"num_object_clones":0,"num_object_copies":314666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3280,"num_objects_unfound":0,"num_objects_dirty":157333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134962,"num_write_kb":539730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5131,"num_bytes_recovered":20931062,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157398,"ondisk_log_size":157398,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2080100,"kb_used_data":1283084,"kb_used_omap":12,"kb_used_meta":796979,"kb_avail":281035420,"statfs":{"total":289910292480,"available":287780270080,"internally_reserved":0,"allocated":1313878016,"data_stored":1313048468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":816106498},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,12,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18345984,"num_objects":4479,"num_object_clones":0,"num_object_copies":8958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-156,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4479,"num_write_kb":17916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004646"},"pg_stats":[{"pgid":"2.f","version":"44'9788","reported_seq":11681,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:26:55.633526+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:26:55.633526+0000","last_peered":"2024-09-16T10:26:55.633526+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:26:55.633526+0000","last_undegraded":"2024-09-16T10:26:55.633526+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9788,"log_dups_size":0,"ondisk_log_size":9788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40091648,"num_objects":9788,"num_object_clones":0,"num_object_copies":19576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3280,"num_objects_unfound":0,"num_objects_dirty":9788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9788,"num_write_kb":39152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":3735552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6508},{"shards":"0,2","objects":3280}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9808","reported_seq":9965,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.998267+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:26:55.998267+0000","last_peered":"2024-09-16T10:26:55.998267+0000","last_clean":"2024-09-16T10:26:55.998267+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:26:55.998267+0000","last_undegraded":"2024-09-16T10:26:55.998267+0000","last_fullsized":"2024-09-16T10:26:55.998267+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9808,"log_dups_size":0,"ondisk_log_size":9808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40148992,"num_objects":9802,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6021,"num_write_kb":24084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'9897","reported_seq":10081,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:56.060067+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:26:56.060067+0000","last_peered":"2024-09-16T10:26:56.060067+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:26:56.060067+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40521728,"num_objects":9893,"num_object_clones":0,"num_object_copies":19786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":23816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5558}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9823","reported_seq":9879,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.974939+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:26:55.974939+0000","last_peered":"2024-09-16T10:26:55.974939+0000","last_clean":"2024-09-16T10:26:55.974939+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:26:55.974939+0000","last_undegraded":"2024-09-16T10:26:55.974939+0000","last_fullsized":"2024-09-16T10:26:55.974939+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9823,"log_dups_size":0,"ondisk_log_size":9823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40235008,"num_objects":9823,"num_object_clones":0,"num_object_copies":19646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9823,"num_write_kb":39292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9811","reported_seq":9967,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.572636+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:26:55.572636+0000","last_peered":"2024-09-16T10:26:55.572636+0000","last_clean":"2024-09-16T10:26:55.572636+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:26:55.572636+0000","last_undegraded":"2024-09-16T10:26:55.572636+0000","last_fullsized":"2024-09-16T10:26:55.572636+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9811,"log_dups_size":0,"ondisk_log_size":9811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40157184,"num_objects":9804,"num_object_clones":0,"num_object_copies":19608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":25812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9710","reported_seq":9873,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.539549+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:26:55.539549+0000","last_peered":"2024-09-16T10:26:55.539549+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:26:55.539549+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9710,"log_dups_size":0,"ondisk_log_size":9710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39751680,"num_objects":9705,"num_object_clones":0,"num_object_copies":19410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6335,"num_write_kb":25340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5488},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10026","reported_seq":10199,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.602431+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:26:55.602431+0000","last_peered":"2024-09-16T10:26:55.602431+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:26:55.602431+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41041920,"num_objects":10020,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6268,"num_write_kb":25072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5617},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9758","reported_seq":9933,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:55.618419+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:26:55.618419+0000","last_peered":"2024-09-16T10:26:55.618419+0000","last_clean":"2024-09-16T10:26:55.618419+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:26:55.618419+0000","last_undegraded":"2024-09-16T10:26:55.618419+0000","last_fullsized":"2024-09-16T10:26:55.618419+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9758,"log_dups_size":0,"ondisk_log_size":9758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39960576,"num_objects":9756,"num_object_clones":0,"num_object_copies":19512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5561,"num_write_kb":22244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'9879","reported_seq":18382,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:58.889232+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:26:58.889232+0000","last_peered":"2024-09-16T10:26:58.889232+0000","last_clean":"2024-09-16T10:26:58.889232+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:26:58.889232+0000","last_undegraded":"2024-09-16T10:26:58.889232+0000","last_fullsized":"2024-09-16T10:26:58.889232+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9879,"log_dups_size":0,"ondisk_log_size":9879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40436678,"num_objects":9874,"num_object_clones":0,"num_object_copies":19748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9877,"num_write_kb":39492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.946222+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:00.946222+0000","last_peered":"2024-09-16T10:26:00.946222+0000","last_clean":"2024-09-16T10:26:00.946222+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:00.946222+0000","last_undegraded":"2024-09-16T10:26:00.946222+0000","last_fullsized":"2024-09-16T10:26:00.946222+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10052","reported_seq":10131,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.568520+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:26:55.568520+0000","last_peered":"2024-09-16T10:26:55.568520+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:26:55.568520+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41141190,"num_objects":10046,"num_object_clones":0,"num_object_copies":20092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10050,"num_write_kb":40180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5612},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9783","reported_seq":9888,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:56.063519+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:26:56.063519+0000","last_peered":"2024-09-16T10:26:56.063519+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:26:56.063519+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9783,"log_dups_size":0,"ondisk_log_size":9783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40051654,"num_objects":9780,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9781,"num_write_kb":39115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5503}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9843","reported_seq":9929,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:56.063429+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:26:56.063429+0000","last_peered":"2024-09-16T10:26:56.063429+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:26:56.063429+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9843,"log_dups_size":0,"ondisk_log_size":9843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40293318,"num_objects":9839,"num_object_clones":0,"num_object_copies":19678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9841,"num_write_kb":39351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5647}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9570","reported_seq":9649,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.597689+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:26:55.597689+0000","last_peered":"2024-09-16T10:26:55.597689+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:26:55.597689+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9570,"log_dups_size":0,"ondisk_log_size":9570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39191011,"num_objects":9569,"num_object_clones":0,"num_object_copies":19138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9569,"num_write_kb":38273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5410},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10005","reported_seq":10138,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.651807+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:26:55.651807+0000","last_peered":"2024-09-16T10:26:55.651807+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:26:55.651807+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40949161,"num_objects":10000,"num_object_clones":0,"num_object_copies":20000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10002,"num_write_kb":39992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5663},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9635","reported_seq":9715,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.645715+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:26:55.645715+0000","last_peered":"2024-09-16T10:26:55.645715+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:26:55.645715+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9635,"log_dups_size":0,"ondisk_log_size":9635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39449542,"num_objects":9633,"num_object_clones":0,"num_object_copies":19266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9633,"num_write_kb":38526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5466},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10007","reported_seq":10099,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:26:55.539641+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:26:55.539641+0000","last_peered":"2024-09-16T10:26:55.539641+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:26:55.539641+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40941452,"num_objects":9999,"num_object_clones":0,"num_object_copies":19998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10004,"num_write_kb":39987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5604},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":644362742,"num_objects":157331,"num_object_clones":0,"num_object_copies":314662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3280,"num_objects_unfound":0,"num_objects_dirty":157331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":134960,"num_write_kb":539728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5130,"num_bytes_recovered":20931043,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1313906688,"data_stored":1313823589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157395,"ondisk_log_size":157395,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018933,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707968,"kb_used_data":451576,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93663872,"statfs":{"total":96636764160,"available":95911804928,"internally_reserved":0,"allocated":462413824,"data_stored":462160190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":262532784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.1339999999999999,"15min":0.95199999999999996},"min":{"1min":0.26800000000000002,"5min":0.184,"15min":0.184},"max":{"1min":6.6299999999999999,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.1779999999999999},{"interface":"front","average":{"1min":3.2330000000000001,"5min":1.742,"15min":1.1799999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":8.9879999999999995,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:25:58 2024","interfaces":[{"interface":"back","average":{"1min":2.5609999999999999,"5min":2.8300000000000001,"15min":2.875},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":10.492000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.2240000000000002,"15min":2.2349999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":8.9659999999999993,"5min":8.9659999999999993,"15min":8.9659999999999993},"last":0.81899999999999995}]}]},{"osd":1,"up_from":40,"seq":171798691878,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":512364,"kb_used_data":292068,"kb_used_omap":4,"kb_used_meta":220283,"kb_avail":93859476,"statfs":{"total":96636764160,"available":96112103424,"internally_reserved":0,"allocated":299077632,"data_stored":298751738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":225570474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":3.8220000000000001,"5min":3.6440000000000001,"15min":3.6139999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.819000000000001,"5min":14.244,"15min":14.244},"last":0.46200000000000002},{"interface":"front","average":{"1min":4.0999999999999996,"5min":2.7440000000000002,"15min":2.5179999999999998},"min":{"1min":0.32100000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":16.850999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.8980000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:25:59 2024","interfaces":[{"interface":"back","average":{"1min":2.6739999999999999,"5min":2.3900000000000001,"15min":2.3420000000000001},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.2199999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":2.9649999999999999},{"interface":"front","average":{"1min":2.4489999999999998,"5min":2.8039999999999998,"15min":2.863},"min":{"1min":0.317,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.2910000000000004,"5min":11.394,"15min":11.394},"last":3}]}]},{"osd":0,"up_from":24,"seq":103079215161,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859768,"kb_used_data":539440,"kb_used_omap":4,"kb_used_meta":320315,"kb_avail":93512072,"statfs":{"total":96636764160,"available":95756361728,"internally_reserved":0,"allocated":552386560,"data_stored":552136540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":328003240},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:25:56 2024","interfaces":[{"interface":"back","average":{"1min":2.6070000000000002,"5min":2.7690000000000001,"15min":2.7959999999999998},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":10.148,"5min":10.272,"15min":10.272},"last":0.35999999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.986,"15min":2.0099999999999998},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.96,"5min":5.96,"15min":5.96},"last":0.51600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:26:00 2024","interfaces":[{"interface":"back","average":{"1min":2.6280000000000001,"5min":1.3740000000000001,"15min":0.85999999999999999},"min":{"1min":0.21199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":3.6629999999999998},{"interface":"front","average":{"1min":2.649,"5min":1.746,"15min":1.357},"min":{"1min":0.25900000000000001,"5min":0.192,"15min":0.192},"max":{"1min":7.2149999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"last":0.38800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":552710144,"data_stored":552706531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":299130880,"data_stored":299055007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":462065664,"data_stored":462062051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:01.565 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 229 20 159719 159699 2.72352 1.66797 0.0328324 0.0286789 2024-09-16T10:27:02.565 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 230 20 160165 160145 2.71926 1.74219 0.0134327 0.0287225 2024-09-16T10:27:03.565 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 231 20 160607 160587 2.71496 1.72656 0.025187 0.0287673 2024-09-16T10:27:04.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:04.565 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 232 20 161027 161007 2.71033 1.64062 0.122939 0.0288165 2024-09-16T10:27:04.660 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:04.660 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:04.683 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-16T10:27:04.637933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653210121,"num_objects":159493,"num_object_clones":0,"num_object_copies":318986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3206,"num_objects_unfound":0,"num_objects_dirty":159493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137122,"num_write_kb":548370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5205,"num_bytes_recovered":21234166,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159158,"ondisk_log_size":159158,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2092788,"kb_used_data":1299868,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281022732,"statfs":{"total":289910292480,"available":287767277568,"internally_reserved":0,"allocated":1331064832,"data_stored":1330235284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":811912194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,6,8,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18280448,"num_objects":4463,"num_object_clones":0,"num_object_copies":8926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-153,"num_objects_unfound":0,"num_objects_dirty":4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4463,"num_write_kb":17852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":626688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005084"},"pg_stats":[{"pgid":"2.f","version":"44'9927","reported_seq":11968,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:00.725770+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:00.725770+0000","last_peered":"2024-09-16T10:27:00.725770+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:00.725770+0000","last_undegraded":"2024-09-16T10:27:00.725770+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40660992,"num_objects":9927,"num_object_clones":0,"num_object_copies":19854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3206,"num_objects_unfound":0,"num_objects_dirty":9927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9927,"num_write_kb":39708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":986,"num_bytes_recovered":4038656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6721},{"shards":"0,2","objects":3206}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9944","reported_seq":10101,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:00.947908+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:00.947908+0000","last_peered":"2024-09-16T10:27:00.947908+0000","last_clean":"2024-09-16T10:27:00.947908+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:00.947908+0000","last_undegraded":"2024-09-16T10:27:00.947908+0000","last_fullsized":"2024-09-16T10:27:00.947908+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9944,"log_dups_size":0,"ondisk_log_size":9944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40706048,"num_objects":9938,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":24628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10041","reported_seq":10224,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:01.083134+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:01.083134+0000","last_peered":"2024-09-16T10:27:01.083134+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:01.083134+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41111552,"num_objects":10037,"num_object_clones":0,"num_object_copies":20074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6098,"num_write_kb":24392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9937","reported_seq":9993,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:01.088579+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:01.088579+0000","last_peered":"2024-09-16T10:27:01.088579+0000","last_clean":"2024-09-16T10:27:01.088579+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:01.088579+0000","last_undegraded":"2024-09-16T10:27:01.088579+0000","last_fullsized":"2024-09-16T10:27:01.088579+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40701952,"num_objects":9937,"num_object_clones":0,"num_object_copies":19874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9937,"num_write_kb":39748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9942","reported_seq":10099,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:00.749229+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:00.749229+0000","last_peered":"2024-09-16T10:27:00.749229+0000","last_clean":"2024-09-16T10:27:00.749229+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:00.749229+0000","last_undegraded":"2024-09-16T10:27:00.749229+0000","last_fullsized":"2024-09-16T10:27:00.749229+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9942,"log_dups_size":0,"ondisk_log_size":9942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40693760,"num_objects":9935,"num_object_clones":0,"num_object_copies":19870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":26336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9855","reported_seq":10015,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.582137+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:00.582137+0000","last_peered":"2024-09-16T10:27:00.582137+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:00.582137+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9855,"log_dups_size":0,"ondisk_log_size":9855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40345600,"num_objects":9850,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6480,"num_write_kb":25920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5633},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10165","reported_seq":10338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.656122+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:00.656122+0000","last_peered":"2024-09-16T10:27:00.656122+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:00.656122+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41611264,"num_objects":10159,"num_object_clones":0,"num_object_copies":20318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":25628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5756},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9881","reported_seq":10056,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:00.671529+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:00.671529+0000","last_peered":"2024-09-16T10:27:00.671529+0000","last_clean":"2024-09-16T10:27:00.671529+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:00.671529+0000","last_undegraded":"2024-09-16T10:27:00.671529+0000","last_fullsized":"2024-09-16T10:27:00.671529+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40464384,"num_objects":9879,"num_object_clones":0,"num_object_copies":19758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5684,"num_write_kb":22736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10002","reported_seq":18505,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:04.055522+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:04.055522+0000","last_peered":"2024-09-16T10:27:04.055522+0000","last_clean":"2024-09-16T10:27:04.055522+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:04.055522+0000","last_undegraded":"2024-09-16T10:27:04.055522+0000","last_fullsized":"2024-09-16T10:27:04.055522+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40940486,"num_objects":9997,"num_object_clones":0,"num_object_copies":19994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":39984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10195","reported_seq":10270,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.717168+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:00.717168+0000","last_peered":"2024-09-16T10:27:00.717168+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:00.717168+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41726918,"num_objects":10189,"num_object_clones":0,"num_object_copies":20378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10193,"num_write_kb":40752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5755},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9921","reported_seq":10026,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:01.074178+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:01.074178+0000","last_peered":"2024-09-16T10:27:01.074178+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:01.074178+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9921,"log_dups_size":0,"ondisk_log_size":9921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40616902,"num_objects":9918,"num_object_clones":0,"num_object_copies":19836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":39667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5641}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9986","reported_seq":10073,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:01.035564+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:01.035564+0000","last_peered":"2024-09-16T10:27:01.035564+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:01.035564+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9986,"log_dups_size":0,"ondisk_log_size":9986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40879046,"num_objects":9982,"num_object_clones":0,"num_object_copies":19964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9984,"num_write_kb":39923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9690","reported_seq":9769,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.602162+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:00.602162+0000","last_peered":"2024-09-16T10:27:00.602162+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:00.602162+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39682531,"num_objects":9689,"num_object_clones":0,"num_object_copies":19378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":38753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5530},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10154","reported_seq":10285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.647076+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:00.647076+0000","last_peered":"2024-09-16T10:27:00.647076+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:00.647076+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41559465,"num_objects":10149,"num_object_clones":0,"num_object_copies":20298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10151,"num_write_kb":40588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5812},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9769","reported_seq":9850,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.725998+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:00.725998+0000","last_peered":"2024-09-16T10:27:00.725998+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:00.725998+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39998406,"num_objects":9767,"num_object_clones":0,"num_object_copies":19534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9767,"num_write_kb":39062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5600},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10146","reported_seq":10239,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.726015+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:00.726015+0000","last_peered":"2024-09-16T10:27:00.726015+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:00.726015+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41510796,"num_objects":10138,"num_object_clones":0,"num_object_copies":20276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10143,"num_write_kb":40543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5743},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":653210102,"num_objects":159491,"num_object_clones":0,"num_object_copies":318982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3206,"num_objects_unfound":0,"num_objects_dirty":159491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137120,"num_write_kb":548368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5204,"num_bytes_recovered":21234147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1331908608,"data_stored":1331825509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159155,"ondisk_log_size":159155,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018934,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715560,"kb_used_data":455072,"kb_used_omap":4,"kb_used_meta":260475,"kb_avail":93656280,"statfs":{"total":96636764160,"available":95904030720,"internally_reserved":0,"allocated":465993728,"data_stored":465740094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":266727088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":4.2720000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":9.5640000000000001},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.76100000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507016,"kb_used_data":299008,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93864824,"statfs":{"total":96636764160,"available":96117579776,"internally_reserved":0,"allocated":306184192,"data_stored":305858298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":212987562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.1349999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":4.875},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":6.5919999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215162,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":870212,"kb_used_data":545788,"kb_used_omap":4,"kb_used_meta":324411,"kb_avail":93501628,"statfs":{"total":96636764160,"available":95745667072,"internally_reserved":0,"allocated":558886912,"data_stored":558636892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":332197544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,3,1,6,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":9.359},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":3.5649999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":10.919},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":9.3290000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":559280128,"data_stored":559276515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306176000,"data_stored":306100127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":466452480,"data_stored":466448867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:04.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:05.234 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:05.237 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:05.257 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-16T10:27:04.637933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653210121,"num_objects":159493,"num_object_clones":0,"num_object_copies":318986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3206,"num_objects_unfound":0,"num_objects_dirty":159493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137122,"num_write_kb":548370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5205,"num_bytes_recovered":21234166,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159158,"ondisk_log_size":159158,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2092788,"kb_used_data":1299868,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281022732,"statfs":{"total":289910292480,"available":287767277568,"internally_reserved":0,"allocated":1331064832,"data_stored":1330235284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":811912194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,6,8,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18280448,"num_objects":4463,"num_object_clones":0,"num_object_copies":8926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-153,"num_objects_unfound":0,"num_objects_dirty":4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4463,"num_write_kb":17852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":626688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005084"},"pg_stats":[{"pgid":"2.f","version":"44'9927","reported_seq":11968,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:00.725770+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:00.725770+0000","last_peered":"2024-09-16T10:27:00.725770+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:00.725770+0000","last_undegraded":"2024-09-16T10:27:00.725770+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40660992,"num_objects":9927,"num_object_clones":0,"num_object_copies":19854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3206,"num_objects_unfound":0,"num_objects_dirty":9927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9927,"num_write_kb":39708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":986,"num_bytes_recovered":4038656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6721},{"shards":"0,2","objects":3206}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9944","reported_seq":10101,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:00.947908+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:00.947908+0000","last_peered":"2024-09-16T10:27:00.947908+0000","last_clean":"2024-09-16T10:27:00.947908+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:00.947908+0000","last_undegraded":"2024-09-16T10:27:00.947908+0000","last_fullsized":"2024-09-16T10:27:00.947908+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9944,"log_dups_size":0,"ondisk_log_size":9944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40706048,"num_objects":9938,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":24628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10041","reported_seq":10224,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:01.083134+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:01.083134+0000","last_peered":"2024-09-16T10:27:01.083134+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:01.083134+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41111552,"num_objects":10037,"num_object_clones":0,"num_object_copies":20074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6098,"num_write_kb":24392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9937","reported_seq":9993,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:01.088579+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:01.088579+0000","last_peered":"2024-09-16T10:27:01.088579+0000","last_clean":"2024-09-16T10:27:01.088579+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:01.088579+0000","last_undegraded":"2024-09-16T10:27:01.088579+0000","last_fullsized":"2024-09-16T10:27:01.088579+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40701952,"num_objects":9937,"num_object_clones":0,"num_object_copies":19874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9937,"num_write_kb":39748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9942","reported_seq":10099,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:00.749229+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:00.749229+0000","last_peered":"2024-09-16T10:27:00.749229+0000","last_clean":"2024-09-16T10:27:00.749229+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:00.749229+0000","last_undegraded":"2024-09-16T10:27:00.749229+0000","last_fullsized":"2024-09-16T10:27:00.749229+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9942,"log_dups_size":0,"ondisk_log_size":9942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40693760,"num_objects":9935,"num_object_clones":0,"num_object_copies":19870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":26336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9855","reported_seq":10015,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.582137+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:00.582137+0000","last_peered":"2024-09-16T10:27:00.582137+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:00.582137+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9855,"log_dups_size":0,"ondisk_log_size":9855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40345600,"num_objects":9850,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6480,"num_write_kb":25920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5633},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10165","reported_seq":10338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.656122+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:00.656122+0000","last_peered":"2024-09-16T10:27:00.656122+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:00.656122+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41611264,"num_objects":10159,"num_object_clones":0,"num_object_copies":20318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":25628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5756},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9881","reported_seq":10056,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:00.671529+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:00.671529+0000","last_peered":"2024-09-16T10:27:00.671529+0000","last_clean":"2024-09-16T10:27:00.671529+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:00.671529+0000","last_undegraded":"2024-09-16T10:27:00.671529+0000","last_fullsized":"2024-09-16T10:27:00.671529+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40464384,"num_objects":9879,"num_object_clones":0,"num_object_copies":19758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5684,"num_write_kb":22736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10002","reported_seq":18505,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:04.055522+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:04.055522+0000","last_peered":"2024-09-16T10:27:04.055522+0000","last_clean":"2024-09-16T10:27:04.055522+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:04.055522+0000","last_undegraded":"2024-09-16T10:27:04.055522+0000","last_fullsized":"2024-09-16T10:27:04.055522+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40940486,"num_objects":9997,"num_object_clones":0,"num_object_copies":19994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":39984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10195","reported_seq":10270,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.717168+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:00.717168+0000","last_peered":"2024-09-16T10:27:00.717168+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:00.717168+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41726918,"num_objects":10189,"num_object_clones":0,"num_object_copies":20378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10193,"num_write_kb":40752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5755},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9921","reported_seq":10026,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:01.074178+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:01.074178+0000","last_peered":"2024-09-16T10:27:01.074178+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:01.074178+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9921,"log_dups_size":0,"ondisk_log_size":9921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40616902,"num_objects":9918,"num_object_clones":0,"num_object_copies":19836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":39667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5641}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9986","reported_seq":10073,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:01.035564+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:01.035564+0000","last_peered":"2024-09-16T10:27:01.035564+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:01.035564+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9986,"log_dups_size":0,"ondisk_log_size":9986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40879046,"num_objects":9982,"num_object_clones":0,"num_object_copies":19964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9984,"num_write_kb":39923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9690","reported_seq":9769,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.602162+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:00.602162+0000","last_peered":"2024-09-16T10:27:00.602162+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:00.602162+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39682531,"num_objects":9689,"num_object_clones":0,"num_object_copies":19378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":38753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5530},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10154","reported_seq":10285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.647076+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:00.647076+0000","last_peered":"2024-09-16T10:27:00.647076+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:00.647076+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41559465,"num_objects":10149,"num_object_clones":0,"num_object_copies":20298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10151,"num_write_kb":40588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5812},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9769","reported_seq":9850,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.725998+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:00.725998+0000","last_peered":"2024-09-16T10:27:00.725998+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:00.725998+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39998406,"num_objects":9767,"num_object_clones":0,"num_object_copies":19534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9767,"num_write_kb":39062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5600},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10146","reported_seq":10239,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.726015+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:00.726015+0000","last_peered":"2024-09-16T10:27:00.726015+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:00.726015+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41510796,"num_objects":10138,"num_object_clones":0,"num_object_copies":20276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10143,"num_write_kb":40543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5743},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":653210102,"num_objects":159491,"num_object_clones":0,"num_object_copies":318982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3206,"num_objects_unfound":0,"num_objects_dirty":159491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137120,"num_write_kb":548368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5204,"num_bytes_recovered":21234147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1331908608,"data_stored":1331825509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159155,"ondisk_log_size":159155,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018934,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715560,"kb_used_data":455072,"kb_used_omap":4,"kb_used_meta":260475,"kb_avail":93656280,"statfs":{"total":96636764160,"available":95904030720,"internally_reserved":0,"allocated":465993728,"data_stored":465740094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":266727088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":4.2720000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":9.5640000000000001},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.76100000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507016,"kb_used_data":299008,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93864824,"statfs":{"total":96636764160,"available":96117579776,"internally_reserved":0,"allocated":306184192,"data_stored":305858298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":212987562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.1349999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":4.875},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":6.5919999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215162,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":870212,"kb_used_data":545788,"kb_used_omap":4,"kb_used_meta":324411,"kb_avail":93501628,"statfs":{"total":96636764160,"available":95745667072,"internally_reserved":0,"allocated":558886912,"data_stored":558636892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":332197544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,3,1,6,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":9.359},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":3.5649999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":10.919},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":9.3290000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":559280128,"data_stored":559276515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306176000,"data_stored":306100127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":466452480,"data_stored":466448867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:05.258 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:05.258 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'9927", 'reported_seq': 11968, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:00.725770+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:00.725770+0000', 'last_peered': '2024-09-16T10:27:00.725770+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:00.725770+0000', 'last_undegraded': '2024-09-16T10:27:00.725770+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9927, 'log_dups_size': 0, 'ondisk_log_size': 9927, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40660992, 'num_objects': 9927, 'num_object_clones': 0, 'num_object_copies': 19854, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3206, 'num_objects_unfound': 0, 'num_objects_dirty': 9927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9927, 'num_write_kb': 39708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 986, 'num_bytes_recovered': 4038656, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6721}, {'shards': '0,2', 'objects': 3206}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'10041", 'reported_seq': 10224, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:01.083134+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:01.083134+0000', 'last_peered': '2024-09-16T10:27:01.083134+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:01.083134+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41111552, 'num_objects': 10037, 'num_object_clones': 0, 'num_object_copies': 20074, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6098, 'num_write_kb': 24392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 5702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'9855", 'reported_seq': 10015, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:00.582137+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:00.582137+0000', 'last_peered': '2024-09-16T10:27:00.582137+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:00.582137+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 9855, 'log_dups_size': 0, 'ondisk_log_size': 9855, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40345600, 'num_objects': 9850, 'num_object_clones': 0, 'num_object_copies': 19700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9850, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6480, 'num_write_kb': 25920, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5633}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'10165", 'reported_seq': 10338, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:00.656122+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:00.656122+0000', 'last_peered': '2024-09-16T10:27:00.656122+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:00.656122+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41611264, 'num_objects': 10159, 'num_object_clones': 0, 'num_object_copies': 20318, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6407, 'num_write_kb': 25628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5756}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'10195", 'reported_seq': 10270, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:00.717168+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:00.717168+0000', 'last_peered': '2024-09-16T10:27:00.717168+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:00.717168+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41726918, 'num_objects': 10189, 'num_object_clones': 0, 'num_object_copies': 20378, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10193, 'num_write_kb': 40752, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5755}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'9921", 'reported_seq': 10026, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:01.074178+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:01.074178+0000', 'last_peered': '2024-09-16T10:27:01.074178+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:01.074178+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 9921, 'log_dups_size': 0, 'ondisk_log_size': 9921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40616902, 'num_objects': 9918, 'num_object_clones': 0, 'num_object_copies': 19836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9918, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9919, 'num_write_kb': 39667, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 5641}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:05.259 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:05.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'9986", 'reported_seq': 10073, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:01.035564+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:01.035564+0000', 'last_peered': '2024-09-16T10:27:01.035564+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:01.035564+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9986, 'log_dups_size': 0, 'ondisk_log_size': 9986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40879046, 'num_objects': 9982, 'num_object_clones': 0, 'num_object_copies': 19964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9984, 'num_write_kb': 39923, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 5790}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:05.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:05.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'9690", 'reported_seq': 9769, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:00.602162+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:00.602162+0000', 'last_peered': '2024-09-16T10:27:00.602162+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:00.602162+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9690, 'log_dups_size': 0, 'ondisk_log_size': 9690, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39682531, 'num_objects': 9689, 'num_object_clones': 0, 'num_object_copies': 19378, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9689, 'num_write_kb': 38753, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5530}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:05.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:05.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'10154", 'reported_seq': 10285, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:00.647076+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:00.647076+0000', 'last_peered': '2024-09-16T10:27:00.647076+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:00.647076+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41559465, 'num_objects': 10149, 'num_object_clones': 0, 'num_object_copies': 20298, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10151, 'num_write_kb': 40588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5812}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:05.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:05.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'9769", 'reported_seq': 9850, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:00.725998+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:00.725998+0000', 'last_peered': '2024-09-16T10:27:00.725998+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:00.725998+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9769, 'log_dups_size': 0, 'ondisk_log_size': 9769, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39998406, 'num_objects': 9767, 'num_object_clones': 0, 'num_object_copies': 19534, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9767, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9767, 'num_write_kb': 39062, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5600}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:05.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:05.260 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'10146", 'reported_seq': 10239, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:00.726015+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:00.726015+0000', 'last_peered': '2024-09-16T10:27:00.726015+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:00.726015+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41510796, 'num_objects': 10138, 'num_object_clones': 0, 'num_object_copies': 20276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10143, 'num_write_kb': 40543, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5743}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:05.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:05.565 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 233 20 161414 161394 2.70518 1.51172 0.0399663 0.0288724 2024-09-16T10:27:05.959 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:05.959 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/318986 objects degraded (13.460%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":314,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":159493,"data_bytes":653210121,"bytes_used":2143014912,"bytes_avail":287767277568,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":318986,"degraded_ratio":0.13460151856194316,"misplaced_objects":3206,"misplaced_total":318986,"misplaced_ratio":0.010050597831879771,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":52201,"recovering_keys_per_sec":0,"num_objects_recovered":153,"num_bytes_recovered":626688,"num_keys_recovered":0,"write_bytes_sec":1522725,"read_op_per_sec":0,"write_op_per_sec":371},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:05.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:05.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:06.554 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:06.557 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:06.565 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 234 20 161804 161784 2.70013 1.52344 0.0252753 0.0289258 2024-09-16T10:27:06.578 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-16T10:27:04.637933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653210121,"num_objects":159493,"num_object_clones":0,"num_object_copies":318986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3206,"num_objects_unfound":0,"num_objects_dirty":159493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137122,"num_write_kb":548370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5205,"num_bytes_recovered":21234166,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159158,"ondisk_log_size":159158,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2092788,"kb_used_data":1299868,"kb_used_omap":12,"kb_used_meta":792883,"kb_avail":281022732,"statfs":{"total":289910292480,"available":287767277568,"internally_reserved":0,"allocated":1331064832,"data_stored":1330235284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":811912194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,3,6,8,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18280448,"num_objects":4463,"num_object_clones":0,"num_object_copies":8926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-153,"num_objects_unfound":0,"num_objects_dirty":4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4463,"num_write_kb":17852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":626688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005084"},"pg_stats":[{"pgid":"2.f","version":"44'9927","reported_seq":11968,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:00.725770+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:00.725770+0000","last_peered":"2024-09-16T10:27:00.725770+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:00.725770+0000","last_undegraded":"2024-09-16T10:27:00.725770+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9927,"log_dups_size":0,"ondisk_log_size":9927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40660992,"num_objects":9927,"num_object_clones":0,"num_object_copies":19854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3206,"num_objects_unfound":0,"num_objects_dirty":9927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9927,"num_write_kb":39708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":986,"num_bytes_recovered":4038656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6721},{"shards":"0,2","objects":3206}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'9944","reported_seq":10101,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:00.947908+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:00.947908+0000","last_peered":"2024-09-16T10:27:00.947908+0000","last_clean":"2024-09-16T10:27:00.947908+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:00.947908+0000","last_undegraded":"2024-09-16T10:27:00.947908+0000","last_fullsized":"2024-09-16T10:27:00.947908+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":9944,"log_dups_size":0,"ondisk_log_size":9944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40706048,"num_objects":9938,"num_object_clones":0,"num_object_copies":19876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6157,"num_write_kb":24628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10041","reported_seq":10224,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:01.083134+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:01.083134+0000","last_peered":"2024-09-16T10:27:01.083134+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:01.083134+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41111552,"num_objects":10037,"num_object_clones":0,"num_object_copies":20074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6098,"num_write_kb":24392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5702}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'9937","reported_seq":9993,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:01.088579+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:01.088579+0000","last_peered":"2024-09-16T10:27:01.088579+0000","last_clean":"2024-09-16T10:27:01.088579+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:01.088579+0000","last_undegraded":"2024-09-16T10:27:01.088579+0000","last_fullsized":"2024-09-16T10:27:01.088579+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9937,"log_dups_size":0,"ondisk_log_size":9937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40701952,"num_objects":9937,"num_object_clones":0,"num_object_copies":19874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9937,"num_write_kb":39748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'9942","reported_seq":10099,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:00.749229+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:00.749229+0000","last_peered":"2024-09-16T10:27:00.749229+0000","last_clean":"2024-09-16T10:27:00.749229+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:00.749229+0000","last_undegraded":"2024-09-16T10:27:00.749229+0000","last_fullsized":"2024-09-16T10:27:00.749229+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":9942,"log_dups_size":0,"ondisk_log_size":9942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40693760,"num_objects":9935,"num_object_clones":0,"num_object_copies":19870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":26336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'9855","reported_seq":10015,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.582137+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:00.582137+0000","last_peered":"2024-09-16T10:27:00.582137+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:00.582137+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":9855,"log_dups_size":0,"ondisk_log_size":9855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40345600,"num_objects":9850,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6480,"num_write_kb":25920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5633},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10165","reported_seq":10338,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.656122+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:00.656122+0000","last_peered":"2024-09-16T10:27:00.656122+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:00.656122+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41611264,"num_objects":10159,"num_object_clones":0,"num_object_copies":20318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":25628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5756},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'9881","reported_seq":10056,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:00.671529+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:00.671529+0000","last_peered":"2024-09-16T10:27:00.671529+0000","last_clean":"2024-09-16T10:27:00.671529+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:00.671529+0000","last_undegraded":"2024-09-16T10:27:00.671529+0000","last_fullsized":"2024-09-16T10:27:00.671529+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40464384,"num_objects":9879,"num_object_clones":0,"num_object_copies":19758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5684,"num_write_kb":22736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10002","reported_seq":18505,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:04.055522+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:04.055522+0000","last_peered":"2024-09-16T10:27:04.055522+0000","last_clean":"2024-09-16T10:27:04.055522+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:04.055522+0000","last_undegraded":"2024-09-16T10:27:04.055522+0000","last_fullsized":"2024-09-16T10:27:04.055522+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40940486,"num_objects":9997,"num_object_clones":0,"num_object_copies":19994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":39984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10195","reported_seq":10270,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.717168+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:00.717168+0000","last_peered":"2024-09-16T10:27:00.717168+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:00.717168+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41726918,"num_objects":10189,"num_object_clones":0,"num_object_copies":20378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10193,"num_write_kb":40752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5755},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'9921","reported_seq":10026,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:01.074178+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:01.074178+0000","last_peered":"2024-09-16T10:27:01.074178+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:01.074178+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":9921,"log_dups_size":0,"ondisk_log_size":9921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40616902,"num_objects":9918,"num_object_clones":0,"num_object_copies":19836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":39667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5641}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'9986","reported_seq":10073,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:01.035564+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:01.035564+0000","last_peered":"2024-09-16T10:27:01.035564+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:01.035564+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9986,"log_dups_size":0,"ondisk_log_size":9986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40879046,"num_objects":9982,"num_object_clones":0,"num_object_copies":19964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9984,"num_write_kb":39923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9690","reported_seq":9769,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.602162+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:00.602162+0000","last_peered":"2024-09-16T10:27:00.602162+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:00.602162+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9690,"log_dups_size":0,"ondisk_log_size":9690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39682531,"num_objects":9689,"num_object_clones":0,"num_object_copies":19378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9689,"num_write_kb":38753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5530},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10154","reported_seq":10285,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.647076+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:00.647076+0000","last_peered":"2024-09-16T10:27:00.647076+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:00.647076+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41559465,"num_objects":10149,"num_object_clones":0,"num_object_copies":20298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10151,"num_write_kb":40588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5812},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":56,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.960132+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:00.960132+0000","last_peered":"2024-09-16T10:26:00.960132+0000","last_clean":"2024-09-16T10:26:00.960132+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:00.960132+0000","last_undegraded":"2024-09-16T10:26:00.960132+0000","last_fullsized":"2024-09-16T10:26:00.960132+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9769","reported_seq":9850,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.725998+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:00.725998+0000","last_peered":"2024-09-16T10:27:00.725998+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:00.725998+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39998406,"num_objects":9767,"num_object_clones":0,"num_object_copies":19534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9767,"num_write_kb":39062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5600},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10146","reported_seq":10239,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:00.726015+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:00.726015+0000","last_peered":"2024-09-16T10:27:00.726015+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:00.726015+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41510796,"num_objects":10138,"num_object_clones":0,"num_object_copies":20276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10143,"num_write_kb":40543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5743},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":653210102,"num_objects":159491,"num_object_clones":0,"num_object_copies":318982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3206,"num_objects_unfound":0,"num_objects_dirty":159491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":137120,"num_write_kb":548368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5204,"num_bytes_recovered":21234147,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1331908608,"data_stored":1331825509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159155,"ondisk_log_size":159155,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018934,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715560,"kb_used_data":455072,"kb_used_omap":4,"kb_used_meta":260475,"kb_avail":93656280,"statfs":{"total":96636764160,"available":95904030720,"internally_reserved":0,"allocated":465993728,"data_stored":465740094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":266727088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":4.2720000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":9.5640000000000001},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.76100000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507016,"kb_used_data":299008,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93864824,"statfs":{"total":96636764160,"available":96117579776,"internally_reserved":0,"allocated":306184192,"data_stored":305858298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":212987562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.1349999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":4.875},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":6.5919999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215162,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":870212,"kb_used_data":545788,"kb_used_omap":4,"kb_used_meta":324411,"kb_avail":93501628,"statfs":{"total":96636764160,"available":95745667072,"internally_reserved":0,"allocated":558886912,"data_stored":558636892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":332197544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,3,1,6,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":9.359},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":3.5649999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":10.919},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":9.3290000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":559280128,"data_stored":559276515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306176000,"data_stored":306100127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":466452480,"data_stored":466448867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:07.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 235 20 162246 162226 2.69599 1.72656 0.016752 0.0289709 2024-09-16T10:27:08.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 236 20 162695 162675 2.692 1.75391 0.111479 0.0290134 2024-09-16T10:27:09.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 237 20 163142 163122 2.68801 1.74609 0.0504943 0.029057 2024-09-16T10:27:09.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:10.095 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:10.098 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:10.126 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-16T10:27:08.639213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661520905,"num_objects":161522,"num_object_clones":0,"num_object_copies":323044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3127,"num_objects_unfound":0,"num_objects_dirty":161522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139151,"num_write_kb":556486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5284,"num_bytes_recovered":21557750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160487,"ondisk_log_size":160487,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2095964,"kb_used_data":1311236,"kb_used_omap":12,"kb_used_meta":784691,"kb_avail":281019556,"statfs":{"total":289910292480,"available":287764025344,"internally_reserved":0,"allocated":1342705664,"data_stored":1341876116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":803523586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,6,12,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17702912,"num_objects":4322,"num_object_clones":0,"num_object_copies":8644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-153,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4322,"num_write_kb":17288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":626688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004647"},"pg_stats":[{"pgid":"2.f","version":"44'10067","reported_seq":12266,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:05.804290+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:05.804290+0000","last_peered":"2024-09-16T10:27:05.804290+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:05.804290+0000","last_undegraded":"2024-09-16T10:27:05.804290+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41234432,"num_objects":10067,"num_object_clones":0,"num_object_copies":20134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3127,"num_objects_unfound":0,"num_objects_dirty":10067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10067,"num_write_kb":40268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":4362240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6940},{"shards":"0,2","objects":3127}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10074","reported_seq":10231,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:06.170036+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:06.170036+0000","last_peered":"2024-09-16T10:27:06.170036+0000","last_clean":"2024-09-16T10:27:06.170036+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:06.170036+0000","last_undegraded":"2024-09-16T10:27:06.170036+0000","last_fullsized":"2024-09-16T10:27:06.170036+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41238528,"num_objects":10068,"num_object_clones":0,"num_object_copies":20136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":25148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10177","reported_seq":10360,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:06.032708+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:06.032708+0000","last_peered":"2024-09-16T10:27:06.032708+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:06.032708+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41668608,"num_objects":10173,"num_object_clones":0,"num_object_copies":20346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6234,"num_write_kb":24936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5838}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10077","reported_seq":10133,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:06.101090+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:06.101090+0000","last_peered":"2024-09-16T10:27:06.101090+0000","last_clean":"2024-09-16T10:27:06.101090+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:06.101090+0000","last_undegraded":"2024-09-16T10:27:06.101090+0000","last_fullsized":"2024-09-16T10:27:06.101090+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41275392,"num_objects":10077,"num_object_clones":0,"num_object_copies":20154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10077,"num_write_kb":40308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10082","reported_seq":10239,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:05.621970+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:05.621970+0000","last_peered":"2024-09-16T10:27:05.621970+0000","last_clean":"2024-09-16T10:27:05.621970+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:05.621970+0000","last_undegraded":"2024-09-16T10:27:05.621970+0000","last_fullsized":"2024-09-16T10:27:05.621970+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41267200,"num_objects":10075,"num_object_clones":0,"num_object_copies":20150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10008","reported_seq":10171,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.705380+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:05.705380+0000","last_peered":"2024-09-16T10:27:05.705380+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:05.705380+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40972288,"num_objects":10003,"num_object_clones":0,"num_object_copies":20006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":26532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5786},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10289","reported_seq":10461,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.744133+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:05.744133+0000","last_peered":"2024-09-16T10:27:05.744133+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:05.744133+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42119168,"num_objects":10283,"num_object_clones":0,"num_object_copies":20566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6531,"num_write_kb":26124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5880},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10019","reported_seq":10194,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:05.746494+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:05.746494+0000","last_peered":"2024-09-16T10:27:05.746494+0000","last_clean":"2024-09-16T10:27:05.746494+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:05.746494+0000","last_undegraded":"2024-09-16T10:27:05.746494+0000","last_fullsized":"2024-09-16T10:27:05.746494+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41029632,"num_objects":10017,"num_object_clones":0,"num_object_copies":20034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10002","reported_seq":18505,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:04.055522+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:04.055522+0000","last_peered":"2024-09-16T10:27:04.055522+0000","last_clean":"2024-09-16T10:27:04.055522+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:04.055522+0000","last_undegraded":"2024-09-16T10:27:04.055522+0000","last_fullsized":"2024-09-16T10:27:04.055522+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40940486,"num_objects":9997,"num_object_clones":0,"num_object_copies":19994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":39984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10337","reported_seq":10416,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.553311+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:05.553311+0000","last_peered":"2024-09-16T10:27:05.553311+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:05.553311+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42308550,"num_objects":10331,"num_object_clones":0,"num_object_copies":20662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":41320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5897},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10056","reported_seq":10162,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:06.111633+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:06.111633+0000","last_peered":"2024-09-16T10:27:06.111633+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:06.111633+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41169862,"num_objects":10053,"num_object_clones":0,"num_object_copies":20106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":40207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10127","reported_seq":10214,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.974983+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:05.974983+0000","last_peered":"2024-09-16T10:27:05.974983+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:05.974983+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41456582,"num_objects":10123,"num_object_clones":0,"num_object_copies":20246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10125,"num_write_kb":40487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5931}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9821","reported_seq":9900,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.719166+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:05.719166+0000","last_peered":"2024-09-16T10:27:05.719166+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:05.719166+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40219107,"num_objects":9820,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9820,"num_write_kb":39277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5661},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10284","reported_seq":10415,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.695184+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:05.695184+0000","last_peered":"2024-09-16T10:27:05.695184+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:05.695184+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42091945,"num_objects":10279,"num_object_clones":0,"num_object_copies":20558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10281,"num_write_kb":41108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5942},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9891","reported_seq":9972,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.750360+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:05.750360+0000","last_peered":"2024-09-16T10:27:05.750360+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:05.750360+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40498118,"num_objects":9889,"num_object_clones":0,"num_object_copies":19778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9889,"num_write_kb":39550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5722},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10273","reported_seq":10366,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.679614+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:05.679614+0000","last_peered":"2024-09-16T10:27:05.679614+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:05.679614+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42030988,"num_objects":10265,"num_object_clones":0,"num_object_copies":20530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10270,"num_write_kb":41051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5870},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":661520886,"num_objects":161520,"num_object_clones":0,"num_object_copies":323040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3127,"num_objects_unfound":0,"num_objects_dirty":161520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139149,"num_write_kb":556484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5283,"num_bytes_recovered":21557731,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1342869504,"data_stored":1342786405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160484,"ondisk_log_size":160484,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018935,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708312,"kb_used_data":460112,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93663528,"statfs":{"total":96636764160,"available":95911452672,"internally_reserved":0,"allocated":471154688,"data_stored":470901054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":254144176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.9530000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.998}]}]},{"osd":1,"up_from":40,"seq":171798691879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":507016,"kb_used_data":299008,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93864824,"statfs":{"total":96636764160,"available":96117579776,"internally_reserved":0,"allocated":306184192,"data_stored":305858298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":212987562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.1349999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.40200000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":4.875},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":6.5919999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215163,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":880636,"kb_used_data":552116,"kb_used_omap":4,"kb_used_meta":328507,"kb_avail":93491204,"statfs":{"total":96636764160,"available":95734992896,"internally_reserved":0,"allocated":565366784,"data_stored":565116764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":336391848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":10.762},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":7.3419999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":10.382999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":565841920,"data_stored":565838307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306176000,"data_stored":306100127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":470851584,"data_stored":470847971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:10.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:10.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 238 20 163532 163512 2.68311 1.52344 0.067789 0.0291106 2024-09-16T10:27:10.710 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:10.713 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:10.730 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T10:27:10.639874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662086153,"num_objects":161660,"num_object_clones":0,"num_object_copies":323320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3127,"num_objects_unfound":0,"num_objects_dirty":161660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139289,"num_write_kb":557038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5284,"num_bytes_recovered":21557750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160525,"ondisk_log_size":160525,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2106844,"kb_used_data":1318020,"kb_used_omap":12,"kb_used_meta":788787,"kb_avail":281008676,"statfs":{"total":289910292480,"available":287752884224,"internally_reserved":0,"allocated":1349652480,"data_stored":1348822932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":807717890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,6,13,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18268160,"num_objects":4460,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-153,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":17840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":626688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004520"},"pg_stats":[{"pgid":"2.f","version":"44'10067","reported_seq":12266,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:05.804290+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:05.804290+0000","last_peered":"2024-09-16T10:27:05.804290+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:05.804290+0000","last_undegraded":"2024-09-16T10:27:05.804290+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41234432,"num_objects":10067,"num_object_clones":0,"num_object_copies":20134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3127,"num_objects_unfound":0,"num_objects_dirty":10067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10067,"num_write_kb":40268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":4362240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6940},{"shards":"0,2","objects":3127}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10074","reported_seq":10231,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:06.170036+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:06.170036+0000","last_peered":"2024-09-16T10:27:06.170036+0000","last_clean":"2024-09-16T10:27:06.170036+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:06.170036+0000","last_undegraded":"2024-09-16T10:27:06.170036+0000","last_fullsized":"2024-09-16T10:27:06.170036+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41238528,"num_objects":10068,"num_object_clones":0,"num_object_copies":20136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":25148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10177","reported_seq":10360,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:06.032708+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:06.032708+0000","last_peered":"2024-09-16T10:27:06.032708+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:06.032708+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41668608,"num_objects":10173,"num_object_clones":0,"num_object_copies":20346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6234,"num_write_kb":24936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5838}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10077","reported_seq":10133,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:06.101090+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:06.101090+0000","last_peered":"2024-09-16T10:27:06.101090+0000","last_clean":"2024-09-16T10:27:06.101090+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:06.101090+0000","last_undegraded":"2024-09-16T10:27:06.101090+0000","last_fullsized":"2024-09-16T10:27:06.101090+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41275392,"num_objects":10077,"num_object_clones":0,"num_object_copies":20154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10077,"num_write_kb":40308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10082","reported_seq":10239,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:05.621970+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:05.621970+0000","last_peered":"2024-09-16T10:27:05.621970+0000","last_clean":"2024-09-16T10:27:05.621970+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:05.621970+0000","last_undegraded":"2024-09-16T10:27:05.621970+0000","last_fullsized":"2024-09-16T10:27:05.621970+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41267200,"num_objects":10075,"num_object_clones":0,"num_object_copies":20150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10008","reported_seq":10171,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.705380+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:05.705380+0000","last_peered":"2024-09-16T10:27:05.705380+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:05.705380+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40972288,"num_objects":10003,"num_object_clones":0,"num_object_copies":20006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":26532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5786},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10289","reported_seq":10461,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.744133+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:05.744133+0000","last_peered":"2024-09-16T10:27:05.744133+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:05.744133+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42119168,"num_objects":10283,"num_object_clones":0,"num_object_copies":20566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6531,"num_write_kb":26124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5880},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10019","reported_seq":10194,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:05.746494+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:05.746494+0000","last_peered":"2024-09-16T10:27:05.746494+0000","last_clean":"2024-09-16T10:27:05.746494+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:05.746494+0000","last_undegraded":"2024-09-16T10:27:05.746494+0000","last_fullsized":"2024-09-16T10:27:05.746494+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41029632,"num_objects":10017,"num_object_clones":0,"num_object_copies":20034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10140","reported_seq":18642,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:09.147927+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:09.147927+0000","last_peered":"2024-09-16T10:27:09.147927+0000","last_clean":"2024-09-16T10:27:09.147927+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:09.147927+0000","last_undegraded":"2024-09-16T10:27:09.147927+0000","last_fullsized":"2024-09-16T10:27:09.147927+0000","mapping_epoch":43,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41505734,"num_objects":10135,"num_object_clones":0,"num_object_copies":20270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10138,"num_write_kb":40536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10337","reported_seq":10416,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.553311+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:05.553311+0000","last_peered":"2024-09-16T10:27:05.553311+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:05.553311+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42308550,"num_objects":10331,"num_object_clones":0,"num_object_copies":20662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":41320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5897},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10056","reported_seq":10162,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:06.111633+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:06.111633+0000","last_peered":"2024-09-16T10:27:06.111633+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:06.111633+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41169862,"num_objects":10053,"num_object_clones":0,"num_object_copies":20106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":40207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10127","reported_seq":10214,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.974983+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:05.974983+0000","last_peered":"2024-09-16T10:27:05.974983+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:05.974983+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41456582,"num_objects":10123,"num_object_clones":0,"num_object_copies":20246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10125,"num_write_kb":40487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5931}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9821","reported_seq":9900,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.719166+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:05.719166+0000","last_peered":"2024-09-16T10:27:05.719166+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:05.719166+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40219107,"num_objects":9820,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9820,"num_write_kb":39277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5661},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10284","reported_seq":10415,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.695184+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:05.695184+0000","last_peered":"2024-09-16T10:27:05.695184+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:05.695184+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42091945,"num_objects":10279,"num_object_clones":0,"num_object_copies":20558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10281,"num_write_kb":41108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5942},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9891","reported_seq":9972,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.750360+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:05.750360+0000","last_peered":"2024-09-16T10:27:05.750360+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:05.750360+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40498118,"num_objects":9889,"num_object_clones":0,"num_object_copies":19778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9889,"num_write_kb":39550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5722},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10273","reported_seq":10366,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.679614+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:05.679614+0000","last_peered":"2024-09-16T10:27:05.679614+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:05.679614+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42030988,"num_objects":10265,"num_object_clones":0,"num_object_copies":20530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10270,"num_write_kb":41051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5870},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":662086134,"num_objects":161658,"num_object_clones":0,"num_object_copies":323316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3127,"num_objects_unfound":0,"num_objects_dirty":161658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139287,"num_write_kb":557036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5283,"num_bytes_recovered":21557731,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1349722112,"data_stored":1349639013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160522,"ondisk_log_size":160522,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018935,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708312,"kb_used_data":460112,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93663528,"statfs":{"total":96636764160,"available":95911452672,"internally_reserved":0,"allocated":471154688,"data_stored":470901054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":254144176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.9530000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.998}]}]},{"osd":1,"up_from":40,"seq":171798691880,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517896,"kb_used_data":305792,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93853944,"statfs":{"total":96636764160,"available":96106438656,"internally_reserved":0,"allocated":313131008,"data_stored":312805114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":217181866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.622},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":10.708},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":10.664999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215163,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":880636,"kb_used_data":552116,"kb_used_omap":4,"kb_used_meta":328507,"kb_avail":93491204,"statfs":{"total":96636764160,"available":95734992896,"internally_reserved":0,"allocated":565366784,"data_stored":565116764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":336391848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":10.762},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":7.3419999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":10.382999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":565841920,"data_stored":565838307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313028608,"data_stored":312952735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":470851584,"data_stored":470847971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:10.732 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:10.732 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'10067", 'reported_seq': 12266, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:05.804290+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:05.804290+0000', 'last_peered': '2024-09-16T10:27:05.804290+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:05.804290+0000', 'last_undegraded': '2024-09-16T10:27:05.804290+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41234432, 'num_objects': 10067, 'num_object_clones': 0, 'num_object_copies': 20134, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3127, 'num_objects_unfound': 0, 'num_objects_dirty': 10067, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10067, 'num_write_kb': 40268, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1065, 'num_bytes_recovered': 4362240, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6940}, {'shards': '0,2', 'objects': 3127}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:10.732 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:10.732 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'10177", 'reported_seq': 10360, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:06.032708+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:06.032708+0000', 'last_peered': '2024-09-16T10:27:06.032708+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:06.032708+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41668608, 'num_objects': 10173, 'num_object_clones': 0, 'num_object_copies': 20346, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6234, 'num_write_kb': 24936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 5838}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:10.732 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:10.732 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'10008", 'reported_seq': 10171, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:05.705380+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:05.705380+0000', 'last_peered': '2024-09-16T10:27:05.705380+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:05.705380+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40972288, 'num_objects': 10003, 'num_object_clones': 0, 'num_object_copies': 20006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6633, 'num_write_kb': 26532, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5786}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:10.732 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:10.732 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'10289", 'reported_seq': 10461, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:05.744133+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:05.744133+0000', 'last_peered': '2024-09-16T10:27:05.744133+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:05.744133+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'200", 'ondisk_log_start': "35'200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42119168, 'num_objects': 10283, 'num_object_clones': 0, 'num_object_copies': 20566, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10283, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6531, 'num_write_kb': 26124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5880}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:10.733 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:10.733 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'10337", 'reported_seq': 10416, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:05.553311+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:05.553311+0000', 'last_peered': '2024-09-16T10:27:05.553311+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:05.553311+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'300", 'ondisk_log_start': "35'300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42308550, 'num_objects': 10331, 'num_object_clones': 0, 'num_object_copies': 20662, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10335, 'num_write_kb': 41320, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5897}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:10.733 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:10.733 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'10056", 'reported_seq': 10162, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:06.111633+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:06.111633+0000', 'last_peered': '2024-09-16T10:27:06.111633+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:06.111633+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41169862, 'num_objects': 10053, 'num_object_clones': 0, 'num_object_copies': 20106, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10054, 'num_write_kb': 40207, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 5776}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:10.733 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:10.733 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'10127", 'reported_seq': 10214, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:05.974983+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:05.974983+0000', 'last_peered': '2024-09-16T10:27:05.974983+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:05.974983+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41456582, 'num_objects': 10123, 'num_object_clones': 0, 'num_object_copies': 20246, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10125, 'num_write_kb': 40487, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 5931}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:10.733 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:10.733 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'9821", 'reported_seq': 9900, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:05.719166+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:05.719166+0000', 'last_peered': '2024-09-16T10:27:05.719166+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:05.719166+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9821, 'log_dups_size': 0, 'ondisk_log_size': 9821, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40219107, 'num_objects': 9820, 'num_object_clones': 0, 'num_object_copies': 19640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9820, 'num_write_kb': 39277, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5661}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:10.733 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:10.733 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'10284", 'reported_seq': 10415, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:05.695184+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:05.695184+0000', 'last_peered': '2024-09-16T10:27:05.695184+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:05.695184+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'200", 'ondisk_log_start': "35'200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42091945, 'num_objects': 10279, 'num_object_clones': 0, 'num_object_copies': 20558, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10281, 'num_write_kb': 41108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5942}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:10.734 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:10.734 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'9891", 'reported_seq': 9972, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:05.750360+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:05.750360+0000', 'last_peered': '2024-09-16T10:27:05.750360+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:05.750360+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9891, 'log_dups_size': 0, 'ondisk_log_size': 9891, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40498118, 'num_objects': 9889, 'num_object_clones': 0, 'num_object_copies': 19778, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9889, 'num_write_kb': 39550, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5722}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:10.734 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:10.734 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'10273", 'reported_seq': 10366, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:05.679614+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:05.679614+0000', 'last_peered': '2024-09-16T10:27:05.679614+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:05.679614+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'200", 'ondisk_log_start': "35'200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42030988, 'num_objects': 10265, 'num_object_clones': 0, 'num_object_copies': 20530, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10270, 'num_write_kb': 41051, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5870}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:10.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:11.428 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42936/323320 objects degraded (13.280%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":320,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":161660,"data_bytes":662086153,"bytes_used":2157408256,"bytes_avail":287752884224,"bytes_total":289910292480,"degraded_objects":42936,"degraded_total":323320,"degraded_ratio":0.13279722875170111,"misplaced_objects":3127,"misplaced_total":323320,"misplaced_ratio":0.0096715328467153281,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":52204,"recovering_keys_per_sec":0,"num_objects_recovered":153,"num_bytes_recovered":626688,"num_keys_recovered":0,"write_bytes_sec":1521773,"read_op_per_sec":0,"write_op_per_sec":371},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:11.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:11.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:11.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout: 239 20 163921 163901 2.67825 1.51953 0.0749125 0.0291618 2024-09-16T10:27:12.003 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:12.005 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:12.026 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-16T10:27:10.639874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662086153,"num_objects":161660,"num_object_clones":0,"num_object_copies":323320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3127,"num_objects_unfound":0,"num_objects_dirty":161660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139289,"num_write_kb":557038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5284,"num_bytes_recovered":21557750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160525,"ondisk_log_size":160525,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2106844,"kb_used_data":1318020,"kb_used_omap":12,"kb_used_meta":788787,"kb_avail":281008676,"statfs":{"total":289910292480,"available":287752884224,"internally_reserved":0,"allocated":1349652480,"data_stored":1348822932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":807717890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,4,6,13,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18268160,"num_objects":4460,"num_object_clones":0,"num_object_copies":8920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-153,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":17840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":626688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004520"},"pg_stats":[{"pgid":"2.f","version":"44'10067","reported_seq":12266,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:05.804290+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:05.804290+0000","last_peered":"2024-09-16T10:27:05.804290+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:05.804290+0000","last_undegraded":"2024-09-16T10:27:05.804290+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41234432,"num_objects":10067,"num_object_clones":0,"num_object_copies":20134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3127,"num_objects_unfound":0,"num_objects_dirty":10067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10067,"num_write_kb":40268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1065,"num_bytes_recovered":4362240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6940},{"shards":"0,2","objects":3127}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10074","reported_seq":10231,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:06.170036+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:06.170036+0000","last_peered":"2024-09-16T10:27:06.170036+0000","last_clean":"2024-09-16T10:27:06.170036+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:06.170036+0000","last_undegraded":"2024-09-16T10:27:06.170036+0000","last_fullsized":"2024-09-16T10:27:06.170036+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41238528,"num_objects":10068,"num_object_clones":0,"num_object_copies":20136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":25148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10177","reported_seq":10360,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:06.032708+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:06.032708+0000","last_peered":"2024-09-16T10:27:06.032708+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:06.032708+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41668608,"num_objects":10173,"num_object_clones":0,"num_object_copies":20346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6234,"num_write_kb":24936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5838}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10077","reported_seq":10133,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:06.101090+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:06.101090+0000","last_peered":"2024-09-16T10:27:06.101090+0000","last_clean":"2024-09-16T10:27:06.101090+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:06.101090+0000","last_undegraded":"2024-09-16T10:27:06.101090+0000","last_fullsized":"2024-09-16T10:27:06.101090+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41275392,"num_objects":10077,"num_object_clones":0,"num_object_copies":20154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10077,"num_write_kb":40308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10082","reported_seq":10239,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:05.621970+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:05.621970+0000","last_peered":"2024-09-16T10:27:05.621970+0000","last_clean":"2024-09-16T10:27:05.621970+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:05.621970+0000","last_undegraded":"2024-09-16T10:27:05.621970+0000","last_fullsized":"2024-09-16T10:27:05.621970+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41267200,"num_objects":10075,"num_object_clones":0,"num_object_copies":20150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10008","reported_seq":10171,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.705380+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:05.705380+0000","last_peered":"2024-09-16T10:27:05.705380+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:05.705380+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40972288,"num_objects":10003,"num_object_clones":0,"num_object_copies":20006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":26532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5786},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10289","reported_seq":10461,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.744133+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:05.744133+0000","last_peered":"2024-09-16T10:27:05.744133+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:05.744133+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42119168,"num_objects":10283,"num_object_clones":0,"num_object_copies":20566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6531,"num_write_kb":26124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5880},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10019","reported_seq":10194,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:05.746494+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:05.746494+0000","last_peered":"2024-09-16T10:27:05.746494+0000","last_clean":"2024-09-16T10:27:05.746494+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:05.746494+0000","last_undegraded":"2024-09-16T10:27:05.746494+0000","last_fullsized":"2024-09-16T10:27:05.746494+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41029632,"num_objects":10017,"num_object_clones":0,"num_object_copies":20034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5822,"num_write_kb":23288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10140","reported_seq":18642,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:09.147927+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:09.147927+0000","last_peered":"2024-09-16T10:27:09.147927+0000","last_clean":"2024-09-16T10:27:09.147927+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:09.147927+0000","last_undegraded":"2024-09-16T10:27:09.147927+0000","last_fullsized":"2024-09-16T10:27:09.147927+0000","mapping_epoch":43,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41505734,"num_objects":10135,"num_object_clones":0,"num_object_copies":20270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10138,"num_write_kb":40536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10337","reported_seq":10416,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.553311+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:05.553311+0000","last_peered":"2024-09-16T10:27:05.553311+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:05.553311+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42308550,"num_objects":10331,"num_object_clones":0,"num_object_copies":20662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10335,"num_write_kb":41320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5897},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10056","reported_seq":10162,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:06.111633+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:06.111633+0000","last_peered":"2024-09-16T10:27:06.111633+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:06.111633+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41169862,"num_objects":10053,"num_object_clones":0,"num_object_copies":20106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":40207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10127","reported_seq":10214,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.974983+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:05.974983+0000","last_peered":"2024-09-16T10:27:05.974983+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:05.974983+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41456582,"num_objects":10123,"num_object_clones":0,"num_object_copies":20246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10125,"num_write_kb":40487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":5931}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9821","reported_seq":9900,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.719166+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:05.719166+0000","last_peered":"2024-09-16T10:27:05.719166+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:05.719166+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9821,"log_dups_size":0,"ondisk_log_size":9821,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40219107,"num_objects":9820,"num_object_clones":0,"num_object_copies":19640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9820,"num_write_kb":39277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5661},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10284","reported_seq":10415,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.695184+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:05.695184+0000","last_peered":"2024-09-16T10:27:05.695184+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:05.695184+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42091945,"num_objects":10279,"num_object_clones":0,"num_object_copies":20558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10281,"num_write_kb":41108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5942},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'9891","reported_seq":9972,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.750360+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:05.750360+0000","last_peered":"2024-09-16T10:27:05.750360+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:05.750360+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9891,"log_dups_size":0,"ondisk_log_size":9891,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40498118,"num_objects":9889,"num_object_clones":0,"num_object_copies":19778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9889,"num_write_kb":39550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5722},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10273","reported_seq":10366,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:05.679614+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:05.679614+0000","last_peered":"2024-09-16T10:27:05.679614+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:05.679614+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42030988,"num_objects":10265,"num_object_clones":0,"num_object_copies":20530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10270,"num_write_kb":41051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5870},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":662086134,"num_objects":161658,"num_object_clones":0,"num_object_copies":323316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3127,"num_objects_unfound":0,"num_objects_dirty":161658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":139287,"num_write_kb":557036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5283,"num_bytes_recovered":21557731,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1349722112,"data_stored":1349639013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160522,"ondisk_log_size":160522,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018935,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708312,"kb_used_data":460112,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93663528,"statfs":{"total":96636764160,"available":95911452672,"internally_reserved":0,"allocated":471154688,"data_stored":470901054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":254144176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.9530000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.998}]}]},{"osd":1,"up_from":40,"seq":171798691880,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517896,"kb_used_data":305792,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93853944,"statfs":{"total":96636764160,"available":96106438656,"internally_reserved":0,"allocated":313131008,"data_stored":312805114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":217181866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.622},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":10.708},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":10.664999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215163,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":880636,"kb_used_data":552116,"kb_used_omap":4,"kb_used_meta":328507,"kb_avail":93491204,"statfs":{"total":96636764160,"available":95734992896,"internally_reserved":0,"allocated":565366784,"data_stored":565116764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":336391848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":10.762},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":7.3419999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":10.382999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":565841920,"data_stored":565838307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313028608,"data_stored":312952735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":470851584,"data_stored":470847971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:12.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Total time run: 240.026 2024-09-16T10:27:12.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Total writes made: 164319 2024-09-16T10:27:12.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Write size: 4096 2024-09-16T10:27:12.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Object size: 4096 2024-09-16T10:27:12.566 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Bandwidth (MB/sec): 2.67417 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Stddev Bandwidth: 2.09098 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Max bandwidth (MB/sec): 13.3008 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Min bandwidth (MB/sec): 0 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Average IOPS: 684 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Stddev IOPS: 535.297 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Max IOPS: 3405 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Min IOPS: 0 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Average Latency(s): 0.0292092 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Stddev Latency(s): 0.0436447 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Max latency(s): 2.61294 2024-09-16T10:27:12.567 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Min latency(s): 0.00139595 2024-09-16T10:27:12.578 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Cleaning up (deleting benchmark objects) 2024-09-16T10:27:15.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:15.605 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:15.606 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:15.627 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-16T10:27:14.641798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670421513,"num_objects":163695,"num_object_clones":0,"num_object_copies":327390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3054,"num_objects_unfound":0,"num_objects_dirty":163695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141430,"num_write_kb":565390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5357,"num_bytes_recovered":21856758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160566,"ondisk_log_size":160566,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2135680,"kb_used_data":1332904,"kb_used_omap":12,"kb_used_meta":802739,"kb_avail":280979840,"statfs":{"total":289910292480,"available":287723356160,"internally_reserved":0,"allocated":1364893696,"data_stored":1364060076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":822004738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,5,2,3,9,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17715200,"num_objects":4325,"num_object_clones":0,"num_object_copies":8650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-152,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4431,"num_write_kb":17512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004629"},"pg_stats":[{"pgid":"2.f","version":"44'10191","reported_seq":12537,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:10.860369+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:10.860369+0000","last_peered":"2024-09-16T10:27:10.860369+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:10.860369+0000","last_undegraded":"2024-09-16T10:27:10.860369+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41742336,"num_objects":10191,"num_object_clones":0,"num_object_copies":20382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3054,"num_objects_unfound":0,"num_objects_dirty":10191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10191,"num_write_kb":40764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":4661248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7137},{"shards":"0,2","objects":3054}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10208","reported_seq":10365,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:11.005264+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:11.005264+0000","last_peered":"2024-09-16T10:27:11.005264+0000","last_clean":"2024-09-16T10:27:11.005264+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:11.005264+0000","last_undegraded":"2024-09-16T10:27:11.005264+0000","last_fullsized":"2024-09-16T10:27:11.005264+0000","mapping_epoch":34,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41787392,"num_objects":10202,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6421,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10301","reported_seq":10486,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:11.082669+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:11.082669+0000","last_peered":"2024-09-16T10:27:11.082669+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:11.082669+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42176512,"num_objects":10297,"num_object_clones":0,"num_object_copies":20594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6358,"num_write_kb":25432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5962}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10204","reported_seq":10257,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:11.118319+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:11.118319+0000","last_peered":"2024-09-16T10:27:11.118319+0000","last_clean":"2024-09-16T10:27:11.118319+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:11.118319+0000","last_undegraded":"2024-09-16T10:27:11.118319+0000","last_fullsized":"2024-09-16T10:27:11.118319+0000","mapping_epoch":34,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41795584,"num_objects":10204,"num_object_clones":0,"num_object_copies":20408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":40816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10212","reported_seq":10367,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:10.829508+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:10.829508+0000","last_peered":"2024-09-16T10:27:10.829508+0000","last_clean":"2024-09-16T10:27:10.829508+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:10.829508+0000","last_undegraded":"2024-09-16T10:27:10.829508+0000","last_fullsized":"2024-09-16T10:27:10.829508+0000","mapping_epoch":33,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41799680,"num_objects":10205,"num_object_clones":0,"num_object_copies":20410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6854,"num_write_kb":27416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10144","reported_seq":10308,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.819737+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:10.819737+0000","last_peered":"2024-09-16T10:27:10.819737+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:10.819737+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41529344,"num_objects":10139,"num_object_clones":0,"num_object_copies":20278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6769,"num_write_kb":27076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5922},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10424","reported_seq":10598,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.836211+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:10.836211+0000","last_peered":"2024-09-16T10:27:10.836211+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:10.836211+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42672128,"num_objects":10418,"num_object_clones":0,"num_object_copies":20836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":26664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6015},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10154","reported_seq":10329,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:10.686880+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:10.686880+0000","last_peered":"2024-09-16T10:27:10.686880+0000","last_clean":"2024-09-16T10:27:10.686880+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:10.686880+0000","last_undegraded":"2024-09-16T10:27:10.686880+0000","last_fullsized":"2024-09-16T10:27:10.686880+0000","mapping_epoch":33,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41582592,"num_objects":10152,"num_object_clones":0,"num_object_copies":20304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":23828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10287","reported_seq":18790,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:14.173678+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:14.173678+0000","last_peered":"2024-09-16T10:27:14.173678+0000","last_clean":"2024-09-16T10:27:14.173678+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:14.173678+0000","last_undegraded":"2024-09-16T10:27:14.173678+0000","last_fullsized":"2024-09-16T10:27:14.173678+0000","mapping_epoch":43,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41673670,"num_objects":10176,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10285,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10474","reported_seq":10553,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.761490+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:10.761490+0000","last_peered":"2024-09-16T10:27:10.761490+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:10.761490+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42869702,"num_objects":10468,"num_object_clones":0,"num_object_copies":20936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10472,"num_write_kb":41868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6034},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10187","reported_seq":10293,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:11.171583+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:11.171583+0000","last_peered":"2024-09-16T10:27:11.171583+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:11.171583+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41706438,"num_objects":10184,"num_object_clones":0,"num_object_copies":20368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":40731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10254","reported_seq":10341,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:11.185863+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:11.185863+0000","last_peered":"2024-09-16T10:27:11.185863+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:11.185863+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41976774,"num_objects":10250,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10252,"num_write_kb":40995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":6058}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9965","reported_seq":10044,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.805316+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:10.805316+0000","last_peered":"2024-09-16T10:27:10.805316+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:10.805316+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9965,"log_dups_size":0,"ondisk_log_size":9965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40808931,"num_objects":9964,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9964,"num_write_kb":39853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5805},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10417","reported_seq":10550,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.836973+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:10.836973+0000","last_peered":"2024-09-16T10:27:10.836973+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:10.836973+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42636713,"num_objects":10412,"num_object_clones":0,"num_object_copies":20824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":41640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6075},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10036","reported_seq":10117,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.751734+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:10.751734+0000","last_peered":"2024-09-16T10:27:10.751734+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:10.751734+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41092038,"num_objects":10034,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10034,"num_write_kb":40130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5867},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10405","reported_seq":10498,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.777786+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:10.777786+0000","last_peered":"2024-09-16T10:27:10.777786+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:10.777786+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42571660,"num_objects":10397,"num_object_clones":0,"num_object_copies":20794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10402,"num_write_kb":41579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6002},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":670421494,"num_objects":163693,"num_object_clones":0,"num_object_copies":327386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3054,"num_objects_unfound":0,"num_objects_dirty":163693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141428,"num_write_kb":565388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5356,"num_bytes_recovered":21856739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1364811776,"data_stored":1364724605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018936,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712516,"kb_used_data":464316,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93659324,"statfs":{"total":96636764160,"available":95907147776,"internally_reserved":0,"allocated":475459584,"data_stored":475205950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":254144176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.6079999999999997},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":3.8610000000000002},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":2.1629999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530408,"kb_used_data":310112,"kb_used_omap":4,"kb_used_meta":220283,"kb_avail":93841432,"statfs":{"total":96636764160,"available":96093626368,"internally_reserved":0,"allocated":317554688,"data_stored":317224722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":225570474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.96299999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":13.579000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":5.6230000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215164,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":892756,"kb_used_data":558476,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93479084,"statfs":{"total":96636764160,"available":95722582016,"internally_reserved":0,"allocated":571879424,"data_stored":571629404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":3.9470000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":7.7409999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":2.1760000000000002},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.399}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":572510208,"data_stored":572506595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":317165568,"data_stored":317085623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":475136000,"data_stored":475132387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:15.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:16.189 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:16.190 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:16.211 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-16T10:27:14.641798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670421513,"num_objects":163695,"num_object_clones":0,"num_object_copies":327390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3054,"num_objects_unfound":0,"num_objects_dirty":163695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141430,"num_write_kb":565390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5357,"num_bytes_recovered":21856758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160566,"ondisk_log_size":160566,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2135680,"kb_used_data":1332904,"kb_used_omap":12,"kb_used_meta":802739,"kb_avail":280979840,"statfs":{"total":289910292480,"available":287723356160,"internally_reserved":0,"allocated":1364893696,"data_stored":1364060076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":822004738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,5,2,3,9,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17715200,"num_objects":4325,"num_object_clones":0,"num_object_copies":8650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-152,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4431,"num_write_kb":17512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":622592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004629"},"pg_stats":[{"pgid":"2.f","version":"44'10191","reported_seq":12537,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:10.860369+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:10.860369+0000","last_peered":"2024-09-16T10:27:10.860369+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:10.860369+0000","last_undegraded":"2024-09-16T10:27:10.860369+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41742336,"num_objects":10191,"num_object_clones":0,"num_object_copies":20382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3054,"num_objects_unfound":0,"num_objects_dirty":10191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10191,"num_write_kb":40764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":4661248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7137},{"shards":"0,2","objects":3054}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10208","reported_seq":10365,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:11.005264+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:11.005264+0000","last_peered":"2024-09-16T10:27:11.005264+0000","last_clean":"2024-09-16T10:27:11.005264+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:11.005264+0000","last_undegraded":"2024-09-16T10:27:11.005264+0000","last_fullsized":"2024-09-16T10:27:11.005264+0000","mapping_epoch":34,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41787392,"num_objects":10202,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6421,"num_write_kb":25684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10301","reported_seq":10486,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:11.082669+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:11.082669+0000","last_peered":"2024-09-16T10:27:11.082669+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:11.082669+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42176512,"num_objects":10297,"num_object_clones":0,"num_object_copies":20594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6358,"num_write_kb":25432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4335},{"shards":"2","objects":5962}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10204","reported_seq":10257,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:11.118319+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:11.118319+0000","last_peered":"2024-09-16T10:27:11.118319+0000","last_clean":"2024-09-16T10:27:11.118319+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:11.118319+0000","last_undegraded":"2024-09-16T10:27:11.118319+0000","last_fullsized":"2024-09-16T10:27:11.118319+0000","mapping_epoch":34,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41795584,"num_objects":10204,"num_object_clones":0,"num_object_copies":20408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10204,"num_write_kb":40816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10212","reported_seq":10367,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:10.829508+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:10.829508+0000","last_peered":"2024-09-16T10:27:10.829508+0000","last_clean":"2024-09-16T10:27:10.829508+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:10.829508+0000","last_undegraded":"2024-09-16T10:27:10.829508+0000","last_fullsized":"2024-09-16T10:27:10.829508+0000","mapping_epoch":33,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41799680,"num_objects":10205,"num_object_clones":0,"num_object_copies":20410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6854,"num_write_kb":27416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10144","reported_seq":10308,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.819737+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:10.819737+0000","last_peered":"2024-09-16T10:27:10.819737+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:10.819737+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41529344,"num_objects":10139,"num_object_clones":0,"num_object_copies":20278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6769,"num_write_kb":27076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5922},{"shards":"0,2","objects":4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10424","reported_seq":10598,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.836211+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:10.836211+0000","last_peered":"2024-09-16T10:27:10.836211+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:10.836211+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42672128,"num_objects":10418,"num_object_clones":0,"num_object_copies":20836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6666,"num_write_kb":26664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6015},{"shards":"0,2","objects":4403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10154","reported_seq":10329,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:10.686880+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:10.686880+0000","last_peered":"2024-09-16T10:27:10.686880+0000","last_clean":"2024-09-16T10:27:10.686880+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:10.686880+0000","last_undegraded":"2024-09-16T10:27:10.686880+0000","last_fullsized":"2024-09-16T10:27:10.686880+0000","mapping_epoch":33,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41582592,"num_objects":10152,"num_object_clones":0,"num_object_copies":20304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5957,"num_write_kb":23828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10287","reported_seq":18790,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:14.173678+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:14.173678+0000","last_peered":"2024-09-16T10:27:14.173678+0000","last_clean":"2024-09-16T10:27:14.173678+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:14.173678+0000","last_undegraded":"2024-09-16T10:27:14.173678+0000","last_fullsized":"2024-09-16T10:27:14.173678+0000","mapping_epoch":43,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41673670,"num_objects":10176,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10285,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10474","reported_seq":10553,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.761490+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:10.761490+0000","last_peered":"2024-09-16T10:27:10.761490+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:10.761490+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42869702,"num_objects":10468,"num_object_clones":0,"num_object_copies":20936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10472,"num_write_kb":41868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6034},{"shards":"0,2","objects":4434}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10187","reported_seq":10293,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:11.171583+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:11.171583+0000","last_peered":"2024-09-16T10:27:11.171583+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:11.171583+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41706438,"num_objects":10184,"num_object_clones":0,"num_object_copies":20368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":40731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4277},{"shards":"2","objects":5907}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10254","reported_seq":10341,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:11.185863+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:11.185863+0000","last_peered":"2024-09-16T10:27:11.185863+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:11.185863+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41976774,"num_objects":10250,"num_object_clones":0,"num_object_copies":20500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10252,"num_write_kb":40995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4192},{"shards":"2","objects":6058}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'9965","reported_seq":10044,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.805316+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:10.805316+0000","last_peered":"2024-09-16T10:27:10.805316+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:10.805316+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":9965,"log_dups_size":0,"ondisk_log_size":9965,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40808931,"num_objects":9964,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9964,"num_write_kb":39853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5805},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10417","reported_seq":10550,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.836973+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:10.836973+0000","last_peered":"2024-09-16T10:27:10.836973+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:10.836973+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42636713,"num_objects":10412,"num_object_clones":0,"num_object_copies":20824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":41640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6075},{"shards":"0,2","objects":4337}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10036","reported_seq":10117,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.751734+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:10.751734+0000","last_peered":"2024-09-16T10:27:10.751734+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:10.751734+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41092038,"num_objects":10034,"num_object_clones":0,"num_object_copies":20068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10034,"num_write_kb":40130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5867},{"shards":"0,2","objects":4167}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10405","reported_seq":10498,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:10.777786+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:10.777786+0000","last_peered":"2024-09-16T10:27:10.777786+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:10.777786+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42571660,"num_objects":10397,"num_object_clones":0,"num_object_copies":20794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10402,"num_write_kb":41579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6002},{"shards":"0,2","objects":4395}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":670421494,"num_objects":163693,"num_object_clones":0,"num_object_copies":327386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42936,"num_objects_misplaced":3054,"num_objects_unfound":0,"num_objects_dirty":163693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141428,"num_write_kb":565388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5356,"num_bytes_recovered":21856739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1364811776,"data_stored":1364724605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018936,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712516,"kb_used_data":464316,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93659324,"statfs":{"total":96636764160,"available":95907147776,"internally_reserved":0,"allocated":475459584,"data_stored":475205950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":254144176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.6079999999999997},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.55600000000000005}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":3.8610000000000002},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":2.1629999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530408,"kb_used_data":310112,"kb_used_omap":4,"kb_used_meta":220283,"kb_avail":93841432,"statfs":{"total":96636764160,"available":96093626368,"internally_reserved":0,"allocated":317554688,"data_stored":317224722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":225570474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.96299999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":13.579000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":5.6230000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215164,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":892756,"kb_used_data":558476,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93479084,"statfs":{"total":96636764160,"available":95722582016,"internally_reserved":0,"allocated":571879424,"data_stored":571629404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,1,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":3.9470000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":7.7409999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":2.1760000000000002},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.399}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":572510208,"data_stored":572506595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":317165568,"data_stored":317085623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":475136000,"data_stored":475132387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:16.212 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:16.212 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'10191", 'reported_seq': 12537, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:10.860369+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:10.860369+0000', 'last_peered': '2024-09-16T10:27:10.860369+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:10.860369+0000', 'last_undegraded': '2024-09-16T10:27:10.860369+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41742336, 'num_objects': 10191, 'num_object_clones': 0, 'num_object_copies': 20382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3054, 'num_objects_unfound': 0, 'num_objects_dirty': 10191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10191, 'num_write_kb': 40764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1138, 'num_bytes_recovered': 4661248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7137}, {'shards': '0,2', 'objects': 3054}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:16.212 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:16.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'10301", 'reported_seq': 10486, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:11.082669+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:11.082669+0000', 'last_peered': '2024-09-16T10:27:11.082669+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:11.082669+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'300", 'ondisk_log_start': "35'300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42176512, 'num_objects': 10297, 'num_object_clones': 0, 'num_object_copies': 20594, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10297, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6358, 'num_write_kb': 25432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4335}, {'shards': '2', 'objects': 5962}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:16.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:16.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'10144", 'reported_seq': 10308, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:10.819737+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:10.819737+0000', 'last_peered': '2024-09-16T10:27:10.819737+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:10.819737+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41529344, 'num_objects': 10139, 'num_object_clones': 0, 'num_object_copies': 20278, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6769, 'num_write_kb': 27076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5922}, {'shards': '0,2', 'objects': 4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:16.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:16.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'10424", 'reported_seq': 10598, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:10.836211+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:10.836211+0000', 'last_peered': '2024-09-16T10:27:10.836211+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:10.836211+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'400", 'ondisk_log_start': "35'400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42672128, 'num_objects': 10418, 'num_object_clones': 0, 'num_object_copies': 20836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6666, 'num_write_kb': 26664, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6015}, {'shards': '0,2', 'objects': 4403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:16.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:16.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'10474", 'reported_seq': 10553, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:10.761490+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:10.761490+0000', 'last_peered': '2024-09-16T10:27:10.761490+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:10.761490+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'400", 'ondisk_log_start': "35'400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42869702, 'num_objects': 10468, 'num_object_clones': 0, 'num_object_copies': 20936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4436, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10472, 'num_write_kb': 41868, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6034}, {'shards': '0,2', 'objects': 4434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:16.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:16.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'10187", 'reported_seq': 10293, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:11.171583+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:11.171583+0000', 'last_peered': '2024-09-16T10:27:11.171583+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:11.171583+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41706438, 'num_objects': 10184, 'num_object_clones': 0, 'num_object_copies': 20368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10184, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10185, 'num_write_kb': 40731, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4277}, {'shards': '2', 'objects': 5907}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:16.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:16.214 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'10254", 'reported_seq': 10341, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:11.185863+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:11.185863+0000', 'last_peered': '2024-09-16T10:27:11.185863+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:11.185863+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'200", 'ondisk_log_start': "35'200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41976774, 'num_objects': 10250, 'num_object_clones': 0, 'num_object_copies': 20500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10252, 'num_write_kb': 40995, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4192}, {'shards': '2', 'objects': 6058}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:16.214 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:16.214 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'9965", 'reported_seq': 10044, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:10.805316+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:10.805316+0000', 'last_peered': '2024-09-16T10:27:10.805316+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:10.805316+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 9965, 'log_dups_size': 0, 'ondisk_log_size': 9965, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40808931, 'num_objects': 9964, 'num_object_clones': 0, 'num_object_copies': 19928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4160, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9964, 'num_write_kb': 39853, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5805}, {'shards': '0,2', 'objects': 4159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:16.214 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:16.214 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'10417", 'reported_seq': 10550, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:10.836973+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:10.836973+0000', 'last_peered': '2024-09-16T10:27:10.836973+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:10.836973+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'400", 'ondisk_log_start': "35'400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42636713, 'num_objects': 10412, 'num_object_clones': 0, 'num_object_copies': 20824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10414, 'num_write_kb': 41640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6075}, {'shards': '0,2', 'objects': 4337}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:16.214 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:16.214 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'10036", 'reported_seq': 10117, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:10.751734+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:10.751734+0000', 'last_peered': '2024-09-16T10:27:10.751734+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:10.751734+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41092038, 'num_objects': 10034, 'num_object_clones': 0, 'num_object_copies': 20068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10034, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10034, 'num_write_kb': 40130, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5867}, {'shards': '0,2', 'objects': 4167}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:16.214 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:16.214 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'10405", 'reported_seq': 10498, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:10.777786+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:10.777786+0000', 'last_peered': '2024-09-16T10:27:10.777786+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:10.777786+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'400", 'ondisk_log_start': "35'400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42571660, 'num_objects': 10397, 'num_object_clones': 0, 'num_object_copies': 20794, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4398, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10402, 'num_write_kb': 41579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6002}, {'shards': '0,2', 'objects': 4395}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:16.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:16.887 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:16.888 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42916/325784 objects degraded (13.173%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":325,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":162892,"data_bytes":667128353,"bytes_used":2192998400,"bytes_avail":287717294080,"bytes_total":289910292480,"degraded_objects":42916,"degraded_total":325784,"degraded_ratio":0.13173145396950126,"misplaced_objects":2963,"misplaced_total":325784,"misplaced_ratio":0.0090949831790388732,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":82912,"recovering_keys_per_sec":0,"num_objects_recovered":263,"num_bytes_recovered":995328,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":1653561,"read_op_per_sec":0,"write_op_per_sec":524},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:16.911 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:16.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:17.446 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:17.448 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:17.475 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-16T10:27:16.642473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667128353,"num_objects":162892,"num_object_clones":0,"num_object_copies":325784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42916,"num_objects_misplaced":2963,"num_objects_unfound":0,"num_objects_dirty":162892,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143417,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5468,"num_bytes_recovered":22229494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2141600,"kb_used_data":1330624,"kb_used_omap":12,"kb_used_meta":810931,"kb_avail":280973920,"statfs":{"total":289910292480,"available":287717294080,"internally_reserved":0,"allocated":1362558976,"data_stored":1361721284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":830393346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,7,14,5,4,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13918232,"num_objects":3399,"num_object_clones":0,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-20,"num_objects_misplaced":-243,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6295,"num_write_kb":19385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":263,"num_bytes_recovered":995328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004539"},"pg_stats":[{"pgid":"2.f","version":"44'10322","reported_seq":12848,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:15.892088+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:15.892088+0000","last_peered":"2024-09-16T10:27:15.892088+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:15.892088+0000","last_undegraded":"2024-09-16T10:27:15.892088+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41558016,"num_objects":10146,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2963,"num_objects_unfound":0,"num_objects_dirty":10146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10322,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1229,"num_bytes_recovered":5033984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7183},{"shards":"0,2","objects":2963}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10326","reported_seq":10483,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:16.181337+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:16.181337+0000","last_peered":"2024-09-16T10:27:16.181337+0000","last_clean":"2024-09-16T10:27:16.181337+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:16.181337+0000","last_undegraded":"2024-09-16T10:27:16.181337+0000","last_fullsized":"2024-09-16T10:27:16.181337+0000","mapping_epoch":34,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41517056,"num_objects":10136,"num_object_clones":0,"num_object_copies":20272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6539,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10428","reported_seq":10613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:16.172013+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:16.172013+0000","last_peered":"2024-09-16T10:27:16.172013+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:16.172013+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41922584,"num_objects":10236,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6485,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4241},{"shards":"2","objects":5901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10325","reported_seq":10381,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:16.101834+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:16.101834+0000","last_peered":"2024-09-16T10:27:16.101834+0000","last_clean":"2024-09-16T10:27:16.101834+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:16.101834+0000","last_undegraded":"2024-09-16T10:27:16.101834+0000","last_fullsized":"2024-09-16T10:27:16.101834+0000","mapping_epoch":34,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41521152,"num_objects":10137,"num_object_clones":0,"num_object_copies":20274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10325,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10344","reported_seq":10501,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:15.876897+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:15.876897+0000","last_peered":"2024-09-16T10:27:15.876897+0000","last_clean":"2024-09-16T10:27:15.876897+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:15.876897+0000","last_undegraded":"2024-09-16T10:27:15.876897+0000","last_fullsized":"2024-09-16T10:27:15.876897+0000","mapping_epoch":33,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41619456,"num_objects":10161,"num_object_clones":0,"num_object_copies":20322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6986,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10263","reported_seq":10427,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.921182+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:15.921182+0000","last_peered":"2024-09-16T10:27:15.921182+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:15.921182+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41402368,"num_objects":10108,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6888,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5891},{"shards":"0,2","objects":4142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10550","reported_seq":10725,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.852080+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:15.852080+0000","last_peered":"2024-09-16T10:27:15.852080+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:15.852080+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42475520,"num_objects":10370,"num_object_clones":0,"num_object_copies":20740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6792,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5969},{"shards":"0,2","objects":4316}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10284","reported_seq":10459,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:15.670082+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:15.670082+0000","last_peered":"2024-09-16T10:27:15.670082+0000","last_clean":"2024-09-16T10:27:15.670082+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:15.670082+0000","last_undegraded":"2024-09-16T10:27:15.670082+0000","last_fullsized":"2024-09-16T10:27:15.670082+0000","mapping_epoch":33,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41394176,"num_objects":10106,"num_object_clones":0,"num_object_copies":20212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10287","reported_seq":18790,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:14.173678+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:14.173678+0000","last_peered":"2024-09-16T10:27:14.173678+0000","last_clean":"2024-09-16T10:27:14.173678+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:14.173678+0000","last_undegraded":"2024-09-16T10:27:14.173678+0000","last_fullsized":"2024-09-16T10:27:14.173678+0000","mapping_epoch":43,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41673670,"num_objects":10176,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10285,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10634","reported_seq":10725,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.880092+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:15.880092+0000","last_peered":"2024-09-16T10:27:15.880092+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:15.880092+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42623942,"num_objects":10408,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5980},{"shards":"0,2","objects":4324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10313","reported_seq":10418,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:16.166657+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:16.166657+0000","last_peered":"2024-09-16T10:27:16.166657+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:16.166657+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41395142,"num_objects":10108,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4176},{"shards":"2","objects":5831}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10391","reported_seq":10478,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:16.169221+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:16.169221+0000","last_peered":"2024-09-16T10:27:16.169221+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:16.169221+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41718726,"num_objects":10187,"num_object_clones":0,"num_object_copies":20374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4092},{"shards":"2","objects":5995}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10119","reported_seq":10199,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.900902+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:15.900902+0000","last_peered":"2024-09-16T10:27:15.900902+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:15.900902+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40612323,"num_objects":9916,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5758},{"shards":"0,2","objects":4058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10547","reported_seq":10682,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.779693+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:15.779693+0000","last_peered":"2024-09-16T10:27:15.779693+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:15.779693+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42464681,"num_objects":10370,"num_object_clones":0,"num_object_copies":20740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10544,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6034},{"shards":"0,2","objects":4251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10155","reported_seq":10237,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.869088+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:15.869088+0000","last_peered":"2024-09-16T10:27:15.869088+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:15.869088+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40899526,"num_objects":9987,"num_object_clones":0,"num_object_copies":19974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5821},{"shards":"0,2","objects":4084}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10562","reported_seq":10673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.913585+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:15.913585+0000","last_peered":"2024-09-16T10:27:15.913585+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:15.913585+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42329996,"num_objects":10338,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5952},{"shards":"0,2","objects":4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":667128334,"num_objects":162890,"num_object_clones":0,"num_object_copies":325780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42916,"num_objects_misplaced":2963,"num_objects_unfound":0,"num_objects_dirty":162890,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143415,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5467,"num_bytes_recovered":22229475,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1360273408,"data_stored":1360182165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018937,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714728,"kb_used_data":462424,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93657112,"statfs":{"total":96636764160,"available":95904882688,"internally_reserved":0,"allocated":473522176,"data_stored":473264470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.7450000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":4.3630000000000004}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":2.8340000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691881,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530408,"kb_used_data":310112,"kb_used_omap":4,"kb_used_meta":220283,"kb_avail":93841432,"statfs":{"total":96636764160,"available":96093626368,"internally_reserved":0,"allocated":317554688,"data_stored":317224722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":225570474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.96299999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":13.579000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":5.6230000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215165,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":896464,"kb_used_data":558088,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93475376,"statfs":{"total":96636764160,"available":95718785024,"internally_reserved":0,"allocated":571482112,"data_stored":571232092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":346484392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":2.1080000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.88800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":3.27},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":2.0790000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":570064896,"data_stored":570061283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":317165568,"data_stored":317085623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":473042944,"data_stored":473035259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:20.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:21.001 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:21.004 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:21.030 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-16T10:27:20.646937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666690081,"num_objects":162785,"num_object_clones":0,"num_object_copies":325570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42916,"num_objects_misplaced":2963,"num_objects_unfound":0,"num_objects_dirty":162785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143524,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5468,"num_bytes_recovered":22229494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2128752,"kb_used_data":1330060,"kb_used_omap":12,"kb_used_meta":798643,"kb_avail":280986768,"statfs":{"total":289910292480,"available":287730450432,"internally_reserved":0,"allocated":1361981440,"data_stored":1361143748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":817810434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,15,4,5,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5169176,"num_objects":1263,"num_object_clones":0,"num_object_copies":2526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-20,"num_objects_misplaced":-164,"num_objects_unfound":0,"num_objects_dirty":1263,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4373,"num_write_kb":11269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007724"},"pg_stats":[{"pgid":"2.f","version":"44'10322","reported_seq":12848,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:15.892088+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:15.892088+0000","last_peered":"2024-09-16T10:27:15.892088+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:15.892088+0000","last_undegraded":"2024-09-16T10:27:15.892088+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41558016,"num_objects":10146,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2963,"num_objects_unfound":0,"num_objects_dirty":10146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10322,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1229,"num_bytes_recovered":5033984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7183},{"shards":"0,2","objects":2963}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10326","reported_seq":10483,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:16.181337+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:16.181337+0000","last_peered":"2024-09-16T10:27:16.181337+0000","last_clean":"2024-09-16T10:27:16.181337+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:16.181337+0000","last_undegraded":"2024-09-16T10:27:16.181337+0000","last_fullsized":"2024-09-16T10:27:16.181337+0000","mapping_epoch":34,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41517056,"num_objects":10136,"num_object_clones":0,"num_object_copies":20272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6539,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10428","reported_seq":10613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:16.172013+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:16.172013+0000","last_peered":"2024-09-16T10:27:16.172013+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:16.172013+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41922584,"num_objects":10236,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6485,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4241},{"shards":"2","objects":5901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10325","reported_seq":10381,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:16.101834+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:16.101834+0000","last_peered":"2024-09-16T10:27:16.101834+0000","last_clean":"2024-09-16T10:27:16.101834+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:16.101834+0000","last_undegraded":"2024-09-16T10:27:16.101834+0000","last_fullsized":"2024-09-16T10:27:16.101834+0000","mapping_epoch":34,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41521152,"num_objects":10137,"num_object_clones":0,"num_object_copies":20274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10325,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10344","reported_seq":10501,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:15.876897+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:15.876897+0000","last_peered":"2024-09-16T10:27:15.876897+0000","last_clean":"2024-09-16T10:27:15.876897+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:15.876897+0000","last_undegraded":"2024-09-16T10:27:15.876897+0000","last_fullsized":"2024-09-16T10:27:15.876897+0000","mapping_epoch":33,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41619456,"num_objects":10161,"num_object_clones":0,"num_object_copies":20322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6986,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10263","reported_seq":10427,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.921182+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:15.921182+0000","last_peered":"2024-09-16T10:27:15.921182+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:15.921182+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41402368,"num_objects":10108,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6888,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5891},{"shards":"0,2","objects":4142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10550","reported_seq":10725,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.852080+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:15.852080+0000","last_peered":"2024-09-16T10:27:15.852080+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:15.852080+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42475520,"num_objects":10370,"num_object_clones":0,"num_object_copies":20740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6792,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5969},{"shards":"0,2","objects":4316}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10284","reported_seq":10459,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:15.670082+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:15.670082+0000","last_peered":"2024-09-16T10:27:15.670082+0000","last_clean":"2024-09-16T10:27:15.670082+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:15.670082+0000","last_undegraded":"2024-09-16T10:27:15.670082+0000","last_fullsized":"2024-09-16T10:27:15.670082+0000","mapping_epoch":33,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41394176,"num_objects":10106,"num_object_clones":0,"num_object_copies":20212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10394","reported_seq":18897,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:19.198370+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:19.198370+0000","last_peered":"2024-09-16T10:27:19.198370+0000","last_clean":"2024-09-16T10:27:19.198370+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:19.198370+0000","last_undegraded":"2024-09-16T10:27:19.198370+0000","last_fullsized":"2024-09-16T10:27:19.198370+0000","mapping_epoch":43,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41235398,"num_objects":10069,"num_object_clones":0,"num_object_copies":20138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10634","reported_seq":10725,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.880092+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:15.880092+0000","last_peered":"2024-09-16T10:27:15.880092+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:15.880092+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42623942,"num_objects":10408,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5980},{"shards":"0,2","objects":4324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10313","reported_seq":10418,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:16.166657+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:16.166657+0000","last_peered":"2024-09-16T10:27:16.166657+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:16.166657+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41395142,"num_objects":10108,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4176},{"shards":"2","objects":5831}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10391","reported_seq":10478,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:16.169221+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:16.169221+0000","last_peered":"2024-09-16T10:27:16.169221+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:16.169221+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41718726,"num_objects":10187,"num_object_clones":0,"num_object_copies":20374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4092},{"shards":"2","objects":5995}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10119","reported_seq":10199,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.900902+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:15.900902+0000","last_peered":"2024-09-16T10:27:15.900902+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:15.900902+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40612323,"num_objects":9916,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5758},{"shards":"0,2","objects":4058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10547","reported_seq":10682,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.779693+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:15.779693+0000","last_peered":"2024-09-16T10:27:15.779693+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:15.779693+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42464681,"num_objects":10370,"num_object_clones":0,"num_object_copies":20740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10544,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6034},{"shards":"0,2","objects":4251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10155","reported_seq":10237,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.869088+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:15.869088+0000","last_peered":"2024-09-16T10:27:15.869088+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:15.869088+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40899526,"num_objects":9987,"num_object_clones":0,"num_object_copies":19974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5821},{"shards":"0,2","objects":4084}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10562","reported_seq":10673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.913585+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:15.913585+0000","last_peered":"2024-09-16T10:27:15.913585+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:15.913585+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42329996,"num_objects":10338,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5952},{"shards":"0,2","objects":4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":666690062,"num_objects":162783,"num_object_clones":0,"num_object_copies":325566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42916,"num_objects_misplaced":2963,"num_objects_unfound":0,"num_objects_dirty":162783,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143522,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5467,"num_bytes_recovered":22229475,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1359650816,"data_stored":1359559573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160757,"ondisk_log_size":160757,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018937,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714728,"kb_used_data":462424,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93657112,"statfs":{"total":96636764160,"available":95904882688,"internally_reserved":0,"allocated":473522176,"data_stored":473264470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.7450000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":4.3630000000000004}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":2.8340000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691882,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517560,"kb_used_data":309548,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93854280,"statfs":{"total":96636764160,"available":96106782720,"internally_reserved":0,"allocated":316977152,"data_stored":316647186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":212987562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.312}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":4.3609999999999998},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.04}]}]},{"osd":0,"up_from":24,"seq":103079215165,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":896464,"kb_used_data":558088,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93475376,"statfs":{"total":96636764160,"available":95718785024,"internally_reserved":0,"allocated":571482112,"data_stored":571232092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":346484392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":2.1080000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.88800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":3.27},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":2.0790000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":570064896,"data_stored":570061283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316542976,"data_stored":316463031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":473042944,"data_stored":473035259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:21.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:21.575 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:21.576 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:21.603 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-16T10:27:20.646937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666690081,"num_objects":162785,"num_object_clones":0,"num_object_copies":325570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42916,"num_objects_misplaced":2963,"num_objects_unfound":0,"num_objects_dirty":162785,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143524,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5468,"num_bytes_recovered":22229494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2128752,"kb_used_data":1330060,"kb_used_omap":12,"kb_used_meta":798643,"kb_avail":280986768,"statfs":{"total":289910292480,"available":287730450432,"internally_reserved":0,"allocated":1361981440,"data_stored":1361143748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":817810434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,15,4,5,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5169176,"num_objects":1263,"num_object_clones":0,"num_object_copies":2526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-20,"num_objects_misplaced":-164,"num_objects_unfound":0,"num_objects_dirty":1263,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4373,"num_write_kb":11269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":671744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007724"},"pg_stats":[{"pgid":"2.f","version":"44'10322","reported_seq":12848,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:15.892088+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:15.892088+0000","last_peered":"2024-09-16T10:27:15.892088+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:15.892088+0000","last_undegraded":"2024-09-16T10:27:15.892088+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41558016,"num_objects":10146,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2963,"num_objects_unfound":0,"num_objects_dirty":10146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10322,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1229,"num_bytes_recovered":5033984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7183},{"shards":"0,2","objects":2963}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10326","reported_seq":10483,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:16.181337+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:16.181337+0000","last_peered":"2024-09-16T10:27:16.181337+0000","last_clean":"2024-09-16T10:27:16.181337+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:16.181337+0000","last_undegraded":"2024-09-16T10:27:16.181337+0000","last_fullsized":"2024-09-16T10:27:16.181337+0000","mapping_epoch":34,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41517056,"num_objects":10136,"num_object_clones":0,"num_object_copies":20272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6539,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10428","reported_seq":10613,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:16.172013+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:16.172013+0000","last_peered":"2024-09-16T10:27:16.172013+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:16.172013+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41922584,"num_objects":10236,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6485,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4241},{"shards":"2","objects":5901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10325","reported_seq":10381,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:16.101834+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:16.101834+0000","last_peered":"2024-09-16T10:27:16.101834+0000","last_clean":"2024-09-16T10:27:16.101834+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:16.101834+0000","last_undegraded":"2024-09-16T10:27:16.101834+0000","last_fullsized":"2024-09-16T10:27:16.101834+0000","mapping_epoch":34,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41521152,"num_objects":10137,"num_object_clones":0,"num_object_copies":20274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10325,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10344","reported_seq":10501,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:15.876897+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:15.876897+0000","last_peered":"2024-09-16T10:27:15.876897+0000","last_clean":"2024-09-16T10:27:15.876897+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:15.876897+0000","last_undegraded":"2024-09-16T10:27:15.876897+0000","last_fullsized":"2024-09-16T10:27:15.876897+0000","mapping_epoch":33,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41619456,"num_objects":10161,"num_object_clones":0,"num_object_copies":20322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6986,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10263","reported_seq":10427,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.921182+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:15.921182+0000","last_peered":"2024-09-16T10:27:15.921182+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:15.921182+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41402368,"num_objects":10108,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6888,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5891},{"shards":"0,2","objects":4142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10550","reported_seq":10725,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.852080+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:15.852080+0000","last_peered":"2024-09-16T10:27:15.852080+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:15.852080+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42475520,"num_objects":10370,"num_object_clones":0,"num_object_copies":20740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6792,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5969},{"shards":"0,2","objects":4316}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10284","reported_seq":10459,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:15.670082+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:15.670082+0000","last_peered":"2024-09-16T10:27:15.670082+0000","last_clean":"2024-09-16T10:27:15.670082+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:15.670082+0000","last_undegraded":"2024-09-16T10:27:15.670082+0000","last_fullsized":"2024-09-16T10:27:15.670082+0000","mapping_epoch":33,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41394176,"num_objects":10106,"num_object_clones":0,"num_object_copies":20212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6087,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10394","reported_seq":18897,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:19.198370+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:19.198370+0000","last_peered":"2024-09-16T10:27:19.198370+0000","last_clean":"2024-09-16T10:27:19.198370+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:19.198370+0000","last_undegraded":"2024-09-16T10:27:19.198370+0000","last_fullsized":"2024-09-16T10:27:19.198370+0000","mapping_epoch":43,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41235398,"num_objects":10069,"num_object_clones":0,"num_object_copies":20138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10634","reported_seq":10725,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.880092+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:15.880092+0000","last_peered":"2024-09-16T10:27:15.880092+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:15.880092+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42623942,"num_objects":10408,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5980},{"shards":"0,2","objects":4324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10313","reported_seq":10418,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:16.166657+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:16.166657+0000","last_peered":"2024-09-16T10:27:16.166657+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:16.166657+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41395142,"num_objects":10108,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4176},{"shards":"2","objects":5831}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10391","reported_seq":10478,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:16.169221+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:16.169221+0000","last_peered":"2024-09-16T10:27:16.169221+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:16.169221+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41718726,"num_objects":10187,"num_object_clones":0,"num_object_copies":20374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4092},{"shards":"2","objects":5995}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10119","reported_seq":10199,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.900902+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:15.900902+0000","last_peered":"2024-09-16T10:27:15.900902+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:15.900902+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40612323,"num_objects":9916,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5758},{"shards":"0,2","objects":4058}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10547","reported_seq":10682,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.779693+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:15.779693+0000","last_peered":"2024-09-16T10:27:15.779693+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:15.779693+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42464681,"num_objects":10370,"num_object_clones":0,"num_object_copies":20740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10544,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":6034},{"shards":"0,2","objects":4251}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10155","reported_seq":10237,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.869088+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:15.869088+0000","last_peered":"2024-09-16T10:27:15.869088+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:15.869088+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'100","ondisk_log_start":"35'100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40899526,"num_objects":9987,"num_object_clones":0,"num_object_copies":19974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10153,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5821},{"shards":"0,2","objects":4084}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10562","reported_seq":10673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:15.913585+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:15.913585+0000","last_peered":"2024-09-16T10:27:15.913585+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:15.913585+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42329996,"num_objects":10338,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5952},{"shards":"0,2","objects":4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":666690062,"num_objects":162783,"num_object_clones":0,"num_object_copies":325566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42916,"num_objects_misplaced":2963,"num_objects_unfound":0,"num_objects_dirty":162783,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143522,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5467,"num_bytes_recovered":22229475,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1359650816,"data_stored":1359559573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160757,"ondisk_log_size":160757,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018937,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714728,"kb_used_data":462424,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93657112,"statfs":{"total":96636764160,"available":95904882688,"internally_reserved":0,"allocated":473522176,"data_stored":473264470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":258338480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.7450000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":4.3630000000000004}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":2.8340000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691882,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517560,"kb_used_data":309548,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93854280,"statfs":{"total":96636764160,"available":96106782720,"internally_reserved":0,"allocated":316977152,"data_stored":316647186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":212987562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.312}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":4.3609999999999998},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.04}]}]},{"osd":0,"up_from":24,"seq":103079215165,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":896464,"kb_used_data":558088,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93475376,"statfs":{"total":96636764160,"available":95718785024,"internally_reserved":0,"allocated":571482112,"data_stored":571232092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":346484392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,11],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":2.1080000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.88800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":3.27},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":2.0790000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":570064896,"data_stored":570061283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316542976,"data_stored":316463031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":473042944,"data_stored":473035259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:21.604 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:21.605 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'10322", 'reported_seq': 12848, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:15.892088+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:15.892088+0000', 'last_peered': '2024-09-16T10:27:15.892088+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:15.892088+0000', 'last_undegraded': '2024-09-16T10:27:15.892088+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'300", 'ondisk_log_start': "35'300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41558016, 'num_objects': 10146, 'num_object_clones': 0, 'num_object_copies': 20292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2963, 'num_objects_unfound': 0, 'num_objects_dirty': 10146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10322, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1229, 'num_bytes_recovered': 5033984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7183}, {'shards': '0,2', 'objects': 2963}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:21.605 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:21.605 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'10428", 'reported_seq': 10613, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:16.172013+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:16.172013+0000', 'last_peered': '2024-09-16T10:27:16.172013+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:16.172013+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'400", 'ondisk_log_start': "35'400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41922584, 'num_objects': 10236, 'num_object_clones': 0, 'num_object_copies': 20472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10236, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6485, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4241}, {'shards': '2', 'objects': 5901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:21.605 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:21.605 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'10263", 'reported_seq': 10427, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:15.921182+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:15.921182+0000', 'last_peered': '2024-09-16T10:27:15.921182+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:15.921182+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'200", 'ondisk_log_start': "35'200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41402368, 'num_objects': 10108, 'num_object_clones': 0, 'num_object_copies': 20216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6888, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5891}, {'shards': '0,2', 'objects': 4142}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:21.605 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:21.605 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'10550", 'reported_seq': 10725, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:15.852080+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:15.852080+0000', 'last_peered': '2024-09-16T10:27:15.852080+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:15.852080+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'500", 'ondisk_log_start': "35'500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42475520, 'num_objects': 10370, 'num_object_clones': 0, 'num_object_copies': 20740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4403, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6792, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5969}, {'shards': '0,2', 'objects': 4316}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:21.605 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'10634", 'reported_seq': 10725, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:15.880092+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:15.880092+0000', 'last_peered': '2024-09-16T10:27:15.880092+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:15.880092+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'600", 'ondisk_log_start': "35'600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42623942, 'num_objects': 10408, 'num_object_clones': 0, 'num_object_copies': 20816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4430, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10632, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5980}, {'shards': '0,2', 'objects': 4324}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'10313", 'reported_seq': 10418, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:16.166657+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:16.166657+0000', 'last_peered': '2024-09-16T10:27:16.166657+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:16.166657+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'300", 'ondisk_log_start': "35'300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41395142, 'num_objects': 10108, 'num_object_clones': 0, 'num_object_copies': 20216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4279, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10311, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4176}, {'shards': '2', 'objects': 5831}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'10391", 'reported_seq': 10478, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:16.169221+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:16.169221+0000', 'last_peered': '2024-09-16T10:27:16.169221+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:16.169221+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'300", 'ondisk_log_start': "35'300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41718726, 'num_objects': 10187, 'num_object_clones': 0, 'num_object_copies': 20374, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10187, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10389, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 4092}, {'shards': '2', 'objects': 5995}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'10119", 'reported_seq': 10199, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:15.900902+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:15.900902+0000', 'last_peered': '2024-09-16T10:27:15.900902+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:15.900902+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40612323, 'num_objects': 9916, 'num_object_clones': 0, 'num_object_copies': 19832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4159, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10118, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5758}, {'shards': '0,2', 'objects': 4058}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'10547", 'reported_seq': 10682, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:15.779693+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:15.779693+0000', 'last_peered': '2024-09-16T10:27:15.779693+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:15.779693+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'500", 'ondisk_log_start': "35'500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42464681, 'num_objects': 10370, 'num_object_clones': 0, 'num_object_copies': 20740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10544, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 6034}, {'shards': '0,2', 'objects': 4251}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:21.606 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'10155", 'reported_seq': 10237, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:15.869088+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:15.869088+0000', 'last_peered': '2024-09-16T10:27:15.869088+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:15.869088+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'100", 'ondisk_log_start': "35'100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40899526, 'num_objects': 9987, 'num_object_clones': 0, 'num_object_copies': 19974, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4168, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10153, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5821}, {'shards': '0,2', 'objects': 4084}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:21.607 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:21.607 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'10562", 'reported_seq': 10673, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:15.913585+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:15.913585+0000', 'last_peered': '2024-09-16T10:27:15.913585+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:15.913585+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'500", 'ondisk_log_start': "35'500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42329996, 'num_objects': 10338, 'num_object_clones': 0, 'num_object_copies': 20676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4389, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10559, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5952}, {'shards': '0,2', 'objects': 4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:21.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:22.266 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:22.267 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42916/325570 objects degraded (13.182%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":331,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":162785,"data_bytes":666690081,"bytes_used":2179842048,"bytes_avail":287730450432,"bytes_total":289910292480,"degraded_objects":42916,"degraded_total":325570,"degraded_ratio":0.13181804220290566,"misplaced_objects":2963,"misplaced_total":325570,"misplaced_ratio":0.0091009613907915357,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":55942,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":671744,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":961002,"read_op_per_sec":0,"write_op_per_sec":364},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:22.292 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:22.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:22.848 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:22.850 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:22.877 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-16T10:27:22.647754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658375201,"num_objects":160755,"num_object_clones":0,"num_object_copies":321510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42906,"num_objects_misplaced":2862,"num_objects_unfound":0,"num_objects_dirty":160755,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145554,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5579,"num_bytes_recovered":22643190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160990,"ondisk_log_size":160990,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2130352,"kb_used_data":1319380,"kb_used_omap":12,"kb_used_meta":810931,"kb_avail":280985168,"statfs":{"total":289910292480,"available":287728812032,"internally_reserved":0,"allocated":1351045120,"data_stored":1350207428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":830393346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,7,6,9,2,0,0,0,0,1,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3710952,"num_objects":-905,"num_object_clones":0,"num_object_copies":-1810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-30,"num_objects_misplaced":-265,"num_objects_unfound":0,"num_objects_dirty":-905,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6265,"num_write_kb":10717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":1085440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007879"},"pg_stats":[{"pgid":"2.f","version":"44'10467","reported_seq":13196,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:20.965867+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:20.965867+0000","last_peered":"2024-09-16T10:27:20.965867+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:20.965867+0000","last_undegraded":"2024-09-16T10:27:20.965867+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40964096,"num_objects":10001,"num_object_clones":0,"num_object_copies":20002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2862,"num_objects_unfound":0,"num_objects_dirty":10001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10467,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1330,"num_bytes_recovered":5447680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7139},{"shards":"0,2","objects":2862}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10452","reported_seq":10609,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:21.139915+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:21.139915+0000","last_peered":"2024-09-16T10:27:21.139915+0000","last_clean":"2024-09-16T10:27:21.139915+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:21.139915+0000","last_undegraded":"2024-09-16T10:27:21.139915+0000","last_fullsized":"2024-09-16T10:27:21.139915+0000","mapping_epoch":34,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41000960,"num_objects":10010,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6665,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10563","reported_seq":10747,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:21.155908+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:21.155908+0000","last_peered":"2024-09-16T10:27:21.155908+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:21.155908+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41369624,"num_objects":10101,"num_object_clones":0,"num_object_copies":20202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6620,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4106},{"shards":"2","objects":5766}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10458","reported_seq":10514,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:20.930119+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:20.930119+0000","last_peered":"2024-09-16T10:27:20.930119+0000","last_clean":"2024-09-16T10:27:20.930119+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:20.930119+0000","last_undegraded":"2024-09-16T10:27:20.930119+0000","last_fullsized":"2024-09-16T10:27:20.930119+0000","mapping_epoch":34,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40976384,"num_objects":10004,"num_object_clones":0,"num_object_copies":20008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10458,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10476","reported_seq":10633,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:20.881866+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:20.881866+0000","last_peered":"2024-09-16T10:27:20.881866+0000","last_clean":"2024-09-16T10:27:20.881866+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:20.881866+0000","last_undegraded":"2024-09-16T10:27:20.881866+0000","last_fullsized":"2024-09-16T10:27:20.881866+0000","mapping_epoch":33,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41078784,"num_objects":10029,"num_object_clones":0,"num_object_copies":20058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7118,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10395","reported_seq":10559,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.796271+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:20.796271+0000","last_peered":"2024-09-16T10:27:20.796271+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:20.796271+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861696,"num_objects":9976,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5759},{"shards":"0,2","objects":4010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10684","reported_seq":10864,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.939512+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:20.939512+0000","last_peered":"2024-09-16T10:27:20.939512+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:20.939512+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41926656,"num_objects":10236,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6926,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5836},{"shards":"0,2","objects":4182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10413","reported_seq":10588,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:20.890681+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:20.890681+0000","last_peered":"2024-09-16T10:27:20.890681+0000","last_clean":"2024-09-16T10:27:20.890681+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:20.890681+0000","last_undegraded":"2024-09-16T10:27:20.890681+0000","last_fullsized":"2024-09-16T10:27:20.890681+0000","mapping_epoch":33,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40865792,"num_objects":9977,"num_object_clones":0,"num_object_copies":19954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10394","reported_seq":18897,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:19.198370+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:19.198370+0000","last_peered":"2024-09-16T10:27:19.198370+0000","last_clean":"2024-09-16T10:27:19.198370+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:19.198370+0000","last_undegraded":"2024-09-16T10:27:19.198370+0000","last_fullsized":"2024-09-16T10:27:19.198370+0000","mapping_epoch":43,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41235398,"num_objects":10069,"num_object_clones":0,"num_object_copies":20138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10778","reported_seq":10870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.941048+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:20.941048+0000","last_peered":"2024-09-16T10:27:20.941048+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:20.941048+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42034118,"num_objects":10264,"num_object_clones":0,"num_object_copies":20528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5837},{"shards":"0,2","objects":4180}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10461","reported_seq":10567,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:21.109193+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:21.109193+0000","last_peered":"2024-09-16T10:27:21.109193+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:21.109193+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40788934,"num_objects":9960,"num_object_clones":0,"num_object_copies":19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4028},{"shards":"2","objects":5683}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10530","reported_seq":10619,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:21.176884+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:21.176884+0000","last_peered":"2024-09-16T10:27:21.176884+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:21.176884+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41149382,"num_objects":10048,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3953},{"shards":"2","objects":5857}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10219","reported_seq":10304,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.857452+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:20.857452+0000","last_peered":"2024-09-16T10:27:20.857452+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:20.857452+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40202723,"num_objects":9816,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10218,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5660},{"shards":"0,2","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10692","reported_seq":10829,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.881948+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:20.881948+0000","last_peered":"2024-09-16T10:27:20.881948+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:20.881948+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41870761,"num_objects":10225,"num_object_clones":0,"num_object_copies":20450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10689,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5890},{"shards":"0,2","objects":4106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10315","reported_seq":10402,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.916674+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:20.916674+0000","last_peered":"2024-09-16T10:27:20.916674+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:20.916674+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244166,"num_objects":9827,"num_object_clones":0,"num_object_copies":19654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10313,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5663},{"shards":"0,2","objects":3924}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10690","reported_seq":10805,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.776866+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:20.776866+0000","last_peered":"2024-09-16T10:27:20.776866+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:20.776866+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41805708,"num_objects":10210,"num_object_clones":0,"num_object_copies":20420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5826},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":658375182,"num_objects":160753,"num_object_clones":0,"num_object_copies":321506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42906,"num_objects_misplaced":2862,"num_objects_unfound":0,"num_objects_dirty":160753,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145552,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5578,"num_bytes_recovered":22643171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1348706304,"data_stored":1348615061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160987,"ondisk_log_size":160987,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018938,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714536,"kb_used_data":458144,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93657304,"statfs":{"total":96636764160,"available":95905079296,"internally_reserved":0,"allocated":469139456,"data_stored":468881750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":262532784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,1,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.9119999999999999},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.48999999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691882,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517560,"kb_used_data":309548,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93854280,"statfs":{"total":96636764160,"available":96106782720,"internally_reserved":0,"allocated":316977152,"data_stored":316647186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":212987562},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.312}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":4.3609999999999998},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.04}]}]},{"osd":0,"up_from":24,"seq":103079215166,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":898256,"kb_used_data":551688,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93473584,"statfs":{"total":96636764160,"available":95716950016,"internally_reserved":0,"allocated":564928512,"data_stored":564678492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":354873000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,3,5,2,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":2.0979999999999999},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":3.8079999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.61199999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.26200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":563478528,"data_stored":563474915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":316542976,"data_stored":316463031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":468684800,"data_stored":468677115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:25.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:26.448 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:26.450 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:26.467 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-16T10:27:24.649091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657797665,"num_objects":160614,"num_object_clones":0,"num_object_copies":321228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42906,"num_objects_misplaced":2862,"num_objects_unfound":0,"num_objects_dirty":160614,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145695,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5579,"num_bytes_recovered":22643190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160931,"ondisk_log_size":160931,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2151788,"kb_used_data":1318676,"kb_used_omap":12,"kb_used_meta":833075,"kb_avail":280963732,"statfs":{"total":289910292480,"available":287706861568,"internally_reserved":0,"allocated":1350324224,"data_stored":1349486532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":853068802},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,2,6,7,7,2,0,0,0,0,1,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12455912,"num_objects":-3040,"num_object_clones":0,"num_object_copies":-6080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-30,"num_objects_misplaced":-192,"num_objects_unfound":0,"num_objects_dirty":-3040,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4412,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008252"},"pg_stats":[{"pgid":"2.f","version":"44'10467","reported_seq":13196,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:20.965867+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:20.965867+0000","last_peered":"2024-09-16T10:27:20.965867+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:20.965867+0000","last_undegraded":"2024-09-16T10:27:20.965867+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40964096,"num_objects":10001,"num_object_clones":0,"num_object_copies":20002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2862,"num_objects_unfound":0,"num_objects_dirty":10001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10467,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1330,"num_bytes_recovered":5447680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7139},{"shards":"0,2","objects":2862}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10452","reported_seq":10609,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:21.139915+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:21.139915+0000","last_peered":"2024-09-16T10:27:21.139915+0000","last_clean":"2024-09-16T10:27:21.139915+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:21.139915+0000","last_undegraded":"2024-09-16T10:27:21.139915+0000","last_fullsized":"2024-09-16T10:27:21.139915+0000","mapping_epoch":34,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41000960,"num_objects":10010,"num_object_clones":0,"num_object_copies":20020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6665,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10563","reported_seq":10747,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:21.155908+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:21.155908+0000","last_peered":"2024-09-16T10:27:21.155908+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:21.155908+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41369624,"num_objects":10101,"num_object_clones":0,"num_object_copies":20202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6620,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4106},{"shards":"2","objects":5766}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10458","reported_seq":10514,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:20.930119+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:20.930119+0000","last_peered":"2024-09-16T10:27:20.930119+0000","last_clean":"2024-09-16T10:27:20.930119+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:20.930119+0000","last_undegraded":"2024-09-16T10:27:20.930119+0000","last_fullsized":"2024-09-16T10:27:20.930119+0000","mapping_epoch":34,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40976384,"num_objects":10004,"num_object_clones":0,"num_object_copies":20008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10458,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10476","reported_seq":10633,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:20.881866+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:20.881866+0000","last_peered":"2024-09-16T10:27:20.881866+0000","last_clean":"2024-09-16T10:27:20.881866+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:20.881866+0000","last_undegraded":"2024-09-16T10:27:20.881866+0000","last_fullsized":"2024-09-16T10:27:20.881866+0000","mapping_epoch":33,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41078784,"num_objects":10029,"num_object_clones":0,"num_object_copies":20058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7118,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10395","reported_seq":10559,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.796271+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:20.796271+0000","last_peered":"2024-09-16T10:27:20.796271+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:20.796271+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861696,"num_objects":9976,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7020,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5759},{"shards":"0,2","objects":4010}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10684","reported_seq":10864,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.939512+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:20.939512+0000","last_peered":"2024-09-16T10:27:20.939512+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:20.939512+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41926656,"num_objects":10236,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6926,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5836},{"shards":"0,2","objects":4182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10413","reported_seq":10588,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:20.890681+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:20.890681+0000","last_peered":"2024-09-16T10:27:20.890681+0000","last_clean":"2024-09-16T10:27:20.890681+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:20.890681+0000","last_undegraded":"2024-09-16T10:27:20.890681+0000","last_fullsized":"2024-09-16T10:27:20.890681+0000","mapping_epoch":33,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40865792,"num_objects":9977,"num_object_clones":0,"num_object_copies":19954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10535","reported_seq":19038,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:24.226152+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:24.226152+0000","last_peered":"2024-09-16T10:27:24.226152+0000","last_clean":"2024-09-16T10:27:24.226152+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:24.226152+0000","last_undegraded":"2024-09-16T10:27:24.226152+0000","last_fullsized":"2024-09-16T10:27:24.226152+0000","mapping_epoch":43,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40657862,"num_objects":9928,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10533,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10778","reported_seq":10870,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.941048+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:20.941048+0000","last_peered":"2024-09-16T10:27:20.941048+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:20.941048+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42034118,"num_objects":10264,"num_object_clones":0,"num_object_copies":20528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10776,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5837},{"shards":"0,2","objects":4180}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10461","reported_seq":10567,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:21.109193+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:21.109193+0000","last_peered":"2024-09-16T10:27:21.109193+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:21.109193+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40788934,"num_objects":9960,"num_object_clones":0,"num_object_copies":19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4279,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10459,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":4028},{"shards":"2","objects":5683}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10530","reported_seq":10619,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:21.176884+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:21.176884+0000","last_peered":"2024-09-16T10:27:21.176884+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:21.176884+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41149382,"num_objects":10048,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10528,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3953},{"shards":"2","objects":5857}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10219","reported_seq":10304,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.857452+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:20.857452+0000","last_peered":"2024-09-16T10:27:20.857452+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:20.857452+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'200","ondisk_log_start":"35'200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40202723,"num_objects":9816,"num_object_clones":0,"num_object_copies":19632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10218,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5660},{"shards":"0,2","objects":3958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10692","reported_seq":10829,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.881948+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:20.881948+0000","last_peered":"2024-09-16T10:27:20.881948+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:20.881948+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41870761,"num_objects":10225,"num_object_clones":0,"num_object_copies":20450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10689,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5890},{"shards":"0,2","objects":4106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10315","reported_seq":10402,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.916674+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:20.916674+0000","last_peered":"2024-09-16T10:27:20.916674+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:20.916674+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244166,"num_objects":9827,"num_object_clones":0,"num_object_copies":19654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10313,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5663},{"shards":"0,2","objects":3924}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10690","reported_seq":10805,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:20.776866+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:20.776866+0000","last_peered":"2024-09-16T10:27:20.776866+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:20.776866+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41805708,"num_objects":10210,"num_object_clones":0,"num_object_copies":20420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5826},{"shards":"0,2","objects":4159}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":657797646,"num_objects":160612,"num_object_clones":0,"num_object_copies":321224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42906,"num_objects_misplaced":2862,"num_objects_unfound":0,"num_objects_dirty":160612,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145693,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5578,"num_bytes_recovered":22643171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1347993600,"data_stored":1347902357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018938,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714536,"kb_used_data":458144,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93657304,"statfs":{"total":96636764160,"available":95905079296,"internally_reserved":0,"allocated":469139456,"data_stored":468881750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":262532784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,1,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.9119999999999999},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.48999999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538996,"kb_used_data":308844,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93832844,"statfs":{"total":96636764160,"available":96084832256,"internally_reserved":0,"allocated":316256256,"data_stored":315926290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":235663018},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.184},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":11.497999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.65500000000000003},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.68100000000000005}]}]},{"osd":0,"up_from":24,"seq":103079215166,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":898256,"kb_used_data":551688,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93473584,"statfs":{"total":96636764160,"available":95716950016,"internally_reserved":0,"allocated":564928512,"data_stored":564678492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":354873000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,3,5,2,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":2.0979999999999999},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":3.8079999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.61199999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.26200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":563478528,"data_stored":563474915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315830272,"data_stored":315750327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":468684800,"data_stored":468677115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:26.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:27.027 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:27.048 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-16T10:27:26.649835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649654817,"num_objects":158626,"num_object_clones":0,"num_object_copies":317252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42886,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":158626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147683,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5704,"num_bytes_recovered":23073270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2135104,"kb_used_data":1308512,"kb_used_omap":12,"kb_used_meta":826547,"kb_avail":280980416,"statfs":{"total":289910292480,"available":287723945984,"internally_reserved":0,"allocated":1339916288,"data_stored":1339078596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":846384130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,0,13,7,1,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20766696,"num_objects":-5069,"num_object_clones":0,"num_object_copies":-10138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-50,"num_objects_misplaced":-297,"num_objects_unfound":0,"num_objects_dirty":-5069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6253,"num_write_kb":2365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":347,"num_bytes_recovered":1216512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008037"},"pg_stats":[{"pgid":"2.f","version":"44'10595","reported_seq":13535,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:25.982993+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:25.982993+0000","last_peered":"2024-09-16T10:27:25.982993+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:25.982993+0000","last_undegraded":"2024-09-16T10:27:25.982993+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40439808,"num_objects":9873,"num_object_clones":0,"num_object_copies":19746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":9873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10595,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1435,"num_bytes_recovered":5877760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7116},{"shards":"0,2","objects":2757}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10567","reported_seq":10724,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:26.062761+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:26.062761+0000","last_peered":"2024-09-16T10:27:26.062761+0000","last_clean":"2024-09-16T10:27:26.062761+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:26.062761+0000","last_undegraded":"2024-09-16T10:27:26.062761+0000","last_fullsized":"2024-09-16T10:27:26.062761+0000","mapping_epoch":34,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40529920,"num_objects":9895,"num_object_clones":0,"num_object_copies":19790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6780,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10687","reported_seq":10881,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.192269+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:26.192269+0000","last_peered":"2024-09-16T10:27:26.192269+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:26.192269+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861720,"num_objects":9977,"num_object_clones":0,"num_object_copies":19954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9977,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6744,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3982},{"shards":"2","objects":5646}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10601","reported_seq":10658,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:26.138052+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:26.138052+0000","last_peered":"2024-09-16T10:27:26.138052+0000","last_clean":"2024-09-16T10:27:26.138052+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:26.138052+0000","last_undegraded":"2024-09-16T10:27:26.138052+0000","last_fullsized":"2024-09-16T10:27:26.138052+0000","mapping_epoch":34,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40390656,"num_objects":9861,"num_object_clones":0,"num_object_copies":19722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10602","reported_seq":10759,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:25.920280+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:25.920280+0000","last_peered":"2024-09-16T10:27:25.920280+0000","last_clean":"2024-09-16T10:27:25.920280+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:25.920280+0000","last_undegraded":"2024-09-16T10:27:25.920280+0000","last_fullsized":"2024-09-16T10:27:25.920280+0000","mapping_epoch":33,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10522","reported_seq":10685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.938947+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:25.938947+0000","last_peered":"2024-09-16T10:27:25.938947+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:25.938947+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40341504,"num_objects":9849,"num_object_clones":0,"num_object_copies":19698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5632},{"shards":"0,2","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10837","reported_seq":11019,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.947221+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:25.947221+0000","last_peered":"2024-09-16T10:27:25.947221+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:25.947221+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41299968,"num_objects":10083,"num_object_clones":0,"num_object_copies":20166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7079,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5684},{"shards":"0,2","objects":4029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10527","reported_seq":10702,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:25.947268+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:25.947268+0000","last_peered":"2024-09-16T10:27:25.947268+0000","last_clean":"2024-09-16T10:27:25.947268+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:25.947268+0000","last_undegraded":"2024-09-16T10:27:25.947268+0000","last_fullsized":"2024-09-16T10:27:25.947268+0000","mapping_epoch":33,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40398848,"num_objects":9863,"num_object_clones":0,"num_object_copies":19726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6330,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10535","reported_seq":19038,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:24.226152+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:24.226152+0000","last_peered":"2024-09-16T10:27:24.226152+0000","last_clean":"2024-09-16T10:27:24.226152+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:24.226152+0000","last_undegraded":"2024-09-16T10:27:24.226152+0000","last_fullsized":"2024-09-16T10:27:24.226152+0000","mapping_epoch":43,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40657862,"num_objects":9928,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10533,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10929","reported_seq":11022,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.951918+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:25.951918+0000","last_peered":"2024-09-16T10:27:25.951918+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:25.951918+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415622,"num_objects":10113,"num_object_clones":0,"num_object_copies":20226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10927,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5686},{"shards":"0,2","objects":4029}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10587","reported_seq":10697,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.158382+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:26.158382+0000","last_peered":"2024-09-16T10:27:26.158382+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:26.158382+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40272838,"num_objects":9834,"num_object_clones":0,"num_object_copies":19668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10585,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3902},{"shards":"2","objects":5559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10690","reported_seq":10785,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.138423+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:26.138423+0000","last_peered":"2024-09-16T10:27:26.138423+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:26.138423+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40494022,"num_objects":9888,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10688,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3793},{"shards":"2","objects":5700}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10356","reported_seq":10444,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.901866+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:25.901866+0000","last_peered":"2024-09-16T10:27:25.901866+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:25.901866+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39641571,"num_objects":9679,"num_object_clones":0,"num_object_copies":19358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10355,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5524},{"shards":"0,2","objects":3821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10828","reported_seq":10973,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.971278+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:25.971278+0000","last_peered":"2024-09-16T10:27:25.971278+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:25.971278+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41313705,"num_objects":10089,"num_object_clones":0,"num_object_copies":20178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10825,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5758},{"shards":"0,2","objects":3970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10447","reported_seq":10544,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.778996+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:25.778996+0000","last_peered":"2024-09-16T10:27:25.778996+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:25.778996+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39703494,"num_objects":9695,"num_object_clones":0,"num_object_copies":19390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10445,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5536},{"shards":"0,2","objects":3792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10806","reported_seq":10921,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.814993+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:25.814993+0000","last_peered":"2024-09-16T10:27:25.814993+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:25.814993+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41330572,"num_objects":10094,"num_object_clones":0,"num_object_copies":20188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10803,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5710},{"shards":"0,2","objects":4043}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":649654798,"num_objects":158624,"num_object_clones":0,"num_object_copies":317248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42886,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":158624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147681,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5703,"num_bytes_recovered":23073251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1337286656,"data_stored":1337195413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160716,"ondisk_log_size":160716,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018939,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716152,"kb_used_data":454000,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93655688,"statfs":{"total":96636764160,"available":95903424512,"internally_reserved":0,"allocated":464896000,"data_stored":464638294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":268431024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.089},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1469999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":3.1890000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538996,"kb_used_data":308844,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93832844,"statfs":{"total":96636764160,"available":96084832256,"internally_reserved":0,"allocated":316256256,"data_stored":315926290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":235663018},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.184},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":11.497999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.65500000000000003},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.68100000000000005}]}]},{"osd":0,"up_from":24,"seq":103079215167,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":879956,"kb_used_data":545668,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93491884,"statfs":{"total":96636764160,"available":95735689216,"internally_reserved":0,"allocated":558764032,"data_stored":558514012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,9,1,0,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.47899999999999998},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":4.5910000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":5.4359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":557023232,"data_stored":557019619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315830272,"data_stored":315750327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":464433152,"data_stored":464425467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:27.049 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:27.049 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'10595", 'reported_seq': 13535, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:25.982993+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:25.982993+0000', 'last_peered': '2024-09-16T10:27:25.982993+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:25.982993+0000', 'last_undegraded': '2024-09-16T10:27:25.982993+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'500", 'ondisk_log_start': "35'500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40439808, 'num_objects': 9873, 'num_object_clones': 0, 'num_object_copies': 19746, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2757, 'num_objects_unfound': 0, 'num_objects_dirty': 9873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10595, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1435, 'num_bytes_recovered': 5877760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7116}, {'shards': '0,2', 'objects': 2757}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:27.050 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:27.050 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'10687", 'reported_seq': 10881, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:26.192269+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:26.192269+0000', 'last_peered': '2024-09-16T10:27:26.192269+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:26.192269+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'600", 'ondisk_log_start': "35'600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40861720, 'num_objects': 9977, 'num_object_clones': 0, 'num_object_copies': 19954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4332, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9977, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6744, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3982}, {'shards': '2', 'objects': 5646}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:27.050 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:27.050 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'10522", 'reported_seq': 10685, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.938947+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:25.938947+0000', 'last_peered': '2024-09-16T10:27:25.938947+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:25.938947+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'500", 'ondisk_log_start': "35'500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40341504, 'num_objects': 9849, 'num_object_clones': 0, 'num_object_copies': 19698, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7147, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5632}, {'shards': '0,2', 'objects': 3883}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:27.050 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:27.050 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'10837", 'reported_seq': 11019, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.947221+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:25.947221+0000', 'last_peered': '2024-09-16T10:27:25.947221+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:25.947221+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41299968, 'num_objects': 10083, 'num_object_clones': 0, 'num_object_copies': 20166, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4401, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7079, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5684}, {'shards': '0,2', 'objects': 4029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:27.050 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:27.050 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'10929", 'reported_seq': 11022, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.951918+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:25.951918+0000', 'last_peered': '2024-09-16T10:27:25.951918+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:25.951918+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'900", 'ondisk_log_start': "35'900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41415622, 'num_objects': 10113, 'num_object_clones': 0, 'num_object_copies': 20226, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4429, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10927, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5686}, {'shards': '0,2', 'objects': 4029}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:27.050 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:27.050 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'10587", 'reported_seq': 10697, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:26.158382+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:26.158382+0000', 'last_peered': '2024-09-16T10:27:26.158382+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:26.158382+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'500", 'ondisk_log_start': "35'500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40272838, 'num_objects': 9834, 'num_object_clones': 0, 'num_object_copies': 19668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4277, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10585, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3902}, {'shards': '2', 'objects': 5559}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:27.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:27.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'10690", 'reported_seq': 10785, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:26.138423+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:26.138423+0000', 'last_peered': '2024-09-16T10:27:26.138423+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:26.138423+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'600", 'ondisk_log_start': "35'600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40494022, 'num_objects': 9888, 'num_object_clones': 0, 'num_object_copies': 19776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4190, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10688, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3793}, {'shards': '2', 'objects': 5700}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:27.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:27.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'10356", 'reported_seq': 10444, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.901866+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:25.901866+0000', 'last_peered': '2024-09-16T10:27:25.901866+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:25.901866+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'300", 'ondisk_log_start': "35'300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39641571, 'num_objects': 9679, 'num_object_clones': 0, 'num_object_copies': 19358, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4156, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10355, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5524}, {'shards': '0,2', 'objects': 3821}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:27.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:27.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'10828", 'reported_seq': 10973, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.971278+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:25.971278+0000', 'last_peered': '2024-09-16T10:27:25.971278+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:25.971278+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41313705, 'num_objects': 10089, 'num_object_clones': 0, 'num_object_copies': 20178, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4334, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10825, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5758}, {'shards': '0,2', 'objects': 3970}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:27.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:27.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'10447", 'reported_seq': 10544, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.778996+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:25.778996+0000', 'last_peered': '2024-09-16T10:27:25.778996+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:25.778996+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'400", 'ondisk_log_start': "35'400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39703494, 'num_objects': 9695, 'num_object_clones': 0, 'num_object_copies': 19390, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4161, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10445, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5536}, {'shards': '0,2', 'objects': 3792}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:27.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:27.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'10806", 'reported_seq': 10921, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.814993+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:25.814993+0000', 'last_peered': '2024-09-16T10:27:25.814993+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:25.814993+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41330572, 'num_objects': 10094, 'num_object_clones': 0, 'num_object_copies': 20188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4387, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10803, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5710}, {'shards': '0,2', 'objects': 4043}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:27.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:27.712 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:27.713 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42886/317252 objects degraded (13.518%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":336,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":158626,"data_bytes":649654817,"bytes_used":2186346496,"bytes_avail":287723945984,"bytes_total":289910292480,"degraded_objects":42886,"degraded_total":317252,"degraded_ratio":0.13517960485670696,"misplaced_objects":2757,"misplaced_total":317252,"misplaced_ratio":0.0086902525437191885,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":101308,"recovering_keys_per_sec":0,"num_objects_recovered":347,"num_bytes_recovered":1216512,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":201678,"read_op_per_sec":0,"write_op_per_sec":520},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:27.735 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:27.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:28.258 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:28.259 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:28.281 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-16T10:27:26.649835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649654817,"num_objects":158626,"num_object_clones":0,"num_object_copies":317252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42886,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":158626,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147683,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5704,"num_bytes_recovered":23073270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2135104,"kb_used_data":1308512,"kb_used_omap":12,"kb_used_meta":826547,"kb_avail":280980416,"statfs":{"total":289910292480,"available":287723945984,"internally_reserved":0,"allocated":1339916288,"data_stored":1339078596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":846384130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,0,13,7,1,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20766696,"num_objects":-5069,"num_object_clones":0,"num_object_copies":-10138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-50,"num_objects_misplaced":-297,"num_objects_unfound":0,"num_objects_dirty":-5069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6253,"num_write_kb":2365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":347,"num_bytes_recovered":1216512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008037"},"pg_stats":[{"pgid":"2.f","version":"44'10595","reported_seq":13535,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:25.982993+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:25.982993+0000","last_peered":"2024-09-16T10:27:25.982993+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:25.982993+0000","last_undegraded":"2024-09-16T10:27:25.982993+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40439808,"num_objects":9873,"num_object_clones":0,"num_object_copies":19746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":9873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10595,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1435,"num_bytes_recovered":5877760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7116},{"shards":"0,2","objects":2757}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10567","reported_seq":10724,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:26.062761+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:26.062761+0000","last_peered":"2024-09-16T10:27:26.062761+0000","last_clean":"2024-09-16T10:27:26.062761+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:26.062761+0000","last_undegraded":"2024-09-16T10:27:26.062761+0000","last_fullsized":"2024-09-16T10:27:26.062761+0000","mapping_epoch":34,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40529920,"num_objects":9895,"num_object_clones":0,"num_object_copies":19790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6780,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10687","reported_seq":10881,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.192269+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:26.192269+0000","last_peered":"2024-09-16T10:27:26.192269+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:26.192269+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861720,"num_objects":9977,"num_object_clones":0,"num_object_copies":19954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9977,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6744,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3982},{"shards":"2","objects":5646}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10601","reported_seq":10658,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:26.138052+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:26.138052+0000","last_peered":"2024-09-16T10:27:26.138052+0000","last_clean":"2024-09-16T10:27:26.138052+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:26.138052+0000","last_undegraded":"2024-09-16T10:27:26.138052+0000","last_fullsized":"2024-09-16T10:27:26.138052+0000","mapping_epoch":34,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40390656,"num_objects":9861,"num_object_clones":0,"num_object_copies":19722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10602","reported_seq":10759,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:25.920280+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:25.920280+0000","last_peered":"2024-09-16T10:27:25.920280+0000","last_clean":"2024-09-16T10:27:25.920280+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:25.920280+0000","last_undegraded":"2024-09-16T10:27:25.920280+0000","last_fullsized":"2024-09-16T10:27:25.920280+0000","mapping_epoch":33,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10522","reported_seq":10685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.938947+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:25.938947+0000","last_peered":"2024-09-16T10:27:25.938947+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:25.938947+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40341504,"num_objects":9849,"num_object_clones":0,"num_object_copies":19698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5632},{"shards":"0,2","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10837","reported_seq":11019,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.947221+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:25.947221+0000","last_peered":"2024-09-16T10:27:25.947221+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:25.947221+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41299968,"num_objects":10083,"num_object_clones":0,"num_object_copies":20166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7079,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5684},{"shards":"0,2","objects":4029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10527","reported_seq":10702,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:25.947268+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:25.947268+0000","last_peered":"2024-09-16T10:27:25.947268+0000","last_clean":"2024-09-16T10:27:25.947268+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:25.947268+0000","last_undegraded":"2024-09-16T10:27:25.947268+0000","last_fullsized":"2024-09-16T10:27:25.947268+0000","mapping_epoch":33,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40398848,"num_objects":9863,"num_object_clones":0,"num_object_copies":19726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6330,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10535","reported_seq":19038,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:24.226152+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:24.226152+0000","last_peered":"2024-09-16T10:27:24.226152+0000","last_clean":"2024-09-16T10:27:24.226152+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:24.226152+0000","last_undegraded":"2024-09-16T10:27:24.226152+0000","last_fullsized":"2024-09-16T10:27:24.226152+0000","mapping_epoch":43,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40657862,"num_objects":9928,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10533,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10929","reported_seq":11022,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.951918+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:25.951918+0000","last_peered":"2024-09-16T10:27:25.951918+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:25.951918+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415622,"num_objects":10113,"num_object_clones":0,"num_object_copies":20226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10927,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5686},{"shards":"0,2","objects":4029}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10587","reported_seq":10697,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.158382+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:26.158382+0000","last_peered":"2024-09-16T10:27:26.158382+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:26.158382+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40272838,"num_objects":9834,"num_object_clones":0,"num_object_copies":19668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10585,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3902},{"shards":"2","objects":5559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10690","reported_seq":10785,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.138423+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:26.138423+0000","last_peered":"2024-09-16T10:27:26.138423+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:26.138423+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40494022,"num_objects":9888,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10688,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3793},{"shards":"2","objects":5700}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10356","reported_seq":10444,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.901866+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:25.901866+0000","last_peered":"2024-09-16T10:27:25.901866+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:25.901866+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39641571,"num_objects":9679,"num_object_clones":0,"num_object_copies":19358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10355,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5524},{"shards":"0,2","objects":3821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10828","reported_seq":10973,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.971278+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:25.971278+0000","last_peered":"2024-09-16T10:27:25.971278+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:25.971278+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41313705,"num_objects":10089,"num_object_clones":0,"num_object_copies":20178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10825,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5758},{"shards":"0,2","objects":3970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10447","reported_seq":10544,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.778996+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:25.778996+0000","last_peered":"2024-09-16T10:27:25.778996+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:25.778996+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39703494,"num_objects":9695,"num_object_clones":0,"num_object_copies":19390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10445,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5536},{"shards":"0,2","objects":3792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10806","reported_seq":10921,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.814993+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:25.814993+0000","last_peered":"2024-09-16T10:27:25.814993+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:25.814993+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41330572,"num_objects":10094,"num_object_clones":0,"num_object_copies":20188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10803,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5710},{"shards":"0,2","objects":4043}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":649654798,"num_objects":158624,"num_object_clones":0,"num_object_copies":317248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42886,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":158624,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147681,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5703,"num_bytes_recovered":23073251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1337286656,"data_stored":1337195413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160716,"ondisk_log_size":160716,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018939,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716152,"kb_used_data":454000,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93655688,"statfs":{"total":96636764160,"available":95903424512,"internally_reserved":0,"allocated":464896000,"data_stored":464638294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":268431024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.089},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1469999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":3.1890000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538996,"kb_used_data":308844,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93832844,"statfs":{"total":96636764160,"available":96084832256,"internally_reserved":0,"allocated":316256256,"data_stored":315926290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":235663018},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.184},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":11.497999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.65500000000000003},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.68100000000000005}]}]},{"osd":0,"up_from":24,"seq":103079215167,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":879956,"kb_used_data":545668,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93491884,"statfs":{"total":96636764160,"available":95735689216,"internally_reserved":0,"allocated":558764032,"data_stored":558514012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,9,1,0,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.47899999999999998},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":4.5910000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":5.4359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":557023232,"data_stored":557019619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315830272,"data_stored":315750327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":464433152,"data_stored":464425467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:31.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:31.826 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:31.827 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:31.847 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-16T10:27:30.650900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649122337,"num_objects":158496,"num_object_clones":0,"num_object_copies":316992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42886,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":158496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147813,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5704,"num_bytes_recovered":23073270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2160776,"kb_used_data":1307944,"kb_used_omap":12,"kb_used_meta":852787,"kb_avail":280954744,"statfs":{"total":289910292480,"available":287697657856,"internally_reserved":0,"allocated":1339334656,"data_stored":1338496964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":873253890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,12,6,1,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18006016,"num_objects":-4396,"num_object_clones":0,"num_object_copies":-8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-30,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":-4396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004884"},"pg_stats":[{"pgid":"2.f","version":"44'10595","reported_seq":13535,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:25.982993+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:25.982993+0000","last_peered":"2024-09-16T10:27:25.982993+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:25.982993+0000","last_undegraded":"2024-09-16T10:27:25.982993+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40439808,"num_objects":9873,"num_object_clones":0,"num_object_copies":19746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":9873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10595,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1435,"num_bytes_recovered":5877760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7116},{"shards":"0,2","objects":2757}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10567","reported_seq":10724,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:26.062761+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:26.062761+0000","last_peered":"2024-09-16T10:27:26.062761+0000","last_clean":"2024-09-16T10:27:26.062761+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:26.062761+0000","last_undegraded":"2024-09-16T10:27:26.062761+0000","last_fullsized":"2024-09-16T10:27:26.062761+0000","mapping_epoch":34,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40529920,"num_objects":9895,"num_object_clones":0,"num_object_copies":19790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6780,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10687","reported_seq":10881,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.192269+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:26.192269+0000","last_peered":"2024-09-16T10:27:26.192269+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:26.192269+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861720,"num_objects":9977,"num_object_clones":0,"num_object_copies":19954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9977,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6744,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3982},{"shards":"2","objects":5646}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10601","reported_seq":10658,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:26.138052+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:26.138052+0000","last_peered":"2024-09-16T10:27:26.138052+0000","last_clean":"2024-09-16T10:27:26.138052+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:26.138052+0000","last_undegraded":"2024-09-16T10:27:26.138052+0000","last_fullsized":"2024-09-16T10:27:26.138052+0000","mapping_epoch":34,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40390656,"num_objects":9861,"num_object_clones":0,"num_object_copies":19722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10602","reported_seq":10759,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:25.920280+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:25.920280+0000","last_peered":"2024-09-16T10:27:25.920280+0000","last_clean":"2024-09-16T10:27:25.920280+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:25.920280+0000","last_undegraded":"2024-09-16T10:27:25.920280+0000","last_fullsized":"2024-09-16T10:27:25.920280+0000","mapping_epoch":33,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10522","reported_seq":10685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.938947+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:25.938947+0000","last_peered":"2024-09-16T10:27:25.938947+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:25.938947+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40341504,"num_objects":9849,"num_object_clones":0,"num_object_copies":19698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5632},{"shards":"0,2","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10837","reported_seq":11019,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.947221+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:25.947221+0000","last_peered":"2024-09-16T10:27:25.947221+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:25.947221+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41299968,"num_objects":10083,"num_object_clones":0,"num_object_copies":20166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7079,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5684},{"shards":"0,2","objects":4029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10527","reported_seq":10702,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:25.947268+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:25.947268+0000","last_peered":"2024-09-16T10:27:25.947268+0000","last_clean":"2024-09-16T10:27:25.947268+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:25.947268+0000","last_undegraded":"2024-09-16T10:27:25.947268+0000","last_fullsized":"2024-09-16T10:27:25.947268+0000","mapping_epoch":33,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40398848,"num_objects":9863,"num_object_clones":0,"num_object_copies":19726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6330,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10665","reported_seq":19166,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:29.294059+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:29.294059+0000","last_peered":"2024-09-16T10:27:29.294059+0000","last_clean":"2024-09-16T10:27:29.294059+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:29.294059+0000","last_undegraded":"2024-09-16T10:27:29.294059+0000","last_fullsized":"2024-09-16T10:27:29.294059+0000","mapping_epoch":43,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40125382,"num_objects":9798,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10929","reported_seq":11022,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.951918+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:25.951918+0000","last_peered":"2024-09-16T10:27:25.951918+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:25.951918+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415622,"num_objects":10113,"num_object_clones":0,"num_object_copies":20226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10927,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5686},{"shards":"0,2","objects":4029}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10587","reported_seq":10697,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.158382+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:26.158382+0000","last_peered":"2024-09-16T10:27:26.158382+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:26.158382+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40272838,"num_objects":9834,"num_object_clones":0,"num_object_copies":19668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10585,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3902},{"shards":"2","objects":5559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10690","reported_seq":10785,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.138423+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:26.138423+0000","last_peered":"2024-09-16T10:27:26.138423+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:26.138423+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40494022,"num_objects":9888,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10688,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3793},{"shards":"2","objects":5700}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10356","reported_seq":10444,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.901866+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:25.901866+0000","last_peered":"2024-09-16T10:27:25.901866+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:25.901866+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39641571,"num_objects":9679,"num_object_clones":0,"num_object_copies":19358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10355,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5524},{"shards":"0,2","objects":3821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10828","reported_seq":10973,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.971278+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:25.971278+0000","last_peered":"2024-09-16T10:27:25.971278+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:25.971278+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41313705,"num_objects":10089,"num_object_clones":0,"num_object_copies":20178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10825,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5758},{"shards":"0,2","objects":3970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10447","reported_seq":10544,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.778996+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:25.778996+0000","last_peered":"2024-09-16T10:27:25.778996+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:25.778996+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39703494,"num_objects":9695,"num_object_clones":0,"num_object_copies":19390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10445,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5536},{"shards":"0,2","objects":3792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10806","reported_seq":10921,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.814993+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:25.814993+0000","last_peered":"2024-09-16T10:27:25.814993+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:25.814993+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41330572,"num_objects":10094,"num_object_clones":0,"num_object_copies":20188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10803,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5710},{"shards":"0,2","objects":4043}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":649122318,"num_objects":158494,"num_object_clones":0,"num_object_copies":316988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42886,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":158494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147811,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5703,"num_bytes_recovered":23073251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1336676352,"data_stored":1336585109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160746,"ondisk_log_size":160746,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018939,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716152,"kb_used_data":454000,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93655688,"statfs":{"total":96636764160,"available":95903424512,"internally_reserved":0,"allocated":464896000,"data_stored":464638294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":268431024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.089},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1469999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":3.1890000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564668,"kb_used_data":308276,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93807172,"statfs":{"total":96636764160,"available":96058544128,"internally_reserved":0,"allocated":315674624,"data_stored":315344658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":262532778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":2.3220000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.5059999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215167,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":879956,"kb_used_data":545668,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93491884,"statfs":{"total":96636764160,"available":95735689216,"internally_reserved":0,"allocated":558764032,"data_stored":558514012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,9,1,0,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.47899999999999998},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":4.5910000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":5.4359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":557023232,"data_stored":557019619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315219968,"data_stored":315140023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":464433152,"data_stored":464425467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:31.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:32.417 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:32.418 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:32.439 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-16T10:27:30.650900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649122337,"num_objects":158496,"num_object_clones":0,"num_object_copies":316992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42886,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":158496,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147813,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5704,"num_bytes_recovered":23073270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2160776,"kb_used_data":1307944,"kb_used_omap":12,"kb_used_meta":852787,"kb_avail":280954744,"statfs":{"total":289910292480,"available":287697657856,"internally_reserved":0,"allocated":1339334656,"data_stored":1338496964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":873253890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,12,6,1,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18006016,"num_objects":-4396,"num_object_clones":0,"num_object_copies":-8792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-30,"num_objects_misplaced":-206,"num_objects_unfound":0,"num_objects_dirty":-4396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":843776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004884"},"pg_stats":[{"pgid":"2.f","version":"44'10595","reported_seq":13535,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:25.982993+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:25.982993+0000","last_peered":"2024-09-16T10:27:25.982993+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:25.982993+0000","last_undegraded":"2024-09-16T10:27:25.982993+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40439808,"num_objects":9873,"num_object_clones":0,"num_object_copies":19746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":9873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10595,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1435,"num_bytes_recovered":5877760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7116},{"shards":"0,2","objects":2757}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10567","reported_seq":10724,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:26.062761+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:26.062761+0000","last_peered":"2024-09-16T10:27:26.062761+0000","last_clean":"2024-09-16T10:27:26.062761+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:26.062761+0000","last_undegraded":"2024-09-16T10:27:26.062761+0000","last_fullsized":"2024-09-16T10:27:26.062761+0000","mapping_epoch":34,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40529920,"num_objects":9895,"num_object_clones":0,"num_object_copies":19790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6780,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10687","reported_seq":10881,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.192269+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:26.192269+0000","last_peered":"2024-09-16T10:27:26.192269+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:26.192269+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861720,"num_objects":9977,"num_object_clones":0,"num_object_copies":19954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9977,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6744,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3982},{"shards":"2","objects":5646}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10601","reported_seq":10658,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:26.138052+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:26.138052+0000","last_peered":"2024-09-16T10:27:26.138052+0000","last_clean":"2024-09-16T10:27:26.138052+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:26.138052+0000","last_undegraded":"2024-09-16T10:27:26.138052+0000","last_fullsized":"2024-09-16T10:27:26.138052+0000","mapping_epoch":34,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40390656,"num_objects":9861,"num_object_clones":0,"num_object_copies":19722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10602","reported_seq":10759,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:25.920280+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:25.920280+0000","last_peered":"2024-09-16T10:27:25.920280+0000","last_clean":"2024-09-16T10:27:25.920280+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:25.920280+0000","last_undegraded":"2024-09-16T10:27:25.920280+0000","last_fullsized":"2024-09-16T10:27:25.920280+0000","mapping_epoch":33,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40562688,"num_objects":9903,"num_object_clones":0,"num_object_copies":19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10522","reported_seq":10685,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.938947+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:25.938947+0000","last_peered":"2024-09-16T10:27:25.938947+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:25.938947+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40341504,"num_objects":9849,"num_object_clones":0,"num_object_copies":19698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5632},{"shards":"0,2","objects":3883}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10837","reported_seq":11019,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.947221+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:25.947221+0000","last_peered":"2024-09-16T10:27:25.947221+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:25.947221+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41299968,"num_objects":10083,"num_object_clones":0,"num_object_copies":20166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7079,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5684},{"shards":"0,2","objects":4029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10527","reported_seq":10702,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:25.947268+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:25.947268+0000","last_peered":"2024-09-16T10:27:25.947268+0000","last_clean":"2024-09-16T10:27:25.947268+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:25.947268+0000","last_undegraded":"2024-09-16T10:27:25.947268+0000","last_fullsized":"2024-09-16T10:27:25.947268+0000","mapping_epoch":33,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40398848,"num_objects":9863,"num_object_clones":0,"num_object_copies":19726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6330,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10665","reported_seq":19166,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:29.294059+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:29.294059+0000","last_peered":"2024-09-16T10:27:29.294059+0000","last_clean":"2024-09-16T10:27:29.294059+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:29.294059+0000","last_undegraded":"2024-09-16T10:27:29.294059+0000","last_fullsized":"2024-09-16T10:27:29.294059+0000","mapping_epoch":43,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40125382,"num_objects":9798,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'10929","reported_seq":11022,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.951918+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:25.951918+0000","last_peered":"2024-09-16T10:27:25.951918+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:25.951918+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41415622,"num_objects":10113,"num_object_clones":0,"num_object_copies":20226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10927,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5686},{"shards":"0,2","objects":4029}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10587","reported_seq":10697,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.158382+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:26.158382+0000","last_peered":"2024-09-16T10:27:26.158382+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:26.158382+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40272838,"num_objects":9834,"num_object_clones":0,"num_object_copies":19668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10585,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3902},{"shards":"2","objects":5559}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10690","reported_seq":10785,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:26.138423+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:26.138423+0000","last_peered":"2024-09-16T10:27:26.138423+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:26.138423+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40494022,"num_objects":9888,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10688,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3793},{"shards":"2","objects":5700}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10356","reported_seq":10444,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.901866+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:25.901866+0000","last_peered":"2024-09-16T10:27:25.901866+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:25.901866+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'300","ondisk_log_start":"35'300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39641571,"num_objects":9679,"num_object_clones":0,"num_object_copies":19358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10355,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5524},{"shards":"0,2","objects":3821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10828","reported_seq":10973,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.971278+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:25.971278+0000","last_peered":"2024-09-16T10:27:25.971278+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:25.971278+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41313705,"num_objects":10089,"num_object_clones":0,"num_object_copies":20178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4334,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10825,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5758},{"shards":"0,2","objects":3970}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10447","reported_seq":10544,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.778996+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:25.778996+0000","last_peered":"2024-09-16T10:27:25.778996+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:25.778996+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39703494,"num_objects":9695,"num_object_clones":0,"num_object_copies":19390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10445,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5536},{"shards":"0,2","objects":3792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10806","reported_seq":10921,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:25.814993+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:25.814993+0000","last_peered":"2024-09-16T10:27:25.814993+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:25.814993+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41330572,"num_objects":10094,"num_object_clones":0,"num_object_copies":20188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4387,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10803,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5710},{"shards":"0,2","objects":4043}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":649122318,"num_objects":158494,"num_object_clones":0,"num_object_copies":316988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42886,"num_objects_misplaced":2757,"num_objects_unfound":0,"num_objects_dirty":158494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147811,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5703,"num_bytes_recovered":23073251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1336676352,"data_stored":1336585109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160746,"ondisk_log_size":160746,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018939,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716152,"kb_used_data":454000,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93655688,"statfs":{"total":96636764160,"available":95903424512,"internally_reserved":0,"allocated":464896000,"data_stored":464638294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":268431024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.089},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1469999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":3.1890000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564668,"kb_used_data":308276,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93807172,"statfs":{"total":96636764160,"available":96058544128,"internally_reserved":0,"allocated":315674624,"data_stored":315344658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":262532778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":2.3220000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.5059999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215167,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":879956,"kb_used_data":545668,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93491884,"statfs":{"total":96636764160,"available":95735689216,"internally_reserved":0,"allocated":558764032,"data_stored":558514012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,9,1,0,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.47899999999999998},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":4.5910000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":5.4359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":557023232,"data_stored":557019619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315219968,"data_stored":315140023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":464433152,"data_stored":464425467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:32.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:32.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'10595", 'reported_seq': 13535, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:25.982993+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:25.982993+0000', 'last_peered': '2024-09-16T10:27:25.982993+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:25.982993+0000', 'last_undegraded': '2024-09-16T10:27:25.982993+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'500", 'ondisk_log_start': "35'500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40439808, 'num_objects': 9873, 'num_object_clones': 0, 'num_object_copies': 19746, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2757, 'num_objects_unfound': 0, 'num_objects_dirty': 9873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10595, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1435, 'num_bytes_recovered': 5877760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7116}, {'shards': '0,2', 'objects': 2757}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:32.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:32.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'10687", 'reported_seq': 10881, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:26.192269+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:26.192269+0000', 'last_peered': '2024-09-16T10:27:26.192269+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:26.192269+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'600", 'ondisk_log_start': "35'600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40861720, 'num_objects': 9977, 'num_object_clones': 0, 'num_object_copies': 19954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4332, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9977, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6744, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3982}, {'shards': '2', 'objects': 5646}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:32.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:32.441 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'10522", 'reported_seq': 10685, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.938947+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:25.938947+0000', 'last_peered': '2024-09-16T10:27:25.938947+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:25.938947+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'500", 'ondisk_log_start': "35'500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40341504, 'num_objects': 9849, 'num_object_clones': 0, 'num_object_copies': 19698, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4219, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7147, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5632}, {'shards': '0,2', 'objects': 3883}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'10837", 'reported_seq': 11019, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.947221+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:25.947221+0000', 'last_peered': '2024-09-16T10:27:25.947221+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:25.947221+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41299968, 'num_objects': 10083, 'num_object_clones': 0, 'num_object_copies': 20166, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4401, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7079, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5684}, {'shards': '0,2', 'objects': 4029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'10929", 'reported_seq': 11022, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.951918+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:25.951918+0000', 'last_peered': '2024-09-16T10:27:25.951918+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:25.951918+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'900", 'ondisk_log_start': "35'900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41415622, 'num_objects': 10113, 'num_object_clones': 0, 'num_object_copies': 20226, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4429, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10927, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5686}, {'shards': '0,2', 'objects': 4029}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'10587", 'reported_seq': 10697, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:26.158382+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:26.158382+0000', 'last_peered': '2024-09-16T10:27:26.158382+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:26.158382+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'500", 'ondisk_log_start': "35'500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40272838, 'num_objects': 9834, 'num_object_clones': 0, 'num_object_copies': 19668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4277, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10585, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3902}, {'shards': '2', 'objects': 5559}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'10690", 'reported_seq': 10785, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:26.138423+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:26.138423+0000', 'last_peered': '2024-09-16T10:27:26.138423+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:26.138423+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'600", 'ondisk_log_start': "35'600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40494022, 'num_objects': 9888, 'num_object_clones': 0, 'num_object_copies': 19776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4190, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10688, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3793}, {'shards': '2', 'objects': 5700}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:32.442 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'10356", 'reported_seq': 10444, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.901866+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:25.901866+0000', 'last_peered': '2024-09-16T10:27:25.901866+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:25.901866+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'300", 'ondisk_log_start': "35'300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39641571, 'num_objects': 9679, 'num_object_clones': 0, 'num_object_copies': 19358, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4156, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10355, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5524}, {'shards': '0,2', 'objects': 3821}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'10828", 'reported_seq': 10973, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.971278+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:25.971278+0000', 'last_peered': '2024-09-16T10:27:25.971278+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:25.971278+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41313705, 'num_objects': 10089, 'num_object_clones': 0, 'num_object_copies': 20178, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4334, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10825, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5758}, {'shards': '0,2', 'objects': 3970}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'10447", 'reported_seq': 10544, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.778996+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:25.778996+0000', 'last_peered': '2024-09-16T10:27:25.778996+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:25.778996+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'400", 'ondisk_log_start': "35'400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39703494, 'num_objects': 9695, 'num_object_clones': 0, 'num_object_copies': 19390, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4161, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10445, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5536}, {'shards': '0,2', 'objects': 3792}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:32.443 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'10806", 'reported_seq': 10921, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:25.814993+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:25.814993+0000', 'last_peered': '2024-09-16T10:27:25.814993+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:25.814993+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41330572, 'num_objects': 10094, 'num_object_clones': 0, 'num_object_copies': 20188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4387, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 10094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10803, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5710}, {'shards': '0,2', 'objects': 4043}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:32.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:33.132 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:33.132 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42845/313200 objects degraded (13.680%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":341,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":156600,"data_bytes":641356321,"bytes_used":2210607104,"bytes_avail":287699685376,"bytes_total":289910292480,"degraded_objects":42845,"degraded_total":313200,"degraded_ratio":0.13679757343550447,"misplaced_objects":2629,"misplaced_total":313200,"misplaced_ratio":0.0083939974457215832,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":113963,"recovering_keys_per_sec":0,"num_objects_recovered":405,"num_bytes_recovered":1368064,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":515},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:33.151 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:33.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:33.715 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:33.718 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:33.750 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-16T10:27:32.651364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":641356321,"num_objects":156600,"num_object_clones":0,"num_object_copies":313200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42845,"num_objects_misplaced":2629,"num_objects_unfound":0,"num_objects_dirty":156600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149709,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5873,"num_bytes_recovered":23597558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160645,"ondisk_log_size":160645,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2158796,"kb_used_data":1297772,"kb_used_omap":12,"kb_used_meta":860979,"kb_avail":280956724,"statfs":{"total":289910292480,"available":287699685376,"internally_reserved":0,"allocated":1328918528,"data_stored":1328080836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":881642498},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,3,3,5,1,0,0,0,0,0,8],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25333760,"num_objects":-6185,"num_object_clones":0,"num_object_copies":-12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-71,"num_objects_misplaced":-334,"num_objects_unfound":0,"num_objects_dirty":-6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":405,"num_bytes_recovered":1368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004426"},"pg_stats":[{"pgid":"2.f","version":"44'10727","reported_seq":13922,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:30.993122+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:30.993122+0000","last_peered":"2024-09-16T10:27:30.993122+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:30.993122+0000","last_undegraded":"2024-09-16T10:27:30.993122+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39899136,"num_objects":9741,"num_object_clones":0,"num_object_copies":19482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2629,"num_objects_unfound":0,"num_objects_dirty":9741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10727,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1563,"num_bytes_recovered":6402048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7112},{"shards":"0,2","objects":2629}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10701","reported_seq":10857,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:31.201152+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:31.201152+0000","last_peered":"2024-09-16T10:27:31.201152+0000","last_clean":"2024-09-16T10:27:31.201152+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:31.201152+0000","last_undegraded":"2024-09-16T10:27:31.201152+0000","last_fullsized":"2024-09-16T10:27:31.201152+0000","mapping_epoch":34,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39981056,"num_objects":9761,"num_object_clones":0,"num_object_copies":19522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6914,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10823","reported_seq":11017,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:31.172068+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:31.172068+0000","last_peered":"2024-09-16T10:27:31.172068+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:31.172068+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40304664,"num_objects":9841,"num_object_clones":0,"num_object_copies":19682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9841,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6880,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3846},{"shards":"2","objects":5510}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10733","reported_seq":10790,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:31.216107+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:31.216107+0000","last_peered":"2024-09-16T10:27:31.216107+0000","last_clean":"2024-09-16T10:27:31.216107+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:31.216107+0000","last_undegraded":"2024-09-16T10:27:31.216107+0000","last_fullsized":"2024-09-16T10:27:31.216107+0000","mapping_epoch":34,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39849984,"num_objects":9729,"num_object_clones":0,"num_object_copies":19458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10718","reported_seq":10875,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:30.952277+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:30.952277+0000","last_peered":"2024-09-16T10:27:30.952277+0000","last_clean":"2024-09-16T10:27:30.952277+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:30.952277+0000","last_undegraded":"2024-09-16T10:27:30.952277+0000","last_fullsized":"2024-09-16T10:27:30.952277+0000","mapping_epoch":33,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40087552,"num_objects":9787,"num_object_clones":0,"num_object_copies":19574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7360,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10621","reported_seq":10803,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:30.824898+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:30.824898+0000","last_peered":"2024-09-16T10:27:30.824898+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:30.824898+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39936000,"num_objects":9750,"num_object_clones":0,"num_object_copies":19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7246,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5542},{"shards":"0,2","objects":3784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'10962","reported_seq":11162,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:30.970679+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:30.970679+0000","last_peered":"2024-09-16T10:27:30.970679+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:30.970679+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40787968,"num_objects":9958,"num_object_clones":0,"num_object_copies":19916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7204,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5568},{"shards":"0,2","objects":3904}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10650","reported_seq":10825,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:30.994309+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:30.994309+0000","last_peered":"2024-09-16T10:27:30.994309+0000","last_clean":"2024-09-16T10:27:30.994309+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:30.994309+0000","last_undegraded":"2024-09-16T10:27:30.994309+0000","last_fullsized":"2024-09-16T10:27:30.994309+0000","mapping_epoch":33,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39895040,"num_objects":9740,"num_object_clones":0,"num_object_copies":19480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10665","reported_seq":19166,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:29.294059+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:29.294059+0000","last_peered":"2024-09-16T10:27:29.294059+0000","last_clean":"2024-09-16T10:27:29.294059+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:29.294059+0000","last_undegraded":"2024-09-16T10:27:29.294059+0000","last_fullsized":"2024-09-16T10:27:29.294059+0000","mapping_epoch":43,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40125382,"num_objects":9798,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10663,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11070","reported_seq":11171,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:30.967809+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:30.967809+0000","last_peered":"2024-09-16T10:27:30.967809+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:30.967809+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40838086,"num_objects":9972,"num_object_clones":0,"num_object_copies":19944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11068,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5550},{"shards":"0,2","objects":3888}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10710","reported_seq":10820,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:31.161698+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:31.161698+0000","last_peered":"2024-09-16T10:27:31.161698+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:31.161698+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39769030,"num_objects":9711,"num_object_clones":0,"num_object_copies":19422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10708,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3779},{"shards":"2","objects":5436}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10801","reported_seq":10896,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:31.172360+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:31.172360+0000","last_peered":"2024-09-16T10:27:31.172360+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:31.172360+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40039366,"num_objects":9777,"num_object_clones":0,"num_object_copies":19554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10799,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3682},{"shards":"2","objects":5589}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10478","reported_seq":10580,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:30.946913+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:30.946913+0000","last_peered":"2024-09-16T10:27:30.946913+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:30.946913+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'400","ondisk_log_start":"35'400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39141859,"num_objects":9557,"num_object_clones":0,"num_object_copies":19114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10477,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5409},{"shards":"0,2","objects":3699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'10953","reported_seq":11101,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:30.957390+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:30.957390+0000","last_peered":"2024-09-16T10:27:30.957390+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:30.957390+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40801705,"num_objects":9964,"num_object_clones":0,"num_object_copies":19928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5635},{"shards":"0,2","objects":3845}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10581","reported_seq":10682,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:30.870287+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:30.870287+0000","last_peered":"2024-09-16T10:27:30.870287+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:30.870287+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'500","ondisk_log_start":"35'500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39154630,"num_objects":9561,"num_object_clones":0,"num_object_copies":19122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5405},{"shards":"0,2","objects":3658}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'10949","reported_seq":11076,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:30.929469+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:30.929469+0000","last_peered":"2024-09-16T10:27:30.929469+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:30.929469+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40744844,"num_objects":9951,"num_object_clones":0,"num_object_copies":19902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10946,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5573},{"shards":"0,2","objects":3900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":641356302,"num_objects":156598,"num_object_clones":0,"num_object_copies":313196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42845,"num_objects_misplaced":2629,"num_objects_unfound":0,"num_objects_dirty":156598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149707,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5872,"num_bytes_recovered":23597539,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1326329856,"data_stored":1326238613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160642,"ondisk_log_size":160642,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018940,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716368,"kb_used_data":450116,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93655472,"statfs":{"total":96636764160,"available":95903203328,"internally_reserved":0,"allocated":460918784,"data_stored":460661078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":272625328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.307}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":2.363}]}]},{"osd":1,"up_from":40,"seq":171798691884,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564668,"kb_used_data":308276,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93807172,"statfs":{"total":96636764160,"available":96058544128,"internally_reserved":0,"allocated":315674624,"data_stored":315344658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":262532778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":2.3220000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.5059999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215168,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877760,"kb_used_data":539380,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93494080,"statfs":{"total":96636764160,"available":95737937920,"internally_reserved":0,"allocated":552325120,"data_stored":552075100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":346484392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2,2,0,0,0,0,0,0,8],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.55500000000000005},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":5.4359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":550830080,"data_stored":550826467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":315219968,"data_stored":315140023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":460279808,"data_stored":460272123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:36.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:37.305 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:37.305 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:37.330 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-16T10:27:36.652935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632705569,"num_objects":154488,"num_object_clones":0,"num_object_copies":308976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42823,"num_objects_misplaced":2532,"num_objects_unfound":0,"num_objects_dirty":154488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151821,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5992,"num_bytes_recovered":23994870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160857,"ondisk_log_size":160857,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2143696,"kb_used_data":1286776,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":280971824,"statfs":{"total":289910292480,"available":287715147776,"internally_reserved":0,"allocated":1317658624,"data_stored":1316820932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,6,7,3,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25092096,"num_objects":-6126,"num_object_clones":0,"num_object_copies":-12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-83,"num_objects_misplaced":-330,"num_objects_unfound":0,"num_objects_dirty":-6126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":1351680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003844"},"pg_stats":[{"pgid":"2.f","version":"44'10854","reported_seq":14244,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:36.035308+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:36.035308+0000","last_peered":"2024-09-16T10:27:36.035308+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:36.035308+0000","last_undegraded":"2024-09-16T10:27:36.035308+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39378944,"num_objects":9614,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2532,"num_objects_unfound":0,"num_objects_dirty":9614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10854,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1660,"num_bytes_recovered":6799360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7082},{"shards":"0,2","objects":2532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10827","reported_seq":10984,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:36.183204+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:36.183204+0000","last_peered":"2024-09-16T10:27:36.183204+0000","last_clean":"2024-09-16T10:27:36.183204+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:36.183204+0000","last_undegraded":"2024-09-16T10:27:36.183204+0000","last_fullsized":"2024-09-16T10:27:36.183204+0000","mapping_epoch":34,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39464960,"num_objects":9635,"num_object_clones":0,"num_object_copies":19270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7040,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10959","reported_seq":11156,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.223662+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:36.223662+0000","last_peered":"2024-09-16T10:27:36.223662+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:36.223662+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39747608,"num_objects":9705,"num_object_clones":0,"num_object_copies":19410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9705,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7016,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3710},{"shards":"2","objects":5375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10863","reported_seq":10920,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:36.197558+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:36.197558+0000","last_peered":"2024-09-16T10:27:36.197558+0000","last_clean":"2024-09-16T10:27:36.197558+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:36.197558+0000","last_undegraded":"2024-09-16T10:27:36.197558+0000","last_fullsized":"2024-09-16T10:27:36.197558+0000","mapping_epoch":34,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39317504,"num_objects":9599,"num_object_clones":0,"num_object_copies":19198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10863,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10848","reported_seq":11005,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:35.877848+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:35.877848+0000","last_peered":"2024-09-16T10:27:35.877848+0000","last_clean":"2024-09-16T10:27:35.877848+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:35.877848+0000","last_undegraded":"2024-09-16T10:27:35.877848+0000","last_fullsized":"2024-09-16T10:27:35.877848+0000","mapping_epoch":33,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39555072,"num_objects":9657,"num_object_clones":0,"num_object_copies":19314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10752","reported_seq":10936,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.978039+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:35.978039+0000","last_peered":"2024-09-16T10:27:35.978039+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:35.978039+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39399424,"num_objects":9619,"num_object_clones":0,"num_object_copies":19238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7377,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5412},{"shards":"0,2","objects":3653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11082","reported_seq":11294,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.916644+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:35.916644+0000","last_peered":"2024-09-16T10:27:35.916644+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:35.916644+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40296448,"num_objects":9838,"num_object_clones":0,"num_object_copies":19676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7324,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5454},{"shards":"0,2","objects":3784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10761","reported_seq":10935,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:36.007321+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:36.007321+0000","last_peered":"2024-09-16T10:27:36.007321+0000","last_clean":"2024-09-16T10:27:36.007321+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:36.007321+0000","last_undegraded":"2024-09-16T10:27:36.007321+0000","last_fullsized":"2024-09-16T10:27:36.007321+0000","mapping_epoch":33,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39440384,"num_objects":9629,"num_object_clones":0,"num_object_copies":19258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6564,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10789","reported_seq":19292,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:34.349332+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:34.349332+0000","last_peered":"2024-09-16T10:27:34.349332+0000","last_clean":"2024-09-16T10:27:34.349332+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:34.349332+0000","last_undegraded":"2024-09-16T10:27:34.349332+0000","last_fullsized":"2024-09-16T10:27:34.349332+0000","mapping_epoch":43,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39617478,"num_objects":9674,"num_object_clones":0,"num_object_copies":19348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11200","reported_seq":11304,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.999784+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:35.999784+0000","last_peered":"2024-09-16T10:27:35.999784+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:35.999784+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40305606,"num_objects":9842,"num_object_clones":0,"num_object_copies":19684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11198,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5421},{"shards":"0,2","objects":3758}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10840","reported_seq":10952,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.161198+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:36.161198+0000","last_peered":"2024-09-16T10:27:36.161198+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:36.161198+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39236550,"num_objects":9581,"num_object_clones":0,"num_object_copies":19162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10838,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3649},{"shards":"2","objects":5307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10938","reported_seq":11035,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.172675+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:36.172675+0000","last_peered":"2024-09-16T10:27:36.172675+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:36.172675+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39478214,"num_objects":9640,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10936,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3545},{"shards":"2","objects":5453}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10614","reported_seq":10728,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.957197+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:35.957197+0000","last_peered":"2024-09-16T10:27:35.957197+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:35.957197+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38584803,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5279},{"shards":"0,2","objects":3563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11107","reported_seq":11260,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.691116+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:35.691116+0000","last_peered":"2024-09-16T10:27:35.691116+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:35.691116+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40170921,"num_objects":9810,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11104,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5483},{"shards":"0,2","objects":3691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10721","reported_seq":10830,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.023233+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:36.023233+0000","last_peered":"2024-09-16T10:27:36.023233+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:36.023233+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38581190,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5268},{"shards":"0,2","objects":3518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11099","reported_seq":11226,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.015147+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:36.015147+0000","last_peered":"2024-09-16T10:27:36.015147+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:36.015147+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40130444,"num_objects":9801,"num_object_clones":0,"num_object_copies":19602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11096,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5423},{"shards":"0,2","objects":3750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":632705550,"num_objects":154486,"num_object_clones":0,"num_object_copies":308972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42823,"num_objects_misplaced":2532,"num_objects_unfound":0,"num_objects_dirty":154486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151819,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5991,"num_bytes_recovered":23994851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1314959360,"data_stored":1314868117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160854,"ondisk_log_size":160854,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018941,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716312,"kb_used_data":445968,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93655528,"statfs":{"total":96636764160,"available":95903260672,"internally_reserved":0,"allocated":456671232,"data_stored":456413526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":276819632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.274}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":4.5149999999999997},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":5.8129999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691885,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551800,"kb_used_data":307696,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93820040,"statfs":{"total":96636764160,"available":96071720960,"internally_reserved":0,"allocated":315080704,"data_stored":314750738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":249949866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.19800000000000001},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.7430000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":3.1120000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":7.7400000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215169,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":875584,"kb_used_data":533112,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93496256,"statfs":{"total":96636764160,"available":95740166144,"internally_reserved":0,"allocated":545906688,"data_stored":545656668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":350678696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,5,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":1.5580000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":4.3810000000000002},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.499}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":544247808,"data_stored":544244195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":314658816,"data_stored":314578871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":456052736,"data_stored":456045051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:37.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:37.860 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:37.860 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:37.884 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-16T10:27:36.652935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632705569,"num_objects":154488,"num_object_clones":0,"num_object_copies":308976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42823,"num_objects_misplaced":2532,"num_objects_unfound":0,"num_objects_dirty":154488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151821,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5992,"num_bytes_recovered":23994870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160857,"ondisk_log_size":160857,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2143696,"kb_used_data":1286776,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":280971824,"statfs":{"total":289910292480,"available":287715147776,"internally_reserved":0,"allocated":1317658624,"data_stored":1316820932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,6,7,3,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25092096,"num_objects":-6126,"num_object_clones":0,"num_object_copies":-12252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-83,"num_objects_misplaced":-330,"num_objects_unfound":0,"num_objects_dirty":-6126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":1351680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003844"},"pg_stats":[{"pgid":"2.f","version":"44'10854","reported_seq":14244,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:36.035308+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:36.035308+0000","last_peered":"2024-09-16T10:27:36.035308+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:36.035308+0000","last_undegraded":"2024-09-16T10:27:36.035308+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39378944,"num_objects":9614,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2532,"num_objects_unfound":0,"num_objects_dirty":9614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10854,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1660,"num_bytes_recovered":6799360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7082},{"shards":"0,2","objects":2532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10827","reported_seq":10984,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:36.183204+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:36.183204+0000","last_peered":"2024-09-16T10:27:36.183204+0000","last_clean":"2024-09-16T10:27:36.183204+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:36.183204+0000","last_undegraded":"2024-09-16T10:27:36.183204+0000","last_fullsized":"2024-09-16T10:27:36.183204+0000","mapping_epoch":34,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39464960,"num_objects":9635,"num_object_clones":0,"num_object_copies":19270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7040,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10959","reported_seq":11156,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.223662+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:36.223662+0000","last_peered":"2024-09-16T10:27:36.223662+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:36.223662+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39747608,"num_objects":9705,"num_object_clones":0,"num_object_copies":19410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9705,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7016,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3710},{"shards":"2","objects":5375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10863","reported_seq":10920,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:36.197558+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:36.197558+0000","last_peered":"2024-09-16T10:27:36.197558+0000","last_clean":"2024-09-16T10:27:36.197558+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:36.197558+0000","last_undegraded":"2024-09-16T10:27:36.197558+0000","last_fullsized":"2024-09-16T10:27:36.197558+0000","mapping_epoch":34,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39317504,"num_objects":9599,"num_object_clones":0,"num_object_copies":19198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10863,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10848","reported_seq":11005,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:35.877848+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:35.877848+0000","last_peered":"2024-09-16T10:27:35.877848+0000","last_clean":"2024-09-16T10:27:35.877848+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:35.877848+0000","last_undegraded":"2024-09-16T10:27:35.877848+0000","last_fullsized":"2024-09-16T10:27:35.877848+0000","mapping_epoch":33,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39555072,"num_objects":9657,"num_object_clones":0,"num_object_copies":19314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10752","reported_seq":10936,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.978039+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:35.978039+0000","last_peered":"2024-09-16T10:27:35.978039+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:35.978039+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39399424,"num_objects":9619,"num_object_clones":0,"num_object_copies":19238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7377,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5412},{"shards":"0,2","objects":3653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11082","reported_seq":11294,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.916644+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:35.916644+0000","last_peered":"2024-09-16T10:27:35.916644+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:35.916644+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40296448,"num_objects":9838,"num_object_clones":0,"num_object_copies":19676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7324,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5454},{"shards":"0,2","objects":3784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10761","reported_seq":10935,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:36.007321+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:36.007321+0000","last_peered":"2024-09-16T10:27:36.007321+0000","last_clean":"2024-09-16T10:27:36.007321+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:36.007321+0000","last_undegraded":"2024-09-16T10:27:36.007321+0000","last_fullsized":"2024-09-16T10:27:36.007321+0000","mapping_epoch":33,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39440384,"num_objects":9629,"num_object_clones":0,"num_object_copies":19258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6564,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10789","reported_seq":19292,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:34.349332+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:34.349332+0000","last_peered":"2024-09-16T10:27:34.349332+0000","last_clean":"2024-09-16T10:27:34.349332+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:34.349332+0000","last_undegraded":"2024-09-16T10:27:34.349332+0000","last_fullsized":"2024-09-16T10:27:34.349332+0000","mapping_epoch":43,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39617478,"num_objects":9674,"num_object_clones":0,"num_object_copies":19348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11200","reported_seq":11304,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.999784+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:35.999784+0000","last_peered":"2024-09-16T10:27:35.999784+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:35.999784+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40305606,"num_objects":9842,"num_object_clones":0,"num_object_copies":19684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11198,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5421},{"shards":"0,2","objects":3758}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10840","reported_seq":10952,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.161198+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:36.161198+0000","last_peered":"2024-09-16T10:27:36.161198+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:36.161198+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39236550,"num_objects":9581,"num_object_clones":0,"num_object_copies":19162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10838,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3649},{"shards":"2","objects":5307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10938","reported_seq":11035,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.172675+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:36.172675+0000","last_peered":"2024-09-16T10:27:36.172675+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:36.172675+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39478214,"num_objects":9640,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10936,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3545},{"shards":"2","objects":5453}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10614","reported_seq":10728,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.957197+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:35.957197+0000","last_peered":"2024-09-16T10:27:35.957197+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:35.957197+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38584803,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5279},{"shards":"0,2","objects":3563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11107","reported_seq":11260,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.691116+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:35.691116+0000","last_peered":"2024-09-16T10:27:35.691116+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:35.691116+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40170921,"num_objects":9810,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11104,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5483},{"shards":"0,2","objects":3691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10721","reported_seq":10830,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.023233+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:36.023233+0000","last_peered":"2024-09-16T10:27:36.023233+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:36.023233+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38581190,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5268},{"shards":"0,2","objects":3518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11099","reported_seq":11226,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.015147+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:36.015147+0000","last_peered":"2024-09-16T10:27:36.015147+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:36.015147+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40130444,"num_objects":9801,"num_object_clones":0,"num_object_copies":19602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11096,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5423},{"shards":"0,2","objects":3750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":632705550,"num_objects":154486,"num_object_clones":0,"num_object_copies":308972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42823,"num_objects_misplaced":2532,"num_objects_unfound":0,"num_objects_dirty":154486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151819,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5991,"num_bytes_recovered":23994851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1314959360,"data_stored":1314868117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160854,"ondisk_log_size":160854,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018941,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716312,"kb_used_data":445968,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93655528,"statfs":{"total":96636764160,"available":95903260672,"internally_reserved":0,"allocated":456671232,"data_stored":456413526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":276819632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.274}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":4.5149999999999997},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":5.8129999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691885,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551800,"kb_used_data":307696,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93820040,"statfs":{"total":96636764160,"available":96071720960,"internally_reserved":0,"allocated":315080704,"data_stored":314750738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":249949866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.19800000000000001},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.7430000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":3.1120000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":7.7400000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215169,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":875584,"kb_used_data":533112,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93496256,"statfs":{"total":96636764160,"available":95740166144,"internally_reserved":0,"allocated":545906688,"data_stored":545656668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":350678696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,5,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":1.5580000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":4.3810000000000002},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.499}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":544247808,"data_stored":544244195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":314658816,"data_stored":314578871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":456052736,"data_stored":456045051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:37.885 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:37.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'10854", 'reported_seq': 14244, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:36.035308+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:36.035308+0000', 'last_peered': '2024-09-16T10:27:36.035308+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:36.035308+0000', 'last_undegraded': '2024-09-16T10:27:36.035308+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39378944, 'num_objects': 9614, 'num_object_clones': 0, 'num_object_copies': 19228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2532, 'num_objects_unfound': 0, 'num_objects_dirty': 9614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10854, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1660, 'num_bytes_recovered': 6799360, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7082}, {'shards': '0,2', 'objects': 2532}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:37.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:37.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'10959", 'reported_seq': 11156, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:36.223662+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:36.223662+0000', 'last_peered': '2024-09-16T10:27:36.223662+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:36.223662+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'900", 'ondisk_log_start': "35'900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39747608, 'num_objects': 9705, 'num_object_clones': 0, 'num_object_copies': 19410, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4331, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9705, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7016, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3710}, {'shards': '2', 'objects': 5375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:37.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:37.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'10752", 'reported_seq': 10936, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:35.978039+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:35.978039+0000', 'last_peered': '2024-09-16T10:27:35.978039+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:35.978039+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'700", 'ondisk_log_start': "35'700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39399424, 'num_objects': 9619, 'num_object_clones': 0, 'num_object_copies': 19238, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4209, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9619, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7377, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5412}, {'shards': '0,2', 'objects': 3653}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:37.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:37.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'11082", 'reported_seq': 11294, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:35.916644+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:35.916644+0000', 'last_peered': '2024-09-16T10:27:35.916644+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:35.916644+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'1000", 'ondisk_log_start': "35'1000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40296448, 'num_objects': 9838, 'num_object_clones': 0, 'num_object_copies': 19676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4386, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9838, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7324, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5454}, {'shards': '0,2', 'objects': 3784}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:37.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:37.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'11200", 'reported_seq': 11304, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:35.999784+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:35.999784+0000', 'last_peered': '2024-09-16T10:27:35.999784+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:35.999784+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'1100", 'ondisk_log_start': "35'1100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40305606, 'num_objects': 9842, 'num_object_clones': 0, 'num_object_copies': 19684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4423, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11198, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5421}, {'shards': '0,2', 'objects': 3758}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'10840", 'reported_seq': 10952, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:36.161198+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:36.161198+0000', 'last_peered': '2024-09-16T10:27:36.161198+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:36.161198+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39236550, 'num_objects': 9581, 'num_object_clones': 0, 'num_object_copies': 19162, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10838, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3649}, {'shards': '2', 'objects': 5307}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'10938", 'reported_seq': 11035, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:36.172675+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:36.172675+0000', 'last_peered': '2024-09-16T10:27:36.172675+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:36.172675+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'900", 'ondisk_log_start': "35'900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39478214, 'num_objects': 9640, 'num_object_clones': 0, 'num_object_copies': 19280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4189, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10936, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3545}, {'shards': '2', 'objects': 5453}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'10614", 'reported_seq': 10728, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:35.957197+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:35.957197+0000', 'last_peered': '2024-09-16T10:27:35.957197+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:35.957197+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'600", 'ondisk_log_start': "35'600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38584803, 'num_objects': 9421, 'num_object_clones': 0, 'num_object_copies': 18842, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10613, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5279}, {'shards': '0,2', 'objects': 3563}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'11107", 'reported_seq': 11260, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:35.691116+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:35.691116+0000', 'last_peered': '2024-09-16T10:27:35.691116+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:35.691116+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'1100", 'ondisk_log_start': "35'1100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40170921, 'num_objects': 9810, 'num_object_clones': 0, 'num_object_copies': 19620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4330, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11104, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5483}, {'shards': '0,2', 'objects': 3691}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'10721", 'reported_seq': 10830, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:36.023233+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:36.023233+0000', 'last_peered': '2024-09-16T10:27:36.023233+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:36.023233+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'700", 'ondisk_log_start': "35'700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38581190, 'num_objects': 9421, 'num_object_clones': 0, 'num_object_copies': 18842, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4155, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9421, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10719, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5268}, {'shards': '0,2', 'objects': 3518}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:37.887 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'11099", 'reported_seq': 11226, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:36.015147+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:36.015147+0000', 'last_peered': '2024-09-16T10:27:36.015147+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:36.015147+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'1000", 'ondisk_log_start': "35'1000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40130444, 'num_objects': 9801, 'num_object_clones': 0, 'num_object_copies': 19602, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4381, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11096, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5423}, {'shards': '0,2', 'objects': 3750}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:37.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:38.557 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:38.558 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42823/308976 objects degraded (13.860%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":347,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":154488,"data_bytes":632705569,"bytes_used":2195144704,"bytes_avail":287715147776,"bytes_total":289910292480,"degraded_objects":42823,"degraded_total":308976,"degraded_ratio":0.13859652529646316,"misplaced_objects":2532,"misplaced_total":308976,"misplaced_ratio":0.0081948112474755329,"recovering_objects_per_sec":34,"recovering_bytes_per_sec":112603,"recovering_keys_per_sec":0,"num_objects_recovered":413,"num_bytes_recovered":1351680,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":510},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:38.582 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:38.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:39.117 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:39.139 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-16T10:27:38.653693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":632705569,"num_objects":154488,"num_object_clones":0,"num_object_copies":308976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42823,"num_objects_misplaced":2532,"num_objects_unfound":0,"num_objects_dirty":154488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151821,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5992,"num_bytes_recovered":23994870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160857,"ondisk_log_size":160857,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2143696,"kb_used_data":1286776,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":280971824,"statfs":{"total":289910292480,"available":287715147776,"internally_reserved":0,"allocated":1317658624,"data_stored":1316820932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,6,7,3,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16949248,"num_objects":-4138,"num_object_clones":0,"num_object_copies":-8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-63,"num_objects_misplaced":-225,"num_objects_unfound":0,"num_objects_dirty":-4138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4138,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":921600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003858"},"pg_stats":[{"pgid":"2.f","version":"44'10854","reported_seq":14244,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:36.035308+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:36.035308+0000","last_peered":"2024-09-16T10:27:36.035308+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:36.035308+0000","last_undegraded":"2024-09-16T10:27:36.035308+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39378944,"num_objects":9614,"num_object_clones":0,"num_object_copies":19228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2532,"num_objects_unfound":0,"num_objects_dirty":9614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10854,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1660,"num_bytes_recovered":6799360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7082},{"shards":"0,2","objects":2532}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10827","reported_seq":10984,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:36.183204+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:36.183204+0000","last_peered":"2024-09-16T10:27:36.183204+0000","last_clean":"2024-09-16T10:27:36.183204+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:36.183204+0000","last_undegraded":"2024-09-16T10:27:36.183204+0000","last_fullsized":"2024-09-16T10:27:36.183204+0000","mapping_epoch":34,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39464960,"num_objects":9635,"num_object_clones":0,"num_object_copies":19270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7040,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'10959","reported_seq":11156,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.223662+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:36.223662+0000","last_peered":"2024-09-16T10:27:36.223662+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:36.223662+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39747608,"num_objects":9705,"num_object_clones":0,"num_object_copies":19410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9705,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7016,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3710},{"shards":"2","objects":5375}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10863","reported_seq":10920,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:36.197558+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:36.197558+0000","last_peered":"2024-09-16T10:27:36.197558+0000","last_clean":"2024-09-16T10:27:36.197558+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:36.197558+0000","last_undegraded":"2024-09-16T10:27:36.197558+0000","last_fullsized":"2024-09-16T10:27:36.197558+0000","mapping_epoch":34,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39317504,"num_objects":9599,"num_object_clones":0,"num_object_copies":19198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10863,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10848","reported_seq":11005,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:35.877848+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:35.877848+0000","last_peered":"2024-09-16T10:27:35.877848+0000","last_clean":"2024-09-16T10:27:35.877848+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:35.877848+0000","last_undegraded":"2024-09-16T10:27:35.877848+0000","last_fullsized":"2024-09-16T10:27:35.877848+0000","mapping_epoch":33,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39555072,"num_objects":9657,"num_object_clones":0,"num_object_copies":19314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7490,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10752","reported_seq":10936,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.978039+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:35.978039+0000","last_peered":"2024-09-16T10:27:35.978039+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:35.978039+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39399424,"num_objects":9619,"num_object_clones":0,"num_object_copies":19238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7377,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5412},{"shards":"0,2","objects":3653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11082","reported_seq":11294,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.916644+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:35.916644+0000","last_peered":"2024-09-16T10:27:35.916644+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:35.916644+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40296448,"num_objects":9838,"num_object_clones":0,"num_object_copies":19676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7324,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5454},{"shards":"0,2","objects":3784}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10761","reported_seq":10935,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:36.007321+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:36.007321+0000","last_peered":"2024-09-16T10:27:36.007321+0000","last_clean":"2024-09-16T10:27:36.007321+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:36.007321+0000","last_undegraded":"2024-09-16T10:27:36.007321+0000","last_fullsized":"2024-09-16T10:27:36.007321+0000","mapping_epoch":33,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39440384,"num_objects":9629,"num_object_clones":0,"num_object_copies":19258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6564,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10789","reported_seq":19292,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:34.349332+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:34.349332+0000","last_peered":"2024-09-16T10:27:34.349332+0000","last_clean":"2024-09-16T10:27:34.349332+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:34.349332+0000","last_undegraded":"2024-09-16T10:27:34.349332+0000","last_fullsized":"2024-09-16T10:27:34.349332+0000","mapping_epoch":43,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39617478,"num_objects":9674,"num_object_clones":0,"num_object_copies":19348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11200","reported_seq":11304,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.999784+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:35.999784+0000","last_peered":"2024-09-16T10:27:35.999784+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:35.999784+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40305606,"num_objects":9842,"num_object_clones":0,"num_object_copies":19684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11198,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5421},{"shards":"0,2","objects":3758}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10840","reported_seq":10952,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.161198+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:36.161198+0000","last_peered":"2024-09-16T10:27:36.161198+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:36.161198+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39236550,"num_objects":9581,"num_object_clones":0,"num_object_copies":19162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10838,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3649},{"shards":"2","objects":5307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'10938","reported_seq":11035,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.172675+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:36.172675+0000","last_peered":"2024-09-16T10:27:36.172675+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:36.172675+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39478214,"num_objects":9640,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10936,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3545},{"shards":"2","objects":5453}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10614","reported_seq":10728,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.957197+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:35.957197+0000","last_peered":"2024-09-16T10:27:35.957197+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:35.957197+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'600","ondisk_log_start":"35'600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38584803,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5279},{"shards":"0,2","objects":3563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11107","reported_seq":11260,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:35.691116+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:35.691116+0000","last_peered":"2024-09-16T10:27:35.691116+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:35.691116+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40170921,"num_objects":9810,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11104,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5483},{"shards":"0,2","objects":3691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10721","reported_seq":10830,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.023233+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:36.023233+0000","last_peered":"2024-09-16T10:27:36.023233+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:36.023233+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38581190,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5268},{"shards":"0,2","objects":3518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11099","reported_seq":11226,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:36.015147+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:36.015147+0000","last_peered":"2024-09-16T10:27:36.015147+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:36.015147+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40130444,"num_objects":9801,"num_object_clones":0,"num_object_copies":19602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11096,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5423},{"shards":"0,2","objects":3750}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":632705550,"num_objects":154486,"num_object_clones":0,"num_object_copies":308972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42823,"num_objects_misplaced":2532,"num_objects_unfound":0,"num_objects_dirty":154486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151819,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5991,"num_bytes_recovered":23994851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1314959360,"data_stored":1314868117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160854,"ondisk_log_size":160854,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018941,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716312,"kb_used_data":445968,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93655528,"statfs":{"total":96636764160,"available":95903260672,"internally_reserved":0,"allocated":456671232,"data_stored":456413526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":276819632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.274}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":4.5149999999999997},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":5.8129999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691885,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551800,"kb_used_data":307696,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93820040,"statfs":{"total":96636764160,"available":96071720960,"internally_reserved":0,"allocated":315080704,"data_stored":314750738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":249949866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.19800000000000001},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.7430000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":3.1120000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":7.7400000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215169,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":875584,"kb_used_data":533112,"kb_used_omap":4,"kb_used_meta":342459,"kb_avail":93496256,"statfs":{"total":96636764160,"available":95740166144,"internally_reserved":0,"allocated":545906688,"data_stored":545656668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":350678696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,5,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":1.5580000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.26800000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":4.3810000000000002},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.499}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":544247808,"data_stored":544244195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":314658816,"data_stored":314578871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":456052736,"data_stored":456045051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:42.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:42.709 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:42.715 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:42.738 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-16T10:27:42.655305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":623993377,"num_objects":152361,"num_object_clones":0,"num_object_copies":304722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42818,"num_objects_misplaced":2419,"num_objects_unfound":0,"num_objects_dirty":152361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153948,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6110,"num_bytes_recovered":24457718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2131280,"kb_used_data":1274360,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":280984240,"statfs":{"total":289910292480,"available":287727861760,"internally_reserved":0,"allocated":1304944640,"data_stored":1304106948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,5,2,1,0,0,0,0,3,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25128960,"num_objects":-6135,"num_object_clones":0,"num_object_copies":-12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-68,"num_objects_misplaced":-338,"num_objects_unfound":0,"num_objects_dirty":-6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6135,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":406,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004404"},"pg_stats":[{"pgid":"2.f","version":"44'10989","reported_seq":14603,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:41.074657+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:41.074657+0000","last_peered":"2024-09-16T10:27:41.074657+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:41.074657+0000","last_undegraded":"2024-09-16T10:27:41.074657+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38825984,"num_objects":9479,"num_object_clones":0,"num_object_copies":18958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2419,"num_objects_unfound":0,"num_objects_dirty":9479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10989,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":7262208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7060},{"shards":"0,2","objects":2419}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10960","reported_seq":11117,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.226808+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:41.226808+0000","last_peered":"2024-09-16T10:27:41.226808+0000","last_clean":"2024-09-16T10:27:41.226808+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:41.226808+0000","last_undegraded":"2024-09-16T10:27:41.226808+0000","last_fullsized":"2024-09-16T10:27:41.226808+0000","mapping_epoch":34,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38920192,"num_objects":9502,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7173,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11110","reported_seq":11309,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.188511+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:41.188511+0000","last_peered":"2024-09-16T10:27:41.188511+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:41.188511+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39129112,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7167,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3559},{"shards":"2","objects":5225}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10983","reported_seq":11040,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.168573+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:41.168573+0000","last_peered":"2024-09-16T10:27:41.168573+0000","last_clean":"2024-09-16T10:27:41.168573+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:41.168573+0000","last_undegraded":"2024-09-16T10:27:41.168573+0000","last_fullsized":"2024-09-16T10:27:41.168573+0000","mapping_epoch":34,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38825984,"num_objects":9479,"num_object_clones":0,"num_object_copies":18958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10983,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10975","reported_seq":11132,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:40.967780+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:40.967780+0000","last_peered":"2024-09-16T10:27:40.967780+0000","last_clean":"2024-09-16T10:27:40.967780+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:40.967780+0000","last_undegraded":"2024-09-16T10:27:40.967780+0000","last_fullsized":"2024-09-16T10:27:40.967780+0000","mapping_epoch":33,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":9530,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7617,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10879","reported_seq":11065,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.011635+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:41.011635+0000","last_peered":"2024-09-16T10:27:41.011635+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:41.011635+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38879232,"num_objects":9492,"num_object_clones":0,"num_object_copies":18984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7504,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5286},{"shards":"0,2","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11220","reported_seq":11434,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.004696+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:41.004696+0000","last_peered":"2024-09-16T10:27:41.004696+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:41.004696+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39731200,"num_objects":9700,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5317},{"shards":"0,2","objects":3646}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10892","reported_seq":11067,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.064390+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:41.064390+0000","last_peered":"2024-09-16T10:27:41.064390+0000","last_clean":"2024-09-16T10:27:41.064390+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:41.064390+0000","last_undegraded":"2024-09-16T10:27:41.064390+0000","last_fullsized":"2024-09-16T10:27:41.064390+0000","mapping_epoch":33,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38903808,"num_objects":9498,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6695,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10928","reported_seq":19431,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:39.457455+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:39.457455+0000","last_peered":"2024-09-16T10:27:39.457455+0000","last_clean":"2024-09-16T10:27:39.457455+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:39.457455+0000","last_undegraded":"2024-09-16T10:27:39.457455+0000","last_fullsized":"2024-09-16T10:27:39.457455+0000","mapping_epoch":43,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39048134,"num_objects":9535,"num_object_clones":0,"num_object_copies":19070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10926,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11339","reported_seq":11444,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.064543+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:41.064543+0000","last_peered":"2024-09-16T10:27:41.064543+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:41.064543+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39736262,"num_objects":9703,"num_object_clones":0,"num_object_copies":19406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5282},{"shards":"0,2","objects":3619}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10960","reported_seq":11072,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.220807+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:41.220807+0000","last_peered":"2024-09-16T10:27:41.220807+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:41.220807+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745030,"num_objects":9461,"num_object_clones":0,"num_object_copies":18922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10958,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3529},{"shards":"2","objects":5187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11057","reported_seq":11154,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.123894+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:41.123894+0000","last_peered":"2024-09-16T10:27:41.123894+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:41.123894+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38990790,"num_objects":9521,"num_object_clones":0,"num_object_copies":19042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11055,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3426},{"shards":"2","objects":5334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10755","reported_seq":10866,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.031407+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:41.031407+0000","last_peered":"2024-09-16T10:27:41.031407+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:41.031407+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38007267,"num_objects":9280,"num_object_clones":0,"num_object_copies":18560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10754,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5138},{"shards":"0,2","objects":3422}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11244","reported_seq":11397,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:40.958845+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:40.958845+0000","last_peered":"2024-09-16T10:27:40.958845+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:40.958845+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39609769,"num_objects":9673,"num_object_clones":0,"num_object_copies":19346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5346},{"shards":"0,2","objects":3554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10846","reported_seq":10959,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:40.913454+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:40.913454+0000","last_peered":"2024-09-16T10:27:40.913454+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:40.913454+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38069190,"num_objects":9296,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5145},{"shards":"0,2","objects":3393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11244","reported_seq":11371,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:40.987055+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:40.987055+0000","last_peered":"2024-09-16T10:27:40.987055+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:40.987055+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39536524,"num_objects":9656,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5278},{"shards":"0,2","objects":3605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":623993358,"num_objects":152359,"num_object_clones":0,"num_object_copies":304718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42818,"num_objects_misplaced":2419,"num_objects_unfound":0,"num_objects_dirty":152359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153946,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6109,"num_bytes_recovered":24457699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1303597056,"data_stored":1303505813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160881,"ondisk_log_size":160881,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716248,"kb_used_data":441808,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93655592,"statfs":{"total":96636764160,"available":95903326208,"internally_reserved":0,"allocated":452411392,"data_stored":452153686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":281013936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,0,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.221},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.635}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":2.0950000000000002},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":1.4830000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555228,"kb_used_data":307028,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93816612,"statfs":{"total":96636764160,"available":96068210688,"internally_reserved":0,"allocated":314396672,"data_stored":314066706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":254144170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.6869999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.536}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.38300000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.0310000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215170,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859804,"kb_used_data":525524,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93512036,"statfs":{"total":96636764160,"available":95756324864,"internally_reserved":0,"allocated":538136576,"data_stored":537886556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,2,1,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":9.2409999999999997},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.50600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":537739264,"data_stored":537735651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313974784,"data_stored":313894839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":451883008,"data_stored":451875323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:42.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:43.326 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:43.326 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:43.347 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-16T10:27:42.655305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":623993377,"num_objects":152361,"num_object_clones":0,"num_object_copies":304722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42818,"num_objects_misplaced":2419,"num_objects_unfound":0,"num_objects_dirty":152361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153948,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6110,"num_bytes_recovered":24457718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2131280,"kb_used_data":1274360,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":280984240,"statfs":{"total":289910292480,"available":287727861760,"internally_reserved":0,"allocated":1304944640,"data_stored":1304106948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,5,2,1,0,0,0,0,3,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25128960,"num_objects":-6135,"num_object_clones":0,"num_object_copies":-12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-68,"num_objects_misplaced":-338,"num_objects_unfound":0,"num_objects_dirty":-6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6135,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":406,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004404"},"pg_stats":[{"pgid":"2.f","version":"44'10989","reported_seq":14603,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:41.074657+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:41.074657+0000","last_peered":"2024-09-16T10:27:41.074657+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:41.074657+0000","last_undegraded":"2024-09-16T10:27:41.074657+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38825984,"num_objects":9479,"num_object_clones":0,"num_object_copies":18958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2419,"num_objects_unfound":0,"num_objects_dirty":9479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10989,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":7262208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7060},{"shards":"0,2","objects":2419}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10960","reported_seq":11117,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.226808+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:41.226808+0000","last_peered":"2024-09-16T10:27:41.226808+0000","last_clean":"2024-09-16T10:27:41.226808+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:41.226808+0000","last_undegraded":"2024-09-16T10:27:41.226808+0000","last_fullsized":"2024-09-16T10:27:41.226808+0000","mapping_epoch":34,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38920192,"num_objects":9502,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7173,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11110","reported_seq":11309,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.188511+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:41.188511+0000","last_peered":"2024-09-16T10:27:41.188511+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:41.188511+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39129112,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7167,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3559},{"shards":"2","objects":5225}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10983","reported_seq":11040,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.168573+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:41.168573+0000","last_peered":"2024-09-16T10:27:41.168573+0000","last_clean":"2024-09-16T10:27:41.168573+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:41.168573+0000","last_undegraded":"2024-09-16T10:27:41.168573+0000","last_fullsized":"2024-09-16T10:27:41.168573+0000","mapping_epoch":34,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38825984,"num_objects":9479,"num_object_clones":0,"num_object_copies":18958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10983,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10975","reported_seq":11132,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:40.967780+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:40.967780+0000","last_peered":"2024-09-16T10:27:40.967780+0000","last_clean":"2024-09-16T10:27:40.967780+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:40.967780+0000","last_undegraded":"2024-09-16T10:27:40.967780+0000","last_fullsized":"2024-09-16T10:27:40.967780+0000","mapping_epoch":33,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":9530,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7617,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10879","reported_seq":11065,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.011635+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:41.011635+0000","last_peered":"2024-09-16T10:27:41.011635+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:41.011635+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38879232,"num_objects":9492,"num_object_clones":0,"num_object_copies":18984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7504,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5286},{"shards":"0,2","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11220","reported_seq":11434,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.004696+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:41.004696+0000","last_peered":"2024-09-16T10:27:41.004696+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:41.004696+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39731200,"num_objects":9700,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5317},{"shards":"0,2","objects":3646}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10892","reported_seq":11067,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.064390+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:41.064390+0000","last_peered":"2024-09-16T10:27:41.064390+0000","last_clean":"2024-09-16T10:27:41.064390+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:41.064390+0000","last_undegraded":"2024-09-16T10:27:41.064390+0000","last_fullsized":"2024-09-16T10:27:41.064390+0000","mapping_epoch":33,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38903808,"num_objects":9498,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6695,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10928","reported_seq":19431,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:39.457455+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:39.457455+0000","last_peered":"2024-09-16T10:27:39.457455+0000","last_clean":"2024-09-16T10:27:39.457455+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:39.457455+0000","last_undegraded":"2024-09-16T10:27:39.457455+0000","last_fullsized":"2024-09-16T10:27:39.457455+0000","mapping_epoch":43,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39048134,"num_objects":9535,"num_object_clones":0,"num_object_copies":19070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10926,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11339","reported_seq":11444,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.064543+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:41.064543+0000","last_peered":"2024-09-16T10:27:41.064543+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:41.064543+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39736262,"num_objects":9703,"num_object_clones":0,"num_object_copies":19406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5282},{"shards":"0,2","objects":3619}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10960","reported_seq":11072,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.220807+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:41.220807+0000","last_peered":"2024-09-16T10:27:41.220807+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:41.220807+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745030,"num_objects":9461,"num_object_clones":0,"num_object_copies":18922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10958,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3529},{"shards":"2","objects":5187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11057","reported_seq":11154,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.123894+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:41.123894+0000","last_peered":"2024-09-16T10:27:41.123894+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:41.123894+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38990790,"num_objects":9521,"num_object_clones":0,"num_object_copies":19042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11055,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3426},{"shards":"2","objects":5334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10755","reported_seq":10866,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.031407+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:41.031407+0000","last_peered":"2024-09-16T10:27:41.031407+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:41.031407+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38007267,"num_objects":9280,"num_object_clones":0,"num_object_copies":18560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10754,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5138},{"shards":"0,2","objects":3422}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11244","reported_seq":11397,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:40.958845+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:40.958845+0000","last_peered":"2024-09-16T10:27:40.958845+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:40.958845+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39609769,"num_objects":9673,"num_object_clones":0,"num_object_copies":19346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5346},{"shards":"0,2","objects":3554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10846","reported_seq":10959,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:40.913454+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:40.913454+0000","last_peered":"2024-09-16T10:27:40.913454+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:40.913454+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38069190,"num_objects":9296,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5145},{"shards":"0,2","objects":3393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11244","reported_seq":11371,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:40.987055+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:40.987055+0000","last_peered":"2024-09-16T10:27:40.987055+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:40.987055+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39536524,"num_objects":9656,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5278},{"shards":"0,2","objects":3605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":623993358,"num_objects":152359,"num_object_clones":0,"num_object_copies":304718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42818,"num_objects_misplaced":2419,"num_objects_unfound":0,"num_objects_dirty":152359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153946,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6109,"num_bytes_recovered":24457699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1303597056,"data_stored":1303505813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160881,"ondisk_log_size":160881,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716248,"kb_used_data":441808,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93655592,"statfs":{"total":96636764160,"available":95903326208,"internally_reserved":0,"allocated":452411392,"data_stored":452153686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":281013936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,0,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.221},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.635}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":2.0950000000000002},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":1.4830000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555228,"kb_used_data":307028,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93816612,"statfs":{"total":96636764160,"available":96068210688,"internally_reserved":0,"allocated":314396672,"data_stored":314066706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":254144170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.6869999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.536}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.38300000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.0310000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215170,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859804,"kb_used_data":525524,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93512036,"statfs":{"total":96636764160,"available":95756324864,"internally_reserved":0,"allocated":538136576,"data_stored":537886556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,2,1,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":9.2409999999999997},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.50600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":537739264,"data_stored":537735651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313974784,"data_stored":313894839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":451883008,"data_stored":451875323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:43.348 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:43.348 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'10989", 'reported_seq': 14603, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:41.074657+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:41.074657+0000', 'last_peered': '2024-09-16T10:27:41.074657+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:41.074657+0000', 'last_undegraded': '2024-09-16T10:27:41.074657+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'900", 'ondisk_log_start': "35'900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38825984, 'num_objects': 9479, 'num_object_clones': 0, 'num_object_copies': 18958, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2419, 'num_objects_unfound': 0, 'num_objects_dirty': 9479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10989, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1773, 'num_bytes_recovered': 7262208, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7060}, {'shards': '0,2', 'objects': 2419}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:43.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:43.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'11110", 'reported_seq': 11309, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:41.188511+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:41.188511+0000', 'last_peered': '2024-09-16T10:27:41.188511+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:41.188511+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'1100", 'ondisk_log_start': "35'1100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39129112, 'num_objects': 9554, 'num_object_clones': 0, 'num_object_copies': 19108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4330, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9554, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7167, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3559}, {'shards': '2', 'objects': 5225}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:43.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:43.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'10879", 'reported_seq': 11065, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:41.011635+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:41.011635+0000', 'last_peered': '2024-09-16T10:27:41.011635+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:41.011635+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38879232, 'num_objects': 9492, 'num_object_clones': 0, 'num_object_copies': 18984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4208, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7504, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5286}, {'shards': '0,2', 'objects': 3526}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:43.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:43.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'11220", 'reported_seq': 11434, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:41.004696+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:41.004696+0000', 'last_peered': '2024-09-16T10:27:41.004696+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:41.004696+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'1200", 'ondisk_log_start': "35'1200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39731200, 'num_objects': 9700, 'num_object_clones': 0, 'num_object_copies': 19400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4385, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9700, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7462, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5317}, {'shards': '0,2', 'objects': 3646}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:43.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:43.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'11339", 'reported_seq': 11444, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:41.064543+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:41.064543+0000', 'last_peered': '2024-09-16T10:27:41.064543+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:41.064543+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'1300", 'ondisk_log_start': "35'1300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39736262, 'num_objects': 9703, 'num_object_clones': 0, 'num_object_copies': 19406, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4423, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9703, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11337, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5282}, {'shards': '0,2', 'objects': 3619}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:43.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:43.349 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'10960", 'reported_seq': 11072, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:41.220807+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:41.220807+0000', 'last_peered': '2024-09-16T10:27:41.220807+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:41.220807+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'900", 'ondisk_log_start': "35'900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38745030, 'num_objects': 9461, 'num_object_clones': 0, 'num_object_copies': 18922, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10958, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 5, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3529}, {'shards': '2', 'objects': 5187}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:43.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:43.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'11057", 'reported_seq': 11154, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:41.123894+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:41.123894+0000', 'last_peered': '2024-09-16T10:27:41.123894+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:41.123894+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'1000", 'ondisk_log_start': "35'1000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38990790, 'num_objects': 9521, 'num_object_clones': 0, 'num_object_copies': 19042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4189, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11055, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3426}, {'shards': '2', 'objects': 5334}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:43.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:43.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'10755", 'reported_seq': 10866, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:41.031407+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:41.031407+0000', 'last_peered': '2024-09-16T10:27:41.031407+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:41.031407+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'700", 'ondisk_log_start': "35'700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38007267, 'num_objects': 9280, 'num_object_clones': 0, 'num_object_copies': 18560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10754, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5138}, {'shards': '0,2', 'objects': 3422}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:43.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:43.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'11244", 'reported_seq': 11397, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:40.958845+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:40.958845+0000', 'last_peered': '2024-09-16T10:27:40.958845+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:40.958845+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'1200", 'ondisk_log_start': "35'1200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39609769, 'num_objects': 9673, 'num_object_clones': 0, 'num_object_copies': 19346, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4330, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11241, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5346}, {'shards': '0,2', 'objects': 3554}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:43.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:43.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'10846", 'reported_seq': 10959, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:40.913454+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:40.913454+0000', 'last_peered': '2024-09-16T10:27:40.913454+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:40.913454+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38069190, 'num_objects': 9296, 'num_object_clones': 0, 'num_object_copies': 18592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4153, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9296, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10844, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5145}, {'shards': '0,2', 'objects': 3393}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:43.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:43.350 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'11244", 'reported_seq': 11371, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:40.987055+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:40.987055+0000', 'last_peered': '2024-09-16T10:27:40.987055+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:40.987055+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'1200", 'ondisk_log_start': "35'1200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39536524, 'num_objects': 9656, 'num_object_clones': 0, 'num_object_copies': 19312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4381, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11241, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5278}, {'shards': '0,2', 'objects': 3605}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:43.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:44.008 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:44.008 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42818/304722 objects degraded (14.051%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":352,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":152361,"data_bytes":623993377,"bytes_used":2182430720,"bytes_avail":287727861760,"bytes_total":289910292480,"degraded_objects":42818,"degraded_total":304722,"degraded_ratio":0.14051496117772921,"misplaced_objects":2419,"misplaced_total":304722,"misplaced_ratio":0.0079383831820478996,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":115328,"recovering_keys_per_sec":0,"num_objects_recovered":406,"num_bytes_recovered":1384448,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":511},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:44.034 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:44.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:44.585 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:44.585 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:44.610 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-09-16T10:27:42.655305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":623993377,"num_objects":152361,"num_object_clones":0,"num_object_copies":304722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42818,"num_objects_misplaced":2419,"num_objects_unfound":0,"num_objects_dirty":152361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153948,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6110,"num_bytes_recovered":24457718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2131280,"kb_used_data":1274360,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":280984240,"statfs":{"total":289910292480,"available":287727861760,"internally_reserved":0,"allocated":1304944640,"data_stored":1304106948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,4,5,2,1,0,0,0,0,3,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25128960,"num_objects":-6135,"num_object_clones":0,"num_object_copies":-12270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-68,"num_objects_misplaced":-338,"num_objects_unfound":0,"num_objects_dirty":-6135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6135,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":406,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004404"},"pg_stats":[{"pgid":"2.f","version":"44'10989","reported_seq":14603,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:41.074657+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:41.074657+0000","last_peered":"2024-09-16T10:27:41.074657+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:41.074657+0000","last_undegraded":"2024-09-16T10:27:41.074657+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38825984,"num_objects":9479,"num_object_clones":0,"num_object_copies":18958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2419,"num_objects_unfound":0,"num_objects_dirty":9479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10989,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":7262208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7060},{"shards":"0,2","objects":2419}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'10960","reported_seq":11117,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.226808+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:41.226808+0000","last_peered":"2024-09-16T10:27:41.226808+0000","last_clean":"2024-09-16T10:27:41.226808+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:41.226808+0000","last_undegraded":"2024-09-16T10:27:41.226808+0000","last_fullsized":"2024-09-16T10:27:41.226808+0000","mapping_epoch":34,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38920192,"num_objects":9502,"num_object_clones":0,"num_object_copies":19004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7173,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11110","reported_seq":11309,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.188511+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:41.188511+0000","last_peered":"2024-09-16T10:27:41.188511+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:41.188511+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39129112,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7167,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3559},{"shards":"2","objects":5225}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'10983","reported_seq":11040,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.168573+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:41.168573+0000","last_peered":"2024-09-16T10:27:41.168573+0000","last_clean":"2024-09-16T10:27:41.168573+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:41.168573+0000","last_undegraded":"2024-09-16T10:27:41.168573+0000","last_fullsized":"2024-09-16T10:27:41.168573+0000","mapping_epoch":34,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38825984,"num_objects":9479,"num_object_clones":0,"num_object_copies":18958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10983,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'10975","reported_seq":11132,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:40.967780+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:40.967780+0000","last_peered":"2024-09-16T10:27:40.967780+0000","last_clean":"2024-09-16T10:27:40.967780+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:40.967780+0000","last_undegraded":"2024-09-16T10:27:40.967780+0000","last_fullsized":"2024-09-16T10:27:40.967780+0000","mapping_epoch":33,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39034880,"num_objects":9530,"num_object_clones":0,"num_object_copies":19060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7617,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'10879","reported_seq":11065,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.011635+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:41.011635+0000","last_peered":"2024-09-16T10:27:41.011635+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:41.011635+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38879232,"num_objects":9492,"num_object_clones":0,"num_object_copies":18984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7504,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5286},{"shards":"0,2","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11220","reported_seq":11434,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.004696+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:41.004696+0000","last_peered":"2024-09-16T10:27:41.004696+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:41.004696+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39731200,"num_objects":9700,"num_object_clones":0,"num_object_copies":19400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7462,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5317},{"shards":"0,2","objects":3646}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'10892","reported_seq":11067,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:41.064390+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:41.064390+0000","last_peered":"2024-09-16T10:27:41.064390+0000","last_clean":"2024-09-16T10:27:41.064390+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:41.064390+0000","last_undegraded":"2024-09-16T10:27:41.064390+0000","last_fullsized":"2024-09-16T10:27:41.064390+0000","mapping_epoch":33,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38903808,"num_objects":9498,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6695,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'10928","reported_seq":19431,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:39.457455+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:39.457455+0000","last_peered":"2024-09-16T10:27:39.457455+0000","last_clean":"2024-09-16T10:27:39.457455+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:39.457455+0000","last_undegraded":"2024-09-16T10:27:39.457455+0000","last_fullsized":"2024-09-16T10:27:39.457455+0000","mapping_epoch":43,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39048134,"num_objects":9535,"num_object_clones":0,"num_object_copies":19070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10926,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11339","reported_seq":11444,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.064543+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:41.064543+0000","last_peered":"2024-09-16T10:27:41.064543+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:41.064543+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39736262,"num_objects":9703,"num_object_clones":0,"num_object_copies":19406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5282},{"shards":"0,2","objects":3619}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'10960","reported_seq":11072,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.220807+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:41.220807+0000","last_peered":"2024-09-16T10:27:41.220807+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:41.220807+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745030,"num_objects":9461,"num_object_clones":0,"num_object_copies":18922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10958,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3529},{"shards":"2","objects":5187}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11057","reported_seq":11154,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.123894+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:41.123894+0000","last_peered":"2024-09-16T10:27:41.123894+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:41.123894+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38990790,"num_objects":9521,"num_object_clones":0,"num_object_copies":19042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11055,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3426},{"shards":"2","objects":5334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10755","reported_seq":10866,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:41.031407+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:41.031407+0000","last_peered":"2024-09-16T10:27:41.031407+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:41.031407+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'700","ondisk_log_start":"35'700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38007267,"num_objects":9280,"num_object_clones":0,"num_object_copies":18560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10754,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5138},{"shards":"0,2","objects":3422}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11244","reported_seq":11397,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:40.958845+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:40.958845+0000","last_peered":"2024-09-16T10:27:40.958845+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:40.958845+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39609769,"num_objects":9673,"num_object_clones":0,"num_object_copies":19346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5346},{"shards":"0,2","objects":3554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10846","reported_seq":10959,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:40.913454+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:40.913454+0000","last_peered":"2024-09-16T10:27:40.913454+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:40.913454+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38069190,"num_objects":9296,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5145},{"shards":"0,2","objects":3393}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11244","reported_seq":11371,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:40.987055+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:40.987055+0000","last_peered":"2024-09-16T10:27:40.987055+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:40.987055+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39536524,"num_objects":9656,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11241,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5278},{"shards":"0,2","objects":3605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":623993358,"num_objects":152359,"num_object_clones":0,"num_object_copies":304718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42818,"num_objects_misplaced":2419,"num_objects_unfound":0,"num_objects_dirty":152359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153946,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6109,"num_bytes_recovered":24457699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1303597056,"data_stored":1303505813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160881,"ondisk_log_size":160881,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018942,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716248,"kb_used_data":441808,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93655592,"statfs":{"total":96636764160,"available":95903326208,"internally_reserved":0,"allocated":452411392,"data_stored":452153686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":281013936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,0,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.221},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.635}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":2.0950000000000002},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":1.4830000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":555228,"kb_used_data":307028,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93816612,"statfs":{"total":96636764160,"available":96068210688,"internally_reserved":0,"allocated":314396672,"data_stored":314066706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":254144170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.6869999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.536}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.38300000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.0310000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215170,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859804,"kb_used_data":525524,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93512036,"statfs":{"total":96636764160,"available":95756324864,"internally_reserved":0,"allocated":538136576,"data_stored":537886556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,2,1,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":9.2409999999999997},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.50600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":537739264,"data_stored":537735651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313974784,"data_stored":313894839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":451883008,"data_stored":451875323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:47.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:48.083 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:48.095 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-16T10:27:46.656513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615625249,"num_objects":150318,"num_object_clones":0,"num_object_copies":300636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42795,"num_objects_misplaced":2279,"num_objects_unfound":0,"num_objects_dirty":150318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155991,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6273,"num_bytes_recovered":25031158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2120524,"kb_used_data":1263596,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":280994996,"statfs":{"total":289910292480,"available":287738875904,"internally_reserved":0,"allocated":1293922304,"data_stored":1293084612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,6,6,0,0,0,4,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25223168,"num_objects":-6158,"num_object_clones":0,"num_object_copies":-12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-50,"num_objects_misplaced":-350,"num_objects_unfound":0,"num_objects_dirty":-6158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6158,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004200"},"pg_stats":[{"pgid":"2.f","version":"44'11126","reported_seq":15022,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:46.101105+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:46.101105+0000","last_peered":"2024-09-16T10:27:46.101105+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:46.101105+0000","last_undegraded":"2024-09-16T10:27:46.101105+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38264832,"num_objects":9342,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2279,"num_objects_unfound":0,"num_objects_dirty":9342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11126,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1913,"num_bytes_recovered":7835648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7063},{"shards":"0,2","objects":2279}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11094","reported_seq":11251,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:46.182638+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:46.182638+0000","last_peered":"2024-09-16T10:27:46.182638+0000","last_clean":"2024-09-16T10:27:46.182638+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:46.182638+0000","last_undegraded":"2024-09-16T10:27:46.182638+0000","last_fullsized":"2024-09-16T10:27:46.182638+0000","mapping_epoch":34,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38371328,"num_objects":9368,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7307,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11225","reported_seq":11424,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.184602+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:46.184602+0000","last_peered":"2024-09-16T10:27:46.184602+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:46.184602+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38658072,"num_objects":9439,"num_object_clones":0,"num_object_copies":18878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7282,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3444},{"shards":"2","objects":5111}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11124","reported_seq":11181,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:46.188466+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:46.188466+0000","last_peered":"2024-09-16T10:27:46.188466+0000","last_clean":"2024-09-16T10:27:46.188466+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:46.188466+0000","last_undegraded":"2024-09-16T10:27:46.188466+0000","last_fullsized":"2024-09-16T10:27:46.188466+0000","mapping_epoch":34,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38248448,"num_objects":9338,"num_object_clones":0,"num_object_copies":18676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11124,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11124","reported_seq":11281,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:46.049271+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:46.049271+0000","last_peered":"2024-09-16T10:27:46.049271+0000","last_clean":"2024-09-16T10:27:46.049271+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:46.049271+0000","last_undegraded":"2024-09-16T10:27:46.049271+0000","last_fullsized":"2024-09-16T10:27:46.049271+0000","mapping_epoch":33,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38424576,"num_objects":9381,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11008","reported_seq":11198,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.038056+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:46.038056+0000","last_peered":"2024-09-16T10:27:46.038056+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:46.038056+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38350848,"num_objects":9363,"num_object_clones":0,"num_object_copies":18726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7633,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5159},{"shards":"0,2","objects":3397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11366","reported_seq":11588,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.019923+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:46.019923+0000","last_peered":"2024-09-16T10:27:46.019923+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:46.019923+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39133184,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7608,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5175},{"shards":"0,2","objects":3500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11004","reported_seq":11179,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:45.877293+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:45.877293+0000","last_peered":"2024-09-16T10:27:45.877293+0000","last_clean":"2024-09-16T10:27:45.877293+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:45.877293+0000","last_undegraded":"2024-09-16T10:27:45.877293+0000","last_fullsized":"2024-09-16T10:27:45.877293+0000","mapping_epoch":33,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38445056,"num_objects":9386,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6807,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11054","reported_seq":19557,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:44.434075+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:44.434075+0000","last_peered":"2024-09-16T10:27:44.434075+0000","last_clean":"2024-09-16T10:27:44.434075+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:44.434075+0000","last_undegraded":"2024-09-16T10:27:44.434075+0000","last_fullsized":"2024-09-16T10:27:44.434075+0000","mapping_epoch":43,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38532038,"num_objects":9409,"num_object_clones":0,"num_object_copies":18818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11052,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11463","reported_seq":11578,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.068278+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:46.068278+0000","last_peered":"2024-09-16T10:27:46.068278+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:46.068278+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39228358,"num_objects":9579,"num_object_clones":0,"num_object_copies":19158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5164},{"shards":"0,2","objects":3495}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11089","reported_seq":11207,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.214811+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:46.214811+0000","last_peered":"2024-09-16T10:27:46.214811+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:46.214811+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38216646,"num_objects":9332,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11087,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3400},{"shards":"2","objects":5061}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11177","reported_seq":11274,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.238921+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:46.238921+0000","last_peered":"2024-09-16T10:27:46.238921+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:46.238921+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38499270,"num_objects":9401,"num_object_clones":0,"num_object_copies":18802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11175,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3306},{"shards":"2","objects":5214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10859","reported_seq":10975,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.025955+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:46.025955+0000","last_peered":"2024-09-16T10:27:46.025955+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:46.025955+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37581283,"num_objects":9176,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10858,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5035},{"shards":"0,2","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11379","reported_seq":11533,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.049343+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:46.049343+0000","last_peered":"2024-09-16T10:27:46.049343+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:46.049343+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39056809,"num_objects":9538,"num_object_clones":0,"num_object_copies":19076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5212},{"shards":"0,2","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10975","reported_seq":11088,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.029557+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:46.029557+0000","last_peered":"2024-09-16T10:27:46.029557+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:46.029557+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37540806,"num_objects":9167,"num_object_clones":0,"num_object_copies":18334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10973,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5016},{"shards":"0,2","objects":3264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11357","reported_seq":11494,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.078395+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:46.078395+0000","last_peered":"2024-09-16T10:27:46.078395+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:46.078395+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39073676,"num_objects":9543,"num_object_clones":0,"num_object_copies":19086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11354,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5170},{"shards":"0,2","objects":3492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":615625230,"num_objects":150316,"num_object_clones":0,"num_object_copies":300632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42795,"num_objects_misplaced":2279,"num_objects_unfound":0,"num_objects_dirty":150316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155989,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6272,"num_bytes_recovered":25031139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1292537856,"data_stored":1292446613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018943,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699824,"kb_used_data":437672,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93672016,"statfs":{"total":96636764160,"available":95920144384,"internally_reserved":0,"allocated":448176128,"data_stored":447918422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":268431024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.2120000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":4.9640000000000004},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.30299999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691887,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562940,"kb_used_data":306548,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93808900,"statfs":{"total":96636764160,"available":96060313600,"internally_reserved":0,"allocated":313905152,"data_stored":313575186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":262532778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.76400000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.97799999999999998},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.94199999999999995}]}]},{"osd":0,"up_from":24,"seq":103079215171,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":857760,"kb_used_data":519376,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93514080,"statfs":{"total":96636764160,"available":95758417920,"internally_reserved":0,"allocated":531841024,"data_stored":531591004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":346484392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,5,0,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":1.4830000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":2.5150000000000001},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.5309999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":531378176,"data_stored":531374563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313516032,"data_stored":313436087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":447643648,"data_stored":447635963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:48.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:48.604 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:48.606 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:48.631 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-16T10:27:46.656513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615625249,"num_objects":150318,"num_object_clones":0,"num_object_copies":300636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42795,"num_objects_misplaced":2279,"num_objects_unfound":0,"num_objects_dirty":150318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155991,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6273,"num_bytes_recovered":25031158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2120524,"kb_used_data":1263596,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":280994996,"statfs":{"total":289910292480,"available":287738875904,"internally_reserved":0,"allocated":1293922304,"data_stored":1293084612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,6,6,0,0,0,4,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25223168,"num_objects":-6158,"num_object_clones":0,"num_object_copies":-12316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-50,"num_objects_misplaced":-350,"num_objects_unfound":0,"num_objects_dirty":-6158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6158,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004200"},"pg_stats":[{"pgid":"2.f","version":"44'11126","reported_seq":15022,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:46.101105+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:46.101105+0000","last_peered":"2024-09-16T10:27:46.101105+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:46.101105+0000","last_undegraded":"2024-09-16T10:27:46.101105+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38264832,"num_objects":9342,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2279,"num_objects_unfound":0,"num_objects_dirty":9342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11126,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1913,"num_bytes_recovered":7835648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7063},{"shards":"0,2","objects":2279}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11094","reported_seq":11251,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:46.182638+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:46.182638+0000","last_peered":"2024-09-16T10:27:46.182638+0000","last_clean":"2024-09-16T10:27:46.182638+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:46.182638+0000","last_undegraded":"2024-09-16T10:27:46.182638+0000","last_fullsized":"2024-09-16T10:27:46.182638+0000","mapping_epoch":34,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38371328,"num_objects":9368,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7307,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11225","reported_seq":11424,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.184602+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:46.184602+0000","last_peered":"2024-09-16T10:27:46.184602+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:46.184602+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38658072,"num_objects":9439,"num_object_clones":0,"num_object_copies":18878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7282,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3444},{"shards":"2","objects":5111}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11124","reported_seq":11181,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:46.188466+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:46.188466+0000","last_peered":"2024-09-16T10:27:46.188466+0000","last_clean":"2024-09-16T10:27:46.188466+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:46.188466+0000","last_undegraded":"2024-09-16T10:27:46.188466+0000","last_fullsized":"2024-09-16T10:27:46.188466+0000","mapping_epoch":34,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38248448,"num_objects":9338,"num_object_clones":0,"num_object_copies":18676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11124,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11124","reported_seq":11281,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:46.049271+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:46.049271+0000","last_peered":"2024-09-16T10:27:46.049271+0000","last_clean":"2024-09-16T10:27:46.049271+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:46.049271+0000","last_undegraded":"2024-09-16T10:27:46.049271+0000","last_fullsized":"2024-09-16T10:27:46.049271+0000","mapping_epoch":33,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38424576,"num_objects":9381,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11008","reported_seq":11198,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.038056+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:46.038056+0000","last_peered":"2024-09-16T10:27:46.038056+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:46.038056+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38350848,"num_objects":9363,"num_object_clones":0,"num_object_copies":18726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7633,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5159},{"shards":"0,2","objects":3397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11366","reported_seq":11588,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.019923+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:46.019923+0000","last_peered":"2024-09-16T10:27:46.019923+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:46.019923+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39133184,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7608,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5175},{"shards":"0,2","objects":3500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11004","reported_seq":11179,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:45.877293+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:45.877293+0000","last_peered":"2024-09-16T10:27:45.877293+0000","last_clean":"2024-09-16T10:27:45.877293+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:45.877293+0000","last_undegraded":"2024-09-16T10:27:45.877293+0000","last_fullsized":"2024-09-16T10:27:45.877293+0000","mapping_epoch":33,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38445056,"num_objects":9386,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6807,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11054","reported_seq":19557,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:44.434075+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:44.434075+0000","last_peered":"2024-09-16T10:27:44.434075+0000","last_clean":"2024-09-16T10:27:44.434075+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:44.434075+0000","last_undegraded":"2024-09-16T10:27:44.434075+0000","last_fullsized":"2024-09-16T10:27:44.434075+0000","mapping_epoch":43,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38532038,"num_objects":9409,"num_object_clones":0,"num_object_copies":18818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11052,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11463","reported_seq":11578,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.068278+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:46.068278+0000","last_peered":"2024-09-16T10:27:46.068278+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:46.068278+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39228358,"num_objects":9579,"num_object_clones":0,"num_object_copies":19158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5164},{"shards":"0,2","objects":3495}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11089","reported_seq":11207,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.214811+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:46.214811+0000","last_peered":"2024-09-16T10:27:46.214811+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:46.214811+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38216646,"num_objects":9332,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11087,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3400},{"shards":"2","objects":5061}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11177","reported_seq":11274,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.238921+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:46.238921+0000","last_peered":"2024-09-16T10:27:46.238921+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:46.238921+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38499270,"num_objects":9401,"num_object_clones":0,"num_object_copies":18802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11175,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3306},{"shards":"2","objects":5214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10859","reported_seq":10975,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.025955+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:46.025955+0000","last_peered":"2024-09-16T10:27:46.025955+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:46.025955+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37581283,"num_objects":9176,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10858,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5035},{"shards":"0,2","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11379","reported_seq":11533,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.049343+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:46.049343+0000","last_peered":"2024-09-16T10:27:46.049343+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:46.049343+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39056809,"num_objects":9538,"num_object_clones":0,"num_object_copies":19076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5212},{"shards":"0,2","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10975","reported_seq":11088,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.029557+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:46.029557+0000","last_peered":"2024-09-16T10:27:46.029557+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:46.029557+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37540806,"num_objects":9167,"num_object_clones":0,"num_object_copies":18334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10973,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5016},{"shards":"0,2","objects":3264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11357","reported_seq":11494,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.078395+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:46.078395+0000","last_peered":"2024-09-16T10:27:46.078395+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:46.078395+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39073676,"num_objects":9543,"num_object_clones":0,"num_object_copies":19086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11354,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5170},{"shards":"0,2","objects":3492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":615625230,"num_objects":150316,"num_object_clones":0,"num_object_copies":300632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42795,"num_objects_misplaced":2279,"num_objects_unfound":0,"num_objects_dirty":150316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155989,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6272,"num_bytes_recovered":25031139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1292537856,"data_stored":1292446613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018943,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699824,"kb_used_data":437672,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93672016,"statfs":{"total":96636764160,"available":95920144384,"internally_reserved":0,"allocated":448176128,"data_stored":447918422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":268431024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.2120000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":4.9640000000000004},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.30299999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691887,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562940,"kb_used_data":306548,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93808900,"statfs":{"total":96636764160,"available":96060313600,"internally_reserved":0,"allocated":313905152,"data_stored":313575186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":262532778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.76400000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.97799999999999998},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.94199999999999995}]}]},{"osd":0,"up_from":24,"seq":103079215171,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":857760,"kb_used_data":519376,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93514080,"statfs":{"total":96636764160,"available":95758417920,"internally_reserved":0,"allocated":531841024,"data_stored":531591004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":346484392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,5,0,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":1.4830000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":2.5150000000000001},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.5309999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":531378176,"data_stored":531374563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313516032,"data_stored":313436087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":447643648,"data_stored":447635963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:48.633 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:48.633 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'11126", 'reported_seq': 15022, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:46.101105+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:46.101105+0000', 'last_peered': '2024-09-16T10:27:46.101105+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:46.101105+0000', 'last_undegraded': '2024-09-16T10:27:46.101105+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'1100", 'ondisk_log_start': "35'1100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38264832, 'num_objects': 9342, 'num_object_clones': 0, 'num_object_copies': 18684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2279, 'num_objects_unfound': 0, 'num_objects_dirty': 9342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11126, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1913, 'num_bytes_recovered': 7835648, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7063}, {'shards': '0,2', 'objects': 2279}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:48.633 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:48.633 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'11225", 'reported_seq': 11424, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:46.184602+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:46.184602+0000', 'last_peered': '2024-09-16T10:27:46.184602+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:46.184602+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'1200", 'ondisk_log_start': "35'1200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38658072, 'num_objects': 9439, 'num_object_clones': 0, 'num_object_copies': 18878, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4329, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9439, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7282, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3444}, {'shards': '2', 'objects': 5111}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:48.633 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:48.633 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'11008", 'reported_seq': 11198, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:46.038056+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:46.038056+0000', 'last_peered': '2024-09-16T10:27:46.038056+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:46.038056+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'1000", 'ondisk_log_start': "35'1000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38350848, 'num_objects': 9363, 'num_object_clones': 0, 'num_object_copies': 18726, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4206, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7633, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5159}, {'shards': '0,2', 'objects': 3397}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:48.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:48.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'11366", 'reported_seq': 11588, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:46.019923+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:46.019923+0000', 'last_peered': '2024-09-16T10:27:46.019923+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:46.019923+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'1300", 'ondisk_log_start': "35'1300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39133184, 'num_objects': 9554, 'num_object_clones': 0, 'num_object_copies': 19108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4381, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9554, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7608, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5175}, {'shards': '0,2', 'objects': 3500}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:48.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:48.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'11463", 'reported_seq': 11578, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:46.068278+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:46.068278+0000', 'last_peered': '2024-09-16T10:27:46.068278+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:46.068278+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'1400", 'ondisk_log_start': "35'1400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39228358, 'num_objects': 9579, 'num_object_clones': 0, 'num_object_copies': 19158, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4417, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9579, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11461, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5164}, {'shards': '0,2', 'objects': 3495}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:48.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:48.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'11089", 'reported_seq': 11207, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:46.214811+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:46.214811+0000', 'last_peered': '2024-09-16T10:27:46.214811+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:46.214811+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'1000", 'ondisk_log_start': "35'1000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38216646, 'num_objects': 9332, 'num_object_clones': 0, 'num_object_copies': 18664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4273, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11087, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3400}, {'shards': '2', 'objects': 5061}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:48.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:48.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'11177", 'reported_seq': 11274, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:46.238921+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:46.238921+0000', 'last_peered': '2024-09-16T10:27:46.238921+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:46.238921+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'1100", 'ondisk_log_start': "35'1100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38499270, 'num_objects': 9401, 'num_object_clones': 0, 'num_object_copies': 18802, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4189, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9401, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11175, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3306}, {'shards': '2', 'objects': 5214}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:48.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:48.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'10859", 'reported_seq': 10975, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:46.025955+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:46.025955+0000', 'last_peered': '2024-09-16T10:27:46.025955+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:46.025955+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'800", 'ondisk_log_start': "35'800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37581283, 'num_objects': 9176, 'num_object_clones': 0, 'num_object_copies': 18352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4142, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9176, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10858, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5035}, {'shards': '0,2', 'objects': 3318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:48.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:48.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'11379", 'reported_seq': 11533, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:46.049343+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:46.049343+0000', 'last_peered': '2024-09-16T10:27:46.049343+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:46.049343+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'1300", 'ondisk_log_start': "35'1300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39056809, 'num_objects': 9538, 'num_object_clones': 0, 'num_object_copies': 19076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4329, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11376, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5212}, {'shards': '0,2', 'objects': 3419}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:48.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:48.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'10975", 'reported_seq': 11088, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:46.029557+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:46.029557+0000', 'last_peered': '2024-09-16T10:27:46.029557+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:46.029557+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'900", 'ondisk_log_start': "35'900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37540806, 'num_objects': 9167, 'num_object_clones': 0, 'num_object_copies': 18334, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4153, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9167, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10973, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5016}, {'shards': '0,2', 'objects': 3264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:48.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:48.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'11357", 'reported_seq': 11494, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:46.078395+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:46.078395+0000', 'last_peered': '2024-09-16T10:27:46.078395+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:46.078395+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'1300", 'ondisk_log_start': "35'1300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39073676, 'num_objects': 9543, 'num_object_clones': 0, 'num_object_copies': 19086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4376, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11354, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5170}, {'shards': '0,2', 'objects': 3492}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:48.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:49.309 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:49.309 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42795/300636 objects degraded (14.235%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":358,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":150318,"data_bytes":615625249,"bytes_used":2171416576,"bytes_avail":287738875904,"bytes_total":289910292480,"degraded_objects":42795,"degraded_total":300636,"degraded_ratio":0.14234822176984793,"misplaced_objects":2279,"misplaced_total":300636,"misplaced_ratio":0.0075805958035631126,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":86327,"recovering_keys_per_sec":0,"num_objects_recovered":281,"num_bytes_recovered":1036288,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":347},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:49.334 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:49.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:49.898 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:49.898 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:49.919 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-16T10:27:48.657022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615625249,"num_objects":150318,"num_object_clones":0,"num_object_copies":300636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42795,"num_objects_misplaced":2279,"num_objects_unfound":0,"num_objects_dirty":150318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155991,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6273,"num_bytes_recovered":25031158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2120524,"kb_used_data":1263596,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":280994996,"statfs":{"total":289910292480,"available":287738875904,"internally_reserved":0,"allocated":1293922304,"data_stored":1293084612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,3,6,6,0,0,0,4,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17080320,"num_objects":-4170,"num_object_clones":0,"num_object_copies":-8340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-28,"num_objects_misplaced":-253,"num_objects_unfound":0,"num_objects_dirty":-4170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4170,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":1036288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004087"},"pg_stats":[{"pgid":"2.f","version":"44'11126","reported_seq":15022,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:46.101105+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:46.101105+0000","last_peered":"2024-09-16T10:27:46.101105+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:46.101105+0000","last_undegraded":"2024-09-16T10:27:46.101105+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38264832,"num_objects":9342,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2279,"num_objects_unfound":0,"num_objects_dirty":9342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11126,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1913,"num_bytes_recovered":7835648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7063},{"shards":"0,2","objects":2279}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11094","reported_seq":11251,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:46.182638+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:46.182638+0000","last_peered":"2024-09-16T10:27:46.182638+0000","last_clean":"2024-09-16T10:27:46.182638+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:46.182638+0000","last_undegraded":"2024-09-16T10:27:46.182638+0000","last_fullsized":"2024-09-16T10:27:46.182638+0000","mapping_epoch":34,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38371328,"num_objects":9368,"num_object_clones":0,"num_object_copies":18736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7307,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11225","reported_seq":11424,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.184602+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:46.184602+0000","last_peered":"2024-09-16T10:27:46.184602+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:46.184602+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38658072,"num_objects":9439,"num_object_clones":0,"num_object_copies":18878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9439,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7282,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3444},{"shards":"2","objects":5111}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11124","reported_seq":11181,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:46.188466+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:46.188466+0000","last_peered":"2024-09-16T10:27:46.188466+0000","last_clean":"2024-09-16T10:27:46.188466+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:46.188466+0000","last_undegraded":"2024-09-16T10:27:46.188466+0000","last_fullsized":"2024-09-16T10:27:46.188466+0000","mapping_epoch":34,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38248448,"num_objects":9338,"num_object_clones":0,"num_object_copies":18676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11124,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11124","reported_seq":11281,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:46.049271+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:46.049271+0000","last_peered":"2024-09-16T10:27:46.049271+0000","last_clean":"2024-09-16T10:27:46.049271+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:46.049271+0000","last_undegraded":"2024-09-16T10:27:46.049271+0000","last_fullsized":"2024-09-16T10:27:46.049271+0000","mapping_epoch":33,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38424576,"num_objects":9381,"num_object_clones":0,"num_object_copies":18762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11008","reported_seq":11198,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.038056+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:46.038056+0000","last_peered":"2024-09-16T10:27:46.038056+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:46.038056+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38350848,"num_objects":9363,"num_object_clones":0,"num_object_copies":18726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7633,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5159},{"shards":"0,2","objects":3397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11366","reported_seq":11588,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.019923+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:46.019923+0000","last_peered":"2024-09-16T10:27:46.019923+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:46.019923+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39133184,"num_objects":9554,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7608,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5175},{"shards":"0,2","objects":3500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11004","reported_seq":11179,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:45.877293+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:45.877293+0000","last_peered":"2024-09-16T10:27:45.877293+0000","last_clean":"2024-09-16T10:27:45.877293+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:45.877293+0000","last_undegraded":"2024-09-16T10:27:45.877293+0000","last_fullsized":"2024-09-16T10:27:45.877293+0000","mapping_epoch":33,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38445056,"num_objects":9386,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6807,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11054","reported_seq":19557,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:44.434075+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:44.434075+0000","last_peered":"2024-09-16T10:27:44.434075+0000","last_clean":"2024-09-16T10:27:44.434075+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:44.434075+0000","last_undegraded":"2024-09-16T10:27:44.434075+0000","last_fullsized":"2024-09-16T10:27:44.434075+0000","mapping_epoch":43,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38532038,"num_objects":9409,"num_object_clones":0,"num_object_copies":18818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11052,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11463","reported_seq":11578,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.068278+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:46.068278+0000","last_peered":"2024-09-16T10:27:46.068278+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:46.068278+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39228358,"num_objects":9579,"num_object_clones":0,"num_object_copies":19158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4417,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5164},{"shards":"0,2","objects":3495}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11089","reported_seq":11207,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.214811+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:46.214811+0000","last_peered":"2024-09-16T10:27:46.214811+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:46.214811+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38216646,"num_objects":9332,"num_object_clones":0,"num_object_copies":18664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4273,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11087,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3400},{"shards":"2","objects":5061}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11177","reported_seq":11274,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.238921+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:46.238921+0000","last_peered":"2024-09-16T10:27:46.238921+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:46.238921+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38499270,"num_objects":9401,"num_object_clones":0,"num_object_copies":18802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11175,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3306},{"shards":"2","objects":5214}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10859","reported_seq":10975,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.025955+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:46.025955+0000","last_peered":"2024-09-16T10:27:46.025955+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:46.025955+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'800","ondisk_log_start":"35'800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37581283,"num_objects":9176,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10858,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5035},{"shards":"0,2","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11379","reported_seq":11533,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.049343+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:46.049343+0000","last_peered":"2024-09-16T10:27:46.049343+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:46.049343+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39056809,"num_objects":9538,"num_object_clones":0,"num_object_copies":19076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5212},{"shards":"0,2","objects":3419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'10975","reported_seq":11088,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.029557+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:46.029557+0000","last_peered":"2024-09-16T10:27:46.029557+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:46.029557+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37540806,"num_objects":9167,"num_object_clones":0,"num_object_copies":18334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10973,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5016},{"shards":"0,2","objects":3264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11357","reported_seq":11494,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:46.078395+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:46.078395+0000","last_peered":"2024-09-16T10:27:46.078395+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:46.078395+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39073676,"num_objects":9543,"num_object_clones":0,"num_object_copies":19086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11354,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5170},{"shards":"0,2","objects":3492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":615625230,"num_objects":150316,"num_object_clones":0,"num_object_copies":300632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42795,"num_objects_misplaced":2279,"num_objects_unfound":0,"num_objects_dirty":150316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155989,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6272,"num_bytes_recovered":25031139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1292537856,"data_stored":1292446613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018943,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699824,"kb_used_data":437672,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93672016,"statfs":{"total":96636764160,"available":95920144384,"internally_reserved":0,"allocated":448176128,"data_stored":447918422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":268431024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,0,0,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.2120000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":4.9640000000000004},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":0.30299999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691887,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562940,"kb_used_data":306548,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93808900,"statfs":{"total":96636764160,"available":96060313600,"internally_reserved":0,"allocated":313905152,"data_stored":313575186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":262532778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.76400000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.97799999999999998},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.94199999999999995}]}]},{"osd":0,"up_from":24,"seq":103079215171,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":857760,"kb_used_data":519376,"kb_used_omap":4,"kb_used_meta":338363,"kb_avail":93514080,"statfs":{"total":96636764160,"available":95758417920,"internally_reserved":0,"allocated":531841024,"data_stored":531591004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":346484392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,5,0,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":1.4830000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":2.5150000000000001},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.5309999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":531378176,"data_stored":531374563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":313516032,"data_stored":313436087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":447643648,"data_stored":447635963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:52.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:53.447 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:53.447 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:53.479 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-16T10:27:52.657967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":607232545,"num_objects":148269,"num_object_clones":0,"num_object_copies":296538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42767,"num_objects_misplaced":2136,"num_objects_unfound":0,"num_objects_dirty":148269,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158040,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6444,"num_bytes_recovered":25616886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160876,"ondisk_log_size":160876,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2109816,"kb_used_data":1252888,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":281005704,"statfs":{"total":289910292480,"available":287749840896,"internally_reserved":0,"allocated":1282957312,"data_stored":1282119620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,6,6,3,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24903680,"num_objects":-6080,"num_object_clones":0,"num_object_copies":-12160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-56,"num_objects_misplaced":-396,"num_objects_unfound":0,"num_objects_dirty":-6080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6080,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":452,"num_bytes_recovered":1622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003399"},"pg_stats":[{"pgid":"2.f","version":"44'11250","reported_seq":15432,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:51.124549+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:51.124549+0000","last_peered":"2024-09-16T10:27:51.124549+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:51.124549+0000","last_undegraded":"2024-09-16T10:27:51.124549+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37756928,"num_objects":9218,"num_object_clones":0,"num_object_copies":18436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2136,"num_objects_unfound":0,"num_objects_dirty":9218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11250,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":8421376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7082},{"shards":"0,2","objects":2136}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11223","reported_seq":11380,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.191271+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:51.191271+0000","last_peered":"2024-09-16T10:27:51.191271+0000","last_clean":"2024-09-16T10:27:51.191271+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:51.191271+0000","last_undegraded":"2024-09-16T10:27:51.191271+0000","last_fullsized":"2024-09-16T10:27:51.191271+0000","mapping_epoch":34,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37842944,"num_objects":9239,"num_object_clones":0,"num_object_copies":18478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7436,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11350","reported_seq":11551,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.257685+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:51.257685+0000","last_peered":"2024-09-16T10:27:51.257685+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:51.257685+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38146072,"num_objects":9314,"num_object_clones":0,"num_object_copies":18628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9314,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7407,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3319},{"shards":"2","objects":4986}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11257","reported_seq":11314,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.172858+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:51.172858+0000","last_peered":"2024-09-16T10:27:51.172858+0000","last_clean":"2024-09-16T10:27:51.172858+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:51.172858+0000","last_undegraded":"2024-09-16T10:27:51.172858+0000","last_fullsized":"2024-09-16T10:27:51.172858+0000","mapping_epoch":34,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37703680,"num_objects":9205,"num_object_clones":0,"num_object_copies":18410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11257,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11252","reported_seq":11409,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.065032+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:51.065032+0000","last_peered":"2024-09-16T10:27:51.065032+0000","last_clean":"2024-09-16T10:27:51.065032+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:51.065032+0000","last_undegraded":"2024-09-16T10:27:51.065032+0000","last_fullsized":"2024-09-16T10:27:51.065032+0000","mapping_epoch":33,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900288,"num_objects":9253,"num_object_clones":0,"num_object_copies":18506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7894,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11138","reported_seq":11330,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.084030+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:51.084030+0000","last_peered":"2024-09-16T10:27:51.084030+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:51.084030+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37818368,"num_objects":9233,"num_object_clones":0,"num_object_copies":18466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7763,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5030},{"shards":"0,2","objects":3267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11524","reported_seq":11762,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.131995+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:51.131995+0000","last_peered":"2024-09-16T10:27:51.131995+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:51.131995+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38486016,"num_objects":9396,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5025},{"shards":"0,2","objects":3342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11131","reported_seq":11306,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.136013+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:51.136013+0000","last_peered":"2024-09-16T10:27:51.136013+0000","last_clean":"2024-09-16T10:27:51.136013+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:51.136013+0000","last_undegraded":"2024-09-16T10:27:51.136013+0000","last_fullsized":"2024-09-16T10:27:51.136013+0000","mapping_epoch":33,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37924864,"num_objects":9259,"num_object_clones":0,"num_object_copies":18518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6934,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11191","reported_seq":19694,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:49.409564+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:49.409564+0000","last_peered":"2024-09-16T10:27:49.409564+0000","last_clean":"2024-09-16T10:27:49.409564+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:49.409564+0000","last_undegraded":"2024-09-16T10:27:49.409564+0000","last_fullsized":"2024-09-16T10:27:49.409564+0000","mapping_epoch":43,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37970886,"num_objects":9272,"num_object_clones":0,"num_object_copies":18544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11189,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11587","reported_seq":11708,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.096619+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:51.096619+0000","last_peered":"2024-09-16T10:27:51.096619+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:51.096619+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38720454,"num_objects":9455,"num_object_clones":0,"num_object_copies":18910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11585,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5042},{"shards":"0,2","objects":3371}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11188","reported_seq":11308,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.252679+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:51.252679+0000","last_peered":"2024-09-16T10:27:51.252679+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:51.252679+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37811142,"num_objects":9233,"num_object_clones":0,"num_object_copies":18466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11186,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3301},{"shards":"2","objects":4963}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11297","reported_seq":11398,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.173560+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:51.173560+0000","last_peered":"2024-09-16T10:27:51.173560+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:51.173560+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38007750,"num_objects":9281,"num_object_clones":0,"num_object_copies":18562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3186},{"shards":"2","objects":5096}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10983","reported_seq":11113,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.122519+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:51.122519+0000","last_peered":"2024-09-16T10:27:51.122519+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:51.122519+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37073379,"num_objects":9052,"num_object_clones":0,"num_object_copies":18104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4918},{"shards":"0,2","objects":3194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11508","reported_seq":11667,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.085469+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:51.085469+0000","last_peered":"2024-09-16T10:27:51.085469+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:51.085469+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38528425,"num_objects":9409,"num_object_clones":0,"num_object_copies":18818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11505,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5085},{"shards":"0,2","objects":3290}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11092","reported_seq":11209,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.109897+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:51.109897+0000","last_peered":"2024-09-16T10:27:51.109897+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:51.109897+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37061574,"num_objects":9050,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11090,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4901},{"shards":"0,2","objects":3147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11502","reported_seq":11645,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.099865+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:51.099865+0000","last_peered":"2024-09-16T10:27:51.099865+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:51.099865+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38479756,"num_objects":9398,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11499,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5028},{"shards":"0,2","objects":3347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":607232526,"num_objects":148267,"num_object_clones":0,"num_object_copies":296534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42767,"num_objects_misplaced":2136,"num_objects_unfound":0,"num_objects_dirty":148267,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158038,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6443,"num_bytes_recovered":25616867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1281466368,"data_stored":1281375125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018944,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699944,"kb_used_data":433692,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93671896,"statfs":{"total":96636764160,"available":95920021504,"internally_reserved":0,"allocated":444100608,"data_stored":443842902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":272625328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.0289999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":3.8959999999999999},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":1.9059999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550216,"kb_used_data":306112,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93821624,"statfs":{"total":96636764160,"available":96073342976,"internally_reserved":0,"allocated":313458688,"data_stored":313128722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":249949866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.56899999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":1.2230000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.3370000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215172,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859656,"kb_used_data":513084,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93512184,"statfs":{"total":96636764160,"available":95756476416,"internally_reserved":0,"allocated":525398016,"data_stored":525147996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":354873000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,3,2,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":5.2910000000000004},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":10.477}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.497},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":5.5190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524898304,"data_stored":524894691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312987648,"data_stored":312907703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":443580416,"data_stored":443572731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:53.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:54.024 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:54.027 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:54.064 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-16T10:27:52.657967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":607232545,"num_objects":148269,"num_object_clones":0,"num_object_copies":296538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42767,"num_objects_misplaced":2136,"num_objects_unfound":0,"num_objects_dirty":148269,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158040,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6444,"num_bytes_recovered":25616886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160876,"ondisk_log_size":160876,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2109816,"kb_used_data":1252888,"kb_used_omap":12,"kb_used_meta":856883,"kb_avail":281005704,"statfs":{"total":289910292480,"available":287749840896,"internally_reserved":0,"allocated":1282957312,"data_stored":1282119620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":877448194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,4,6,6,3,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24903680,"num_objects":-6080,"num_object_clones":0,"num_object_copies":-12160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-56,"num_objects_misplaced":-396,"num_objects_unfound":0,"num_objects_dirty":-6080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6080,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":452,"num_bytes_recovered":1622016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003399"},"pg_stats":[{"pgid":"2.f","version":"44'11250","reported_seq":15432,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:51.124549+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:51.124549+0000","last_peered":"2024-09-16T10:27:51.124549+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:51.124549+0000","last_undegraded":"2024-09-16T10:27:51.124549+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37756928,"num_objects":9218,"num_object_clones":0,"num_object_copies":18436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2136,"num_objects_unfound":0,"num_objects_dirty":9218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11250,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":8421376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7082},{"shards":"0,2","objects":2136}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11223","reported_seq":11380,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.191271+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:51.191271+0000","last_peered":"2024-09-16T10:27:51.191271+0000","last_clean":"2024-09-16T10:27:51.191271+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:51.191271+0000","last_undegraded":"2024-09-16T10:27:51.191271+0000","last_fullsized":"2024-09-16T10:27:51.191271+0000","mapping_epoch":34,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37842944,"num_objects":9239,"num_object_clones":0,"num_object_copies":18478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7436,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11350","reported_seq":11551,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.257685+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:51.257685+0000","last_peered":"2024-09-16T10:27:51.257685+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:51.257685+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38146072,"num_objects":9314,"num_object_clones":0,"num_object_copies":18628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9314,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7407,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3319},{"shards":"2","objects":4986}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11257","reported_seq":11314,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.172858+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:51.172858+0000","last_peered":"2024-09-16T10:27:51.172858+0000","last_clean":"2024-09-16T10:27:51.172858+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:51.172858+0000","last_undegraded":"2024-09-16T10:27:51.172858+0000","last_fullsized":"2024-09-16T10:27:51.172858+0000","mapping_epoch":34,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37703680,"num_objects":9205,"num_object_clones":0,"num_object_copies":18410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11257,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11252","reported_seq":11409,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.065032+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:51.065032+0000","last_peered":"2024-09-16T10:27:51.065032+0000","last_clean":"2024-09-16T10:27:51.065032+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:51.065032+0000","last_undegraded":"2024-09-16T10:27:51.065032+0000","last_fullsized":"2024-09-16T10:27:51.065032+0000","mapping_epoch":33,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900288,"num_objects":9253,"num_object_clones":0,"num_object_copies":18506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7894,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11138","reported_seq":11330,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.084030+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:51.084030+0000","last_peered":"2024-09-16T10:27:51.084030+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:51.084030+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37818368,"num_objects":9233,"num_object_clones":0,"num_object_copies":18466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7763,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5030},{"shards":"0,2","objects":3267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11524","reported_seq":11762,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.131995+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:51.131995+0000","last_peered":"2024-09-16T10:27:51.131995+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:51.131995+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38486016,"num_objects":9396,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5025},{"shards":"0,2","objects":3342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11131","reported_seq":11306,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.136013+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:51.136013+0000","last_peered":"2024-09-16T10:27:51.136013+0000","last_clean":"2024-09-16T10:27:51.136013+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:51.136013+0000","last_undegraded":"2024-09-16T10:27:51.136013+0000","last_fullsized":"2024-09-16T10:27:51.136013+0000","mapping_epoch":33,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37924864,"num_objects":9259,"num_object_clones":0,"num_object_copies":18518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6934,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948714+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:00.948714+0000","last_peered":"2024-09-16T10:26:00.948714+0000","last_clean":"2024-09-16T10:26:00.948714+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:00.948714+0000","last_undegraded":"2024-09-16T10:26:00.948714+0000","last_fullsized":"2024-09-16T10:26:00.948714+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11191","reported_seq":19694,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:49.409564+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:49.409564+0000","last_peered":"2024-09-16T10:27:49.409564+0000","last_clean":"2024-09-16T10:27:49.409564+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:49.409564+0000","last_undegraded":"2024-09-16T10:27:49.409564+0000","last_fullsized":"2024-09-16T10:27:49.409564+0000","mapping_epoch":43,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37970886,"num_objects":9272,"num_object_clones":0,"num_object_copies":18544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11189,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11587","reported_seq":11708,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.096619+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:51.096619+0000","last_peered":"2024-09-16T10:27:51.096619+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:51.096619+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38720454,"num_objects":9455,"num_object_clones":0,"num_object_copies":18910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11585,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5042},{"shards":"0,2","objects":3371}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11188","reported_seq":11308,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.252679+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:51.252679+0000","last_peered":"2024-09-16T10:27:51.252679+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:51.252679+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37811142,"num_objects":9233,"num_object_clones":0,"num_object_copies":18466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11186,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3301},{"shards":"2","objects":4963}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11297","reported_seq":11398,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.173560+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:51.173560+0000","last_peered":"2024-09-16T10:27:51.173560+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:51.173560+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38007750,"num_objects":9281,"num_object_clones":0,"num_object_copies":18562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3186},{"shards":"2","objects":5096}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10983","reported_seq":11113,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.122519+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:51.122519+0000","last_peered":"2024-09-16T10:27:51.122519+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:51.122519+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37073379,"num_objects":9052,"num_object_clones":0,"num_object_copies":18104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4918},{"shards":"0,2","objects":3194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11508","reported_seq":11667,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.085469+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:51.085469+0000","last_peered":"2024-09-16T10:27:51.085469+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:51.085469+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38528425,"num_objects":9409,"num_object_clones":0,"num_object_copies":18818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11505,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5085},{"shards":"0,2","objects":3290}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11092","reported_seq":11209,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.109897+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:51.109897+0000","last_peered":"2024-09-16T10:27:51.109897+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:51.109897+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37061574,"num_objects":9050,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11090,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4901},{"shards":"0,2","objects":3147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11502","reported_seq":11645,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.099865+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:51.099865+0000","last_peered":"2024-09-16T10:27:51.099865+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:51.099865+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38479756,"num_objects":9398,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11499,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5028},{"shards":"0,2","objects":3347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":607232526,"num_objects":148267,"num_object_clones":0,"num_object_copies":296534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42767,"num_objects_misplaced":2136,"num_objects_unfound":0,"num_objects_dirty":148267,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158038,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6443,"num_bytes_recovered":25616867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1281466368,"data_stored":1281375125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018944,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699944,"kb_used_data":433692,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93671896,"statfs":{"total":96636764160,"available":95920021504,"internally_reserved":0,"allocated":444100608,"data_stored":443842902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":272625328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.0289999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":3.8959999999999999},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":1.9059999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691888,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550216,"kb_used_data":306112,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93821624,"statfs":{"total":96636764160,"available":96073342976,"internally_reserved":0,"allocated":313458688,"data_stored":313128722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":249949866},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.56899999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":1.2230000000000001},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.3370000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215172,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859656,"kb_used_data":513084,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93512184,"statfs":{"total":96636764160,"available":95756476416,"internally_reserved":0,"allocated":525398016,"data_stored":525147996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":354873000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,3,2,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":5.2910000000000004},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":10.477}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.497},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":5.5190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524898304,"data_stored":524894691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312987648,"data_stored":312907703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":443580416,"data_stored":443572731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:54.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:54.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'11250", 'reported_seq': 15432, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:51.124549+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:51.124549+0000', 'last_peered': '2024-09-16T10:27:51.124549+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:51.124549+0000', 'last_undegraded': '2024-09-16T10:27:51.124549+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'1200", 'ondisk_log_start': "35'1200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37756928, 'num_objects': 9218, 'num_object_clones': 0, 'num_object_copies': 18436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2136, 'num_objects_unfound': 0, 'num_objects_dirty': 9218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11250, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2056, 'num_bytes_recovered': 8421376, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7082}, {'shards': '0,2', 'objects': 2136}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:54.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:54.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'11350", 'reported_seq': 11551, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:51.257685+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:51.257685+0000', 'last_peered': '2024-09-16T10:27:51.257685+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:51.257685+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'1300", 'ondisk_log_start': "35'1300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38146072, 'num_objects': 9314, 'num_object_clones': 0, 'num_object_copies': 18628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4329, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9314, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7407, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3319}, {'shards': '2', 'objects': 4986}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:54.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:54.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'11138", 'reported_seq': 11330, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:51.084030+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:51.084030+0000', 'last_peered': '2024-09-16T10:27:51.084030+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:51.084030+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'1100", 'ondisk_log_start': "35'1100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37818368, 'num_objects': 9233, 'num_object_clones': 0, 'num_object_copies': 18466, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4205, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7763, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5030}, {'shards': '0,2', 'objects': 3267}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:54.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:54.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'11524", 'reported_seq': 11762, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:51.131995+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:51.131995+0000', 'last_peered': '2024-09-16T10:27:51.131995+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:51.131995+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'1500", 'ondisk_log_start': "35'1500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38486016, 'num_objects': 9396, 'num_object_clones': 0, 'num_object_copies': 18792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9396, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7766, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5025}, {'shards': '0,2', 'objects': 3342}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:54.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:54.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'11587", 'reported_seq': 11708, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:51.096619+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:51.096619+0000', 'last_peered': '2024-09-16T10:27:51.096619+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:51.096619+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'1500", 'ondisk_log_start': "35'1500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38720454, 'num_objects': 9455, 'num_object_clones': 0, 'num_object_copies': 18910, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4415, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11585, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5042}, {'shards': '0,2', 'objects': 3371}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:54.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:54.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'11188", 'reported_seq': 11308, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:51.252679+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:51.252679+0000', 'last_peered': '2024-09-16T10:27:51.252679+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:51.252679+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'1100", 'ondisk_log_start': "35'1100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37811142, 'num_objects': 9233, 'num_object_clones': 0, 'num_object_copies': 18466, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4272, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11186, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3301}, {'shards': '2', 'objects': 4963}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:54.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:54.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'11297", 'reported_seq': 11398, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:51.173560+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:51.173560+0000', 'last_peered': '2024-09-16T10:27:51.173560+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:51.173560+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'1200", 'ondisk_log_start': "35'1200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38007750, 'num_objects': 9281, 'num_object_clones': 0, 'num_object_copies': 18562, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4187, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11295, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3186}, {'shards': '2', 'objects': 5096}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:54.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:54.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'10983", 'reported_seq': 11113, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:51.122519+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:51.122519+0000', 'last_peered': '2024-09-16T10:27:51.122519+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:51.122519+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'900", 'ondisk_log_start': "35'900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37073379, 'num_objects': 9052, 'num_object_clones': 0, 'num_object_copies': 18104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4135, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10982, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4918}, {'shards': '0,2', 'objects': 3194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:54.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:54.067 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'11508", 'reported_seq': 11667, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:51.085469+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:51.085469+0000', 'last_peered': '2024-09-16T10:27:51.085469+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:51.085469+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'1500", 'ondisk_log_start': "35'1500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38528425, 'num_objects': 9409, 'num_object_clones': 0, 'num_object_copies': 18818, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11505, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5085}, {'shards': '0,2', 'objects': 3290}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:54.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:54.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'11092", 'reported_seq': 11209, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:51.109897+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:51.109897+0000', 'last_peered': '2024-09-16T10:27:51.109897+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:51.109897+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'1000", 'ondisk_log_start': "35'1000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37061574, 'num_objects': 9050, 'num_object_clones': 0, 'num_object_copies': 18100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4151, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11090, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4901}, {'shards': '0,2', 'objects': 3147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:54.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:54.068 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'11502", 'reported_seq': 11645, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:51.099865+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:51.099865+0000', 'last_peered': '2024-09-16T10:27:51.099865+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:51.099865+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'1500", 'ondisk_log_start': "35'1500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38479756, 'num_objects': 9398, 'num_object_clones': 0, 'num_object_copies': 18796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11499, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 5028}, {'shards': '0,2', 'objects': 3347}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:54.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:27:54.740 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:54.741 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42767/296538 objects degraded (14.422%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":363,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":148269,"data_bytes":607232545,"bytes_used":2160451584,"bytes_avail":287749840896,"bytes_total":289910292480,"degraded_objects":42767,"degraded_total":296538,"degraded_ratio":0.14422097673822579,"misplaced_objects":2136,"misplaced_total":296538,"misplaced_ratio":0.0072031240515549443,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":135129,"recovering_keys_per_sec":0,"num_objects_recovered":452,"num_bytes_recovered":1622016,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":506},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:27:54.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:27:54.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:55.313 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:55.314 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:55.338 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-16T10:27:54.658480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606622241,"num_objects":148120,"num_object_clones":0,"num_object_copies":296240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42767,"num_objects_misplaced":2136,"num_objects_unfound":0,"num_objects_dirty":148120,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158189,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6444,"num_bytes_recovered":25616886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2117324,"kb_used_data":1252204,"kb_used_omap":12,"kb_used_meta":865075,"kb_avail":280998196,"statfs":{"total":289910292480,"available":287742152704,"internally_reserved":0,"allocated":1282256896,"data_stored":1281419204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":885836802},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,5,7,10,5,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17371136,"num_objects":-4241,"num_object_clones":0,"num_object_copies":-8482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-51,"num_objects_misplaced":-283,"num_objects_unfound":0,"num_objects_dirty":-4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":1159168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003174"},"pg_stats":[{"pgid":"2.f","version":"44'11250","reported_seq":15432,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:51.124549+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:51.124549+0000","last_peered":"2024-09-16T10:27:51.124549+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:51.124549+0000","last_undegraded":"2024-09-16T10:27:51.124549+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37756928,"num_objects":9218,"num_object_clones":0,"num_object_copies":18436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2136,"num_objects_unfound":0,"num_objects_dirty":9218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11250,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2056,"num_bytes_recovered":8421376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7082},{"shards":"0,2","objects":2136}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11223","reported_seq":11380,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.191271+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:51.191271+0000","last_peered":"2024-09-16T10:27:51.191271+0000","last_clean":"2024-09-16T10:27:51.191271+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:51.191271+0000","last_undegraded":"2024-09-16T10:27:51.191271+0000","last_fullsized":"2024-09-16T10:27:51.191271+0000","mapping_epoch":34,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37842944,"num_objects":9239,"num_object_clones":0,"num_object_copies":18478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7436,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11350","reported_seq":11551,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.257685+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:51.257685+0000","last_peered":"2024-09-16T10:27:51.257685+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:51.257685+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38146072,"num_objects":9314,"num_object_clones":0,"num_object_copies":18628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9314,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7407,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3319},{"shards":"2","objects":4986}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11257","reported_seq":11314,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.172858+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:51.172858+0000","last_peered":"2024-09-16T10:27:51.172858+0000","last_clean":"2024-09-16T10:27:51.172858+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:51.172858+0000","last_undegraded":"2024-09-16T10:27:51.172858+0000","last_fullsized":"2024-09-16T10:27:51.172858+0000","mapping_epoch":34,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37703680,"num_objects":9205,"num_object_clones":0,"num_object_copies":18410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11257,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11252","reported_seq":11409,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.065032+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:51.065032+0000","last_peered":"2024-09-16T10:27:51.065032+0000","last_clean":"2024-09-16T10:27:51.065032+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:51.065032+0000","last_undegraded":"2024-09-16T10:27:51.065032+0000","last_fullsized":"2024-09-16T10:27:51.065032+0000","mapping_epoch":33,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37900288,"num_objects":9253,"num_object_clones":0,"num_object_copies":18506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7894,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11138","reported_seq":11330,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.084030+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:51.084030+0000","last_peered":"2024-09-16T10:27:51.084030+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:51.084030+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37818368,"num_objects":9233,"num_object_clones":0,"num_object_copies":18466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7763,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5030},{"shards":"0,2","objects":3267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11524","reported_seq":11762,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.131995+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:51.131995+0000","last_peered":"2024-09-16T10:27:51.131995+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:51.131995+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38486016,"num_objects":9396,"num_object_clones":0,"num_object_copies":18792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7766,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5025},{"shards":"0,2","objects":3342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11131","reported_seq":11306,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:51.136013+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:51.136013+0000","last_peered":"2024-09-16T10:27:51.136013+0000","last_clean":"2024-09-16T10:27:51.136013+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:51.136013+0000","last_undegraded":"2024-09-16T10:27:51.136013+0000","last_fullsized":"2024-09-16T10:27:51.136013+0000","mapping_epoch":33,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37924864,"num_objects":9259,"num_object_clones":0,"num_object_copies":18518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6934,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11340","reported_seq":19843,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:54.617945+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:54.617945+0000","last_peered":"2024-09-16T10:27:54.617945+0000","last_clean":"2024-09-16T10:27:54.617945+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:54.617945+0000","last_undegraded":"2024-09-16T10:27:54.617945+0000","last_fullsized":"2024-09-16T10:27:54.617945+0000","mapping_epoch":43,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37360582,"num_objects":9123,"num_object_clones":0,"num_object_copies":18246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11338,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11587","reported_seq":11708,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.096619+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:51.096619+0000","last_peered":"2024-09-16T10:27:51.096619+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:51.096619+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38720454,"num_objects":9455,"num_object_clones":0,"num_object_copies":18910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11585,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5042},{"shards":"0,2","objects":3371}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11188","reported_seq":11308,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.252679+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:51.252679+0000","last_peered":"2024-09-16T10:27:51.252679+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:51.252679+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37811142,"num_objects":9233,"num_object_clones":0,"num_object_copies":18466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11186,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3301},{"shards":"2","objects":4963}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11297","reported_seq":11398,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.173560+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:51.173560+0000","last_peered":"2024-09-16T10:27:51.173560+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:51.173560+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38007750,"num_objects":9281,"num_object_clones":0,"num_object_copies":18562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3186},{"shards":"2","objects":5096}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'10983","reported_seq":11113,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.122519+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:51.122519+0000","last_peered":"2024-09-16T10:27:51.122519+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:51.122519+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'900","ondisk_log_start":"35'900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37073379,"num_objects":9052,"num_object_clones":0,"num_object_copies":18104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4918},{"shards":"0,2","objects":3194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11508","reported_seq":11667,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.085469+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:51.085469+0000","last_peered":"2024-09-16T10:27:51.085469+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:51.085469+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38528425,"num_objects":9409,"num_object_clones":0,"num_object_copies":18818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11505,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5085},{"shards":"0,2","objects":3290}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11092","reported_seq":11209,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.109897+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:51.109897+0000","last_peered":"2024-09-16T10:27:51.109897+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:51.109897+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1000","ondisk_log_start":"35'1000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37061574,"num_objects":9050,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11090,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4901},{"shards":"0,2","objects":3147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11502","reported_seq":11645,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:51.099865+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:51.099865+0000","last_peered":"2024-09-16T10:27:51.099865+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:51.099865+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38479756,"num_objects":9398,"num_object_clones":0,"num_object_copies":18796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11499,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":5028},{"shards":"0,2","objects":3347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":606622222,"num_objects":148118,"num_object_clones":0,"num_object_copies":296236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42767,"num_objects_misplaced":2136,"num_objects_unfound":0,"num_objects_dirty":148118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158187,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6443,"num_bytes_recovered":25616867,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1280765952,"data_stored":1280674709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018944,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":699944,"kb_used_data":433692,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93671896,"statfs":{"total":96636764160,"available":95920021504,"internally_reserved":0,"allocated":444100608,"data_stored":443842902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":272625328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.0289999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":3.8959999999999999},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":1.9059999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557724,"kb_used_data":305428,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93814116,"statfs":{"total":96636764160,"available":96065654784,"internally_reserved":0,"allocated":312758272,"data_stored":312428306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":258338474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.8939999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":1.135},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.3529999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215172,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859656,"kb_used_data":513084,"kb_used_omap":4,"kb_used_meta":346555,"kb_avail":93512184,"statfs":{"total":96636764160,"available":95756476416,"internally_reserved":0,"allocated":525398016,"data_stored":525147996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":354873000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,3,2,0,0,0,0,6],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":5.2910000000000004},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":10.477}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.497},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":5.5190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":524898304,"data_stored":524894691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312287232,"data_stored":312207287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":443580416,"data_stored":443572731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:58.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:58.860 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:58.861 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:58.888 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-16T10:27:58.659506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598352417,"num_objects":146101,"num_object_clones":0,"num_object_copies":292202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42762,"num_objects_misplaced":2032,"num_objects_unfound":0,"num_objects_dirty":146101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160208,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6553,"num_bytes_recovered":26042870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160844,"ondisk_log_size":160844,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2116640,"kb_used_data":1241672,"kb_used_omap":12,"kb_used_meta":874931,"kb_avail":280998880,"statfs":{"total":289910292480,"available":287742853120,"internally_reserved":0,"allocated":1271472128,"data_stored":1270634436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":895929346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3,9,5,5,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17272832,"num_objects":-4217,"num_object_clones":0,"num_object_copies":-8434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-33,"num_objects_misplaced":-247,"num_objects_unfound":0,"num_objects_dirty":-4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4217,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":1011712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002993"},"pg_stats":[{"pgid":"2.f","version":"44'11379","reported_seq":15767,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:56.230822+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:56.230822+0000","last_peered":"2024-09-16T10:27:56.230822+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:56.230822+0000","last_undegraded":"2024-09-16T10:27:56.230822+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37228544,"num_objects":9089,"num_object_clones":0,"num_object_copies":18178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2032,"num_objects_unfound":0,"num_objects_dirty":9089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11379,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2160,"num_bytes_recovered":8847360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7057},{"shards":"0,2","objects":2032}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11347","reported_seq":11504,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.225948+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:56.225948+0000","last_peered":"2024-09-16T10:27:56.225948+0000","last_clean":"2024-09-16T10:27:56.225948+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:56.225948+0000","last_undegraded":"2024-09-16T10:27:56.225948+0000","last_fullsized":"2024-09-16T10:27:56.225948+0000","mapping_epoch":34,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37335040,"num_objects":9115,"num_object_clones":0,"num_object_copies":18230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11490","reported_seq":11691,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.177572+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:56.177572+0000","last_peered":"2024-09-16T10:27:56.177572+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:56.177572+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37572632,"num_objects":9174,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7547,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3179},{"shards":"2","objects":4846}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11384","reported_seq":11441,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.240011+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:56.240011+0000","last_peered":"2024-09-16T10:27:56.240011+0000","last_clean":"2024-09-16T10:27:56.240011+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:56.240011+0000","last_undegraded":"2024-09-16T10:27:56.240011+0000","last_fullsized":"2024-09-16T10:27:56.240011+0000","mapping_epoch":34,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37183488,"num_objects":9078,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11384,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11384","reported_seq":11541,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.171315+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:56.171315+0000","last_peered":"2024-09-16T10:27:56.171315+0000","last_clean":"2024-09-16T10:27:56.171315+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:56.171315+0000","last_undegraded":"2024-09-16T10:27:56.171315+0000","last_fullsized":"2024-09-16T10:27:56.171315+0000","mapping_epoch":33,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37359616,"num_objects":9121,"num_object_clones":0,"num_object_copies":18242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8026,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11274","reported_seq":11470,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.189492+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:56.189492+0000","last_peered":"2024-09-16T10:27:56.189492+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:56.189492+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37261312,"num_objects":9097,"num_object_clones":0,"num_object_copies":18194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7899,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4896},{"shards":"0,2","objects":3131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11658","reported_seq":11896,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.203708+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:56.203708+0000","last_peered":"2024-09-16T10:27:56.203708+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:56.203708+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37937152,"num_objects":9262,"num_object_clones":0,"num_object_copies":18524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7900,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4891},{"shards":"0,2","objects":3208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11262","reported_seq":11436,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.178067+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:56.178067+0000","last_peered":"2024-09-16T10:27:56.178067+0000","last_clean":"2024-09-16T10:27:56.178067+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:56.178067+0000","last_undegraded":"2024-09-16T10:27:56.178067+0000","last_fullsized":"2024-09-16T10:27:56.178067+0000","mapping_epoch":33,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37388288,"num_objects":9128,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7065,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11340","reported_seq":19843,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:54.617945+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:54.617945+0000","last_peered":"2024-09-16T10:27:54.617945+0000","last_clean":"2024-09-16T10:27:54.617945+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:54.617945+0000","last_undegraded":"2024-09-16T10:27:54.617945+0000","last_fullsized":"2024-09-16T10:27:54.617945+0000","mapping_epoch":43,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37360582,"num_objects":9123,"num_object_clones":0,"num_object_copies":18246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11338,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11729","reported_seq":11850,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.193012+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:56.193012+0000","last_peered":"2024-09-16T10:27:56.193012+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:56.193012+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38138822,"num_objects":9313,"num_object_clones":0,"num_object_copies":18626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11727,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4900},{"shards":"0,2","objects":3229}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11327","reported_seq":11447,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.139637+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:56.139637+0000","last_peered":"2024-09-16T10:27:56.139637+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:56.139637+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37241798,"num_objects":9094,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3162},{"shards":"2","objects":4824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11431","reported_seq":11536,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.193843+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:56.193843+0000","last_peered":"2024-09-16T10:27:56.193843+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:56.193843+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37458886,"num_objects":9147,"num_object_clones":0,"num_object_copies":18294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11429,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3052},{"shards":"2","objects":4964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11113","reported_seq":11245,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.147620+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:56.147620+0000","last_peered":"2024-09-16T10:27:56.147620+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:56.147620+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36540899,"num_objects":8922,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11112,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4789},{"shards":"0,2","objects":3064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11636","reported_seq":11793,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.153304+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:56.153304+0000","last_peered":"2024-09-16T10:27:56.153304+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:56.153304+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38004137,"num_objects":9281,"num_object_clones":0,"num_object_copies":18562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11633,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4957},{"shards":"0,2","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11238","reported_seq":11355,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.149401+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:56.149401+0000","last_peered":"2024-09-16T10:27:56.149401+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:56.149401+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36463558,"num_objects":8904,"num_object_clones":0,"num_object_copies":17808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11236,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4755},{"shards":"0,2","objects":3001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11649","reported_seq":11792,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.132510+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:56.132510+0000","last_peered":"2024-09-16T10:27:56.132510+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:56.132510+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37877644,"num_objects":9251,"num_object_clones":0,"num_object_copies":18502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4881},{"shards":"0,2","objects":3200}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":598352398,"num_objects":146099,"num_object_clones":0,"num_object_copies":292198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42762,"num_objects_misplaced":2032,"num_objects_unfound":0,"num_objects_dirty":146099,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160206,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6552,"num_bytes_recovered":26042851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1269784576,"data_stored":1269693333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018945,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717828,"kb_used_data":429436,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93654012,"statfs":{"total":96636764160,"available":95901708288,"internally_reserved":0,"allocated":439742464,"data_stored":439484758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.361},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.9169999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":6.7249999999999996},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":3.887}]}]},{"osd":1,"up_from":40,"seq":171798691889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557724,"kb_used_data":305428,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93814116,"statfs":{"total":96636764160,"available":96065654784,"internally_reserved":0,"allocated":312758272,"data_stored":312428306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":258338474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.8939999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":1.135},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.3529999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":841088,"kb_used_data":506808,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93530752,"statfs":{"total":96636764160,"available":95775490048,"internally_reserved":0,"allocated":518971392,"data_stored":518721372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,1,3,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.871},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.83299999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.95099999999999996},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.89900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":518307840,"data_stored":518304227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312287232,"data_stored":312207287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":439189504,"data_stored":439181819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:58.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:27:59.429 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:27:59.429 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:27:59.455 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-16T10:27:58.659506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598352417,"num_objects":146101,"num_object_clones":0,"num_object_copies":292202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42762,"num_objects_misplaced":2032,"num_objects_unfound":0,"num_objects_dirty":146101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160208,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6553,"num_bytes_recovered":26042870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160844,"ondisk_log_size":160844,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2116640,"kb_used_data":1241672,"kb_used_omap":12,"kb_used_meta":874931,"kb_avail":280998880,"statfs":{"total":289910292480,"available":287742853120,"internally_reserved":0,"allocated":1271472128,"data_stored":1270634436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":895929346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,3,9,5,5,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17272832,"num_objects":-4217,"num_object_clones":0,"num_object_copies":-8434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-33,"num_objects_misplaced":-247,"num_objects_unfound":0,"num_objects_dirty":-4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4217,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":1011712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002993"},"pg_stats":[{"pgid":"2.f","version":"44'11379","reported_seq":15767,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:56.230822+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:56.230822+0000","last_peered":"2024-09-16T10:27:56.230822+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:56.230822+0000","last_undegraded":"2024-09-16T10:27:56.230822+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37228544,"num_objects":9089,"num_object_clones":0,"num_object_copies":18178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2032,"num_objects_unfound":0,"num_objects_dirty":9089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11379,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2160,"num_bytes_recovered":8847360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7057},{"shards":"0,2","objects":2032}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11347","reported_seq":11504,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.225948+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:56.225948+0000","last_peered":"2024-09-16T10:27:56.225948+0000","last_clean":"2024-09-16T10:27:56.225948+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:56.225948+0000","last_undegraded":"2024-09-16T10:27:56.225948+0000","last_fullsized":"2024-09-16T10:27:56.225948+0000","mapping_epoch":34,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37335040,"num_objects":9115,"num_object_clones":0,"num_object_copies":18230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11490","reported_seq":11691,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.177572+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:56.177572+0000","last_peered":"2024-09-16T10:27:56.177572+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:56.177572+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37572632,"num_objects":9174,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7547,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3179},{"shards":"2","objects":4846}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11384","reported_seq":11441,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.240011+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:56.240011+0000","last_peered":"2024-09-16T10:27:56.240011+0000","last_clean":"2024-09-16T10:27:56.240011+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:56.240011+0000","last_undegraded":"2024-09-16T10:27:56.240011+0000","last_fullsized":"2024-09-16T10:27:56.240011+0000","mapping_epoch":34,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37183488,"num_objects":9078,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11384,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11384","reported_seq":11541,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.171315+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:56.171315+0000","last_peered":"2024-09-16T10:27:56.171315+0000","last_clean":"2024-09-16T10:27:56.171315+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:56.171315+0000","last_undegraded":"2024-09-16T10:27:56.171315+0000","last_fullsized":"2024-09-16T10:27:56.171315+0000","mapping_epoch":33,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37359616,"num_objects":9121,"num_object_clones":0,"num_object_copies":18242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8026,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11274","reported_seq":11470,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.189492+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:56.189492+0000","last_peered":"2024-09-16T10:27:56.189492+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:56.189492+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37261312,"num_objects":9097,"num_object_clones":0,"num_object_copies":18194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7899,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4896},{"shards":"0,2","objects":3131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11658","reported_seq":11896,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.203708+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:56.203708+0000","last_peered":"2024-09-16T10:27:56.203708+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:56.203708+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37937152,"num_objects":9262,"num_object_clones":0,"num_object_copies":18524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7900,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4891},{"shards":"0,2","objects":3208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11262","reported_seq":11436,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.178067+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:56.178067+0000","last_peered":"2024-09-16T10:27:56.178067+0000","last_clean":"2024-09-16T10:27:56.178067+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:56.178067+0000","last_undegraded":"2024-09-16T10:27:56.178067+0000","last_fullsized":"2024-09-16T10:27:56.178067+0000","mapping_epoch":33,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37388288,"num_objects":9128,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7065,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11340","reported_seq":19843,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:54.617945+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:54.617945+0000","last_peered":"2024-09-16T10:27:54.617945+0000","last_clean":"2024-09-16T10:27:54.617945+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:54.617945+0000","last_undegraded":"2024-09-16T10:27:54.617945+0000","last_fullsized":"2024-09-16T10:27:54.617945+0000","mapping_epoch":43,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37360582,"num_objects":9123,"num_object_clones":0,"num_object_copies":18246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11338,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11729","reported_seq":11850,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.193012+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:56.193012+0000","last_peered":"2024-09-16T10:27:56.193012+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:56.193012+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38138822,"num_objects":9313,"num_object_clones":0,"num_object_copies":18626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11727,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4900},{"shards":"0,2","objects":3229}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11327","reported_seq":11447,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.139637+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:56.139637+0000","last_peered":"2024-09-16T10:27:56.139637+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:56.139637+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37241798,"num_objects":9094,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3162},{"shards":"2","objects":4824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11431","reported_seq":11536,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.193843+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:56.193843+0000","last_peered":"2024-09-16T10:27:56.193843+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:56.193843+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37458886,"num_objects":9147,"num_object_clones":0,"num_object_copies":18294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11429,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3052},{"shards":"2","objects":4964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11113","reported_seq":11245,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.147620+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:56.147620+0000","last_peered":"2024-09-16T10:27:56.147620+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:56.147620+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36540899,"num_objects":8922,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11112,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4789},{"shards":"0,2","objects":3064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11636","reported_seq":11793,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.153304+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:56.153304+0000","last_peered":"2024-09-16T10:27:56.153304+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:56.153304+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38004137,"num_objects":9281,"num_object_clones":0,"num_object_copies":18562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11633,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4957},{"shards":"0,2","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11238","reported_seq":11355,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.149401+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:56.149401+0000","last_peered":"2024-09-16T10:27:56.149401+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:56.149401+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36463558,"num_objects":8904,"num_object_clones":0,"num_object_copies":17808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11236,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4755},{"shards":"0,2","objects":3001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11649","reported_seq":11792,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.132510+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:56.132510+0000","last_peered":"2024-09-16T10:27:56.132510+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:56.132510+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37877644,"num_objects":9251,"num_object_clones":0,"num_object_copies":18502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4881},{"shards":"0,2","objects":3200}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":598352398,"num_objects":146099,"num_object_clones":0,"num_object_copies":292198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42762,"num_objects_misplaced":2032,"num_objects_unfound":0,"num_objects_dirty":146099,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160206,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6552,"num_bytes_recovered":26042851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1269784576,"data_stored":1269693333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018945,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717828,"kb_used_data":429436,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93654012,"statfs":{"total":96636764160,"available":95901708288,"internally_reserved":0,"allocated":439742464,"data_stored":439484758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.361},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.9169999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":6.7249999999999996},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":3.887}]}]},{"osd":1,"up_from":40,"seq":171798691889,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557724,"kb_used_data":305428,"kb_used_omap":4,"kb_used_meta":252283,"kb_avail":93814116,"statfs":{"total":96636764160,"available":96065654784,"internally_reserved":0,"allocated":312758272,"data_stored":312428306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":258338474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.8939999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":1.135},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.3529999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":841088,"kb_used_data":506808,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93530752,"statfs":{"total":96636764160,"available":95775490048,"internally_reserved":0,"allocated":518971392,"data_stored":518721372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,1,3,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.871},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.83299999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.95099999999999996},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.89900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":518307840,"data_stored":518304227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":312287232,"data_stored":312207287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":439189504,"data_stored":439181819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:27:59.456 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:27:59.457 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'11379", 'reported_seq': 15767, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:27:56.230822+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:27:56.230822+0000', 'last_peered': '2024-09-16T10:27:56.230822+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:27:56.230822+0000', 'last_undegraded': '2024-09-16T10:27:56.230822+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'1300", 'ondisk_log_start': "35'1300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37228544, 'num_objects': 9089, 'num_object_clones': 0, 'num_object_copies': 18178, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2032, 'num_objects_unfound': 0, 'num_objects_dirty': 9089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11379, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2160, 'num_bytes_recovered': 8847360, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7057}, {'shards': '0,2', 'objects': 2032}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:59.457 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:27:59.457 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'11490", 'reported_seq': 11691, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:56.177572+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:27:56.177572+0000', 'last_peered': '2024-09-16T10:27:56.177572+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:27:56.177572+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'1400", 'ondisk_log_start': "35'1400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37572632, 'num_objects': 9174, 'num_object_clones': 0, 'num_object_copies': 18348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4329, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9174, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7547, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3179}, {'shards': '2', 'objects': 4846}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:59.457 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:27:59.457 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'11274", 'reported_seq': 11470, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:56.189492+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:27:56.189492+0000', 'last_peered': '2024-09-16T10:27:56.189492+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:27:56.189492+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'1200", 'ondisk_log_start': "35'1200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37261312, 'num_objects': 9097, 'num_object_clones': 0, 'num_object_copies': 18194, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4203, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9097, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7899, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4896}, {'shards': '0,2', 'objects': 3131}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:59.457 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:27:59.457 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'11658", 'reported_seq': 11896, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:56.203708+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:27:56.203708+0000', 'last_peered': '2024-09-16T10:27:56.203708+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:27:56.203708+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'1600", 'ondisk_log_start': "35'1600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37937152, 'num_objects': 9262, 'num_object_clones': 0, 'num_object_copies': 18524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7900, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4891}, {'shards': '0,2', 'objects': 3208}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:59.457 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:27:59.457 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'11729", 'reported_seq': 11850, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:56.193012+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:27:56.193012+0000', 'last_peered': '2024-09-16T10:27:56.193012+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:27:56.193012+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'1700", 'ondisk_log_start': "35'1700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38138822, 'num_objects': 9313, 'num_object_clones': 0, 'num_object_copies': 18626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4415, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11727, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4900}, {'shards': '0,2', 'objects': 3229}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:59.458 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:27:59.458 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'11327", 'reported_seq': 11447, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:56.139637+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:27:56.139637+0000', 'last_peered': '2024-09-16T10:27:56.139637+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:27:56.139637+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'1300", 'ondisk_log_start': "35'1300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37241798, 'num_objects': 9094, 'num_object_clones': 0, 'num_object_copies': 18188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4272, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11325, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3162}, {'shards': '2', 'objects': 4824}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:59.458 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:27:59.458 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'11431", 'reported_seq': 11536, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:56.193843+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:27:56.193843+0000', 'last_peered': '2024-09-16T10:27:56.193843+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:27:56.193843+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'1400", 'ondisk_log_start': "35'1400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37458886, 'num_objects': 9147, 'num_object_clones': 0, 'num_object_copies': 18294, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4185, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11429, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3052}, {'shards': '2', 'objects': 4964}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:27:59.458 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:27:59.458 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'11113", 'reported_seq': 11245, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:56.147620+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:27:56.147620+0000', 'last_peered': '2024-09-16T10:27:56.147620+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:27:56.147620+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'1100", 'ondisk_log_start': "35'1100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36540899, 'num_objects': 8922, 'num_object_clones': 0, 'num_object_copies': 17844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11112, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4789}, {'shards': '0,2', 'objects': 3064}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:59.458 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:27:59.458 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'11636", 'reported_seq': 11793, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:56.153304+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:27:56.153304+0000', 'last_peered': '2024-09-16T10:27:56.153304+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:27:56.153304+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'1600", 'ondisk_log_start': "35'1600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38004137, 'num_objects': 9281, 'num_object_clones': 0, 'num_object_copies': 18562, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11633, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4957}, {'shards': '0,2', 'objects': 3162}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:59.458 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:27:59.458 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'11238", 'reported_seq': 11355, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:56.149401+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:27:56.149401+0000', 'last_peered': '2024-09-16T10:27:56.149401+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:27:56.149401+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'1200", 'ondisk_log_start': "35'1200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36463558, 'num_objects': 8904, 'num_object_clones': 0, 'num_object_copies': 17808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4151, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11236, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4755}, {'shards': '0,2', 'objects': 3001}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:59.459 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:27:59.459 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'11649", 'reported_seq': 11792, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:27:56.132510+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:27:56.132510+0000', 'last_peered': '2024-09-16T10:27:56.132510+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:27:56.132510+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'1600", 'ondisk_log_start': "35'1600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37877644, 'num_objects': 9251, 'num_object_clones': 0, 'num_object_copies': 18502, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11646, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4881}, {'shards': '0,2', 'objects': 3200}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:27:59.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:00.117 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42762/292202 objects degraded (14.634%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":368,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":146101,"data_bytes":598352417,"bytes_used":2167439360,"bytes_avail":287742853120,"bytes_total":289910292480,"degraded_objects":42762,"degraded_total":292202,"degraded_ratio":0.14634396752931192,"misplaced_objects":2032,"misplaced_total":292202,"misplaced_ratio":0.0069540934011403073,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":84288,"recovering_keys_per_sec":0,"num_objects_recovered":280,"num_bytes_recovered":1011712,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":351},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:00.144 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:00.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:00.710 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:00.711 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:00.739 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-16T10:28:00.659964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597852705,"num_objects":145979,"num_object_clones":0,"num_object_copies":291958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42762,"num_objects_misplaced":2032,"num_objects_unfound":0,"num_objects_dirty":145979,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160330,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6553,"num_bytes_recovered":26042870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2120240,"kb_used_data":1241176,"kb_used_omap":12,"kb_used_meta":879027,"kb_avail":280995280,"statfs":{"total":289910292480,"available":287739166720,"internally_reserved":0,"allocated":1270964224,"data_stored":1270126532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":900123650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,6,5,6,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17772544,"num_objects":-4339,"num_object_clones":0,"num_object_copies":-8678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-33,"num_objects_misplaced":-247,"num_objects_unfound":0,"num_objects_dirty":-4339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4339,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":1011712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002941"},"pg_stats":[{"pgid":"2.f","version":"44'11379","reported_seq":15767,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:27:56.230822+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:27:56.230822+0000","last_peered":"2024-09-16T10:27:56.230822+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:27:56.230822+0000","last_undegraded":"2024-09-16T10:27:56.230822+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37228544,"num_objects":9089,"num_object_clones":0,"num_object_copies":18178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2032,"num_objects_unfound":0,"num_objects_dirty":9089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11379,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2160,"num_bytes_recovered":8847360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7057},{"shards":"0,2","objects":2032}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11347","reported_seq":11504,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.225948+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:27:56.225948+0000","last_peered":"2024-09-16T10:27:56.225948+0000","last_clean":"2024-09-16T10:27:56.225948+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:27:56.225948+0000","last_undegraded":"2024-09-16T10:27:56.225948+0000","last_fullsized":"2024-09-16T10:27:56.225948+0000","mapping_epoch":34,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37335040,"num_objects":9115,"num_object_clones":0,"num_object_copies":18230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11490","reported_seq":11691,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.177572+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:27:56.177572+0000","last_peered":"2024-09-16T10:27:56.177572+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:27:56.177572+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37572632,"num_objects":9174,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7547,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3179},{"shards":"2","objects":4846}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11384","reported_seq":11441,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.240011+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:27:56.240011+0000","last_peered":"2024-09-16T10:27:56.240011+0000","last_clean":"2024-09-16T10:27:56.240011+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:27:56.240011+0000","last_undegraded":"2024-09-16T10:27:56.240011+0000","last_fullsized":"2024-09-16T10:27:56.240011+0000","mapping_epoch":34,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37183488,"num_objects":9078,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11384,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11384","reported_seq":11541,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.171315+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:27:56.171315+0000","last_peered":"2024-09-16T10:27:56.171315+0000","last_clean":"2024-09-16T10:27:56.171315+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:27:56.171315+0000","last_undegraded":"2024-09-16T10:27:56.171315+0000","last_fullsized":"2024-09-16T10:27:56.171315+0000","mapping_epoch":33,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37359616,"num_objects":9121,"num_object_clones":0,"num_object_copies":18242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8026,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11274","reported_seq":11470,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.189492+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:27:56.189492+0000","last_peered":"2024-09-16T10:27:56.189492+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:27:56.189492+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37261312,"num_objects":9097,"num_object_clones":0,"num_object_copies":18194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7899,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4896},{"shards":"0,2","objects":3131}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11658","reported_seq":11896,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.203708+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:27:56.203708+0000","last_peered":"2024-09-16T10:27:56.203708+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:27:56.203708+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37937152,"num_objects":9262,"num_object_clones":0,"num_object_copies":18524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7900,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4891},{"shards":"0,2","objects":3208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11262","reported_seq":11436,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:56.178067+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:27:56.178067+0000","last_peered":"2024-09-16T10:27:56.178067+0000","last_clean":"2024-09-16T10:27:56.178067+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:27:56.178067+0000","last_undegraded":"2024-09-16T10:27:56.178067+0000","last_fullsized":"2024-09-16T10:27:56.178067+0000","mapping_epoch":33,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37388288,"num_objects":9128,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7065,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11462","reported_seq":19965,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:59.530472+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:59.530472+0000","last_peered":"2024-09-16T10:27:59.530472+0000","last_clean":"2024-09-16T10:27:59.530472+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:59.530472+0000","last_undegraded":"2024-09-16T10:27:59.530472+0000","last_fullsized":"2024-09-16T10:27:59.530472+0000","mapping_epoch":43,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36860870,"num_objects":9001,"num_object_clones":0,"num_object_copies":18002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11729","reported_seq":11850,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.193012+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:27:56.193012+0000","last_peered":"2024-09-16T10:27:56.193012+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:27:56.193012+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38138822,"num_objects":9313,"num_object_clones":0,"num_object_copies":18626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4415,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11727,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4900},{"shards":"0,2","objects":3229}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11327","reported_seq":11447,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.139637+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:27:56.139637+0000","last_peered":"2024-09-16T10:27:56.139637+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:27:56.139637+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37241798,"num_objects":9094,"num_object_clones":0,"num_object_copies":18188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3162},{"shards":"2","objects":4824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11431","reported_seq":11536,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.193843+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:27:56.193843+0000","last_peered":"2024-09-16T10:27:56.193843+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:27:56.193843+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37458886,"num_objects":9147,"num_object_clones":0,"num_object_copies":18294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11429,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3052},{"shards":"2","objects":4964}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11113","reported_seq":11245,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.147620+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:27:56.147620+0000","last_peered":"2024-09-16T10:27:56.147620+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:27:56.147620+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1100","ondisk_log_start":"35'1100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36540899,"num_objects":8922,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11112,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4789},{"shards":"0,2","objects":3064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11636","reported_seq":11793,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.153304+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:27:56.153304+0000","last_peered":"2024-09-16T10:27:56.153304+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:27:56.153304+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38004137,"num_objects":9281,"num_object_clones":0,"num_object_copies":18562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11633,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4957},{"shards":"0,2","objects":3162}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11238","reported_seq":11355,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.149401+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:27:56.149401+0000","last_peered":"2024-09-16T10:27:56.149401+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:27:56.149401+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36463558,"num_objects":8904,"num_object_clones":0,"num_object_copies":17808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11236,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4755},{"shards":"0,2","objects":3001}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11649","reported_seq":11792,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:27:56.132510+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:27:56.132510+0000","last_peered":"2024-09-16T10:27:56.132510+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:27:56.132510+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37877644,"num_objects":9251,"num_object_clones":0,"num_object_copies":18502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11646,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4881},{"shards":"0,2","objects":3200}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":597852686,"num_objects":145977,"num_object_clones":0,"num_object_copies":291954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42762,"num_objects_misplaced":2032,"num_objects_unfound":0,"num_objects_dirty":145977,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160328,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6552,"num_bytes_recovered":26042851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1269317632,"data_stored":1269226389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160863,"ondisk_log_size":160863,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018945,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717828,"kb_used_data":429436,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93654012,"statfs":{"total":96636764160,"available":95901708288,"internally_reserved":0,"allocated":439742464,"data_stored":439484758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,2],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.3320000000000001,"15min":1.018},"min":{"1min":0.23599999999999999,"5min":0.184,"15min":0.184},"max":{"1min":6.274,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.361},{"interface":"front","average":{"1min":2.3370000000000002,"5min":2.0299999999999998,"15min":1.276},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":7.734,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.9169999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":2.6640000000000001,"15min":2.8199999999999998},"min":{"1min":0.32000000000000001,"5min":0.246,"15min":0.246},"max":{"1min":5.9790000000000001,"5min":10.492000000000001,"15min":10.492000000000001},"last":6.7249999999999996},{"interface":"front","average":{"1min":2.8599999999999999,"5min":2.3479999999999999,"15min":2.2759999999999998},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":13.807,"5min":13.807,"15min":13.807},"last":3.887}]}]},{"osd":1,"up_from":40,"seq":171798691890,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561324,"kb_used_data":304932,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93810516,"statfs":{"total":96636764160,"available":96061968384,"internally_reserved":0,"allocated":312250368,"data_stored":311920402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":262532778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.34000000000000002},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.31900000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215173,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":841088,"kb_used_data":506808,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93530752,"statfs":{"total":96636764160,"available":95775490048,"internally_reserved":0,"allocated":518971392,"data_stored":518721372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":342290088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,1,3,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.871},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.83299999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.95099999999999996},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.89900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":518307840,"data_stored":518304227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311820288,"data_stored":311740343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":439189504,"data_stored":439181819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:03.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:04.308 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:04.311 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:04.337 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-16T10:28:02.660462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":589799969,"num_objects":144013,"num_object_clones":0,"num_object_copies":288026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42735,"num_objects_misplaced":1890,"num_objects_unfound":0,"num_objects_dirty":144013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162296,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6722,"num_bytes_recovered":26624502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160832,"ondisk_log_size":160832,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2136364,"kb_used_data":1231044,"kb_used_omap":12,"kb_used_meta":905267,"kb_avail":280979156,"statfs":{"total":289910292480,"available":287722655744,"internally_reserved":0,"allocated":1260589056,"data_stored":1259751364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":926993410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,2,7,3,1,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25264128,"num_objects":-6168,"num_object_clones":0,"num_object_copies":-12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-60,"num_objects_misplaced":-389,"num_objects_unfound":0,"num_objects_dirty":-6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6168,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":1593344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002976"},"pg_stats":[{"pgid":"2.f","version":"44'11506","reported_seq":16180,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:01.270428+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:01.270428+0000","last_peered":"2024-09-16T10:28:01.270428+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:01.270428+0000","last_undegraded":"2024-09-16T10:28:01.270428+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36708352,"num_objects":8962,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1890,"num_objects_unfound":0,"num_objects_dirty":8962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2302,"num_bytes_recovered":9428992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7072},{"shards":"0,2","objects":1890}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11479","reported_seq":11635,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.222852+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:01.222852+0000","last_peered":"2024-09-16T10:28:01.222852+0000","last_clean":"2024-09-16T10:28:01.222852+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:01.222852+0000","last_undegraded":"2024-09-16T10:28:01.222852+0000","last_fullsized":"2024-09-16T10:28:01.222852+0000","mapping_epoch":34,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36794368,"num_objects":8983,"num_object_clones":0,"num_object_copies":17966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7692,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11626","reported_seq":11829,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.259460+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:01.259460+0000","last_peered":"2024-09-16T10:28:01.259460+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:01.259460+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37015576,"num_objects":9038,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9038,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7683,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3043},{"shards":"2","objects":4711}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11507","reported_seq":11564,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.221545+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:01.221545+0000","last_peered":"2024-09-16T10:28:01.221545+0000","last_clean":"2024-09-16T10:28:01.221545+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:01.221545+0000","last_undegraded":"2024-09-16T10:28:01.221545+0000","last_fullsized":"2024-09-16T10:28:01.221545+0000","mapping_epoch":34,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36679680,"num_objects":8955,"num_object_clones":0,"num_object_copies":17910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11517","reported_seq":11674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.275752+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:01.275752+0000","last_peered":"2024-09-16T10:28:01.275752+0000","last_clean":"2024-09-16T10:28:01.275752+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:01.275752+0000","last_undegraded":"2024-09-16T10:28:01.275752+0000","last_fullsized":"2024-09-16T10:28:01.275752+0000","mapping_epoch":33,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36814848,"num_objects":8988,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8159,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11410","reported_seq":11612,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.190016+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:01.190016+0000","last_peered":"2024-09-16T10:28:01.190016+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:01.190016+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36704256,"num_objects":8961,"num_object_clones":0,"num_object_copies":17922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8035,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4763},{"shards":"0,2","objects":2995}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11792","reported_seq":12039,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.244530+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:01.244530+0000","last_peered":"2024-09-16T10:28:01.244530+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:01.244530+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37388288,"num_objects":9128,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8034,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4762},{"shards":"0,2","objects":3074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11396","reported_seq":11571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.221090+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:01.221090+0000","last_peered":"2024-09-16T10:28:01.221090+0000","last_clean":"2024-09-16T10:28:01.221090+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:01.221090+0000","last_undegraded":"2024-09-16T10:28:01.221090+0000","last_fullsized":"2024-09-16T10:28:01.221090+0000","mapping_epoch":33,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36839424,"num_objects":8994,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7199,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11462","reported_seq":19965,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:59.530472+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:59.530472+0000","last_peered":"2024-09-16T10:27:59.530472+0000","last_clean":"2024-09-16T10:27:59.530472+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:59.530472+0000","last_undegraded":"2024-09-16T10:27:59.530472+0000","last_fullsized":"2024-09-16T10:27:59.530472+0000","mapping_epoch":43,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36860870,"num_objects":9001,"num_object_clones":0,"num_object_copies":18002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11884","reported_seq":12021,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.235837+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:01.235837+0000","last_peered":"2024-09-16T10:28:01.235837+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:01.235837+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37503942,"num_objects":9158,"num_object_clones":0,"num_object_copies":18316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11882,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4753},{"shards":"0,2","objects":3074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11439","reported_seq":11561,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.107007+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:01.107007+0000","last_peered":"2024-09-16T10:28:01.107007+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:01.107007+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36783046,"num_objects":8982,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11437,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3050},{"shards":"2","objects":4713}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11569","reported_seq":11673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.201756+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:01.201756+0000","last_peered":"2024-09-16T10:28:01.201756+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:01.201756+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36893638,"num_objects":9009,"num_object_clones":0,"num_object_copies":18018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11567,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2914},{"shards":"2","objects":4826}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11240","reported_seq":11375,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.169870+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:01.169870+0000","last_peered":"2024-09-16T10:28:01.169870+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:01.169870+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36020707,"num_objects":8795,"num_object_clones":0,"num_object_copies":17590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11239,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4664},{"shards":"0,2","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11764","reported_seq":11929,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.202180+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:01.202180+0000","last_peered":"2024-09-16T10:28:01.202180+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:01.202180+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37479849,"num_objects":9153,"num_object_clones":0,"num_object_copies":18306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11761,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4832},{"shards":"0,2","objects":3034}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11361","reported_seq":11482,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.194258+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:01.194258+0000","last_peered":"2024-09-16T10:28:01.194258+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:01.194258+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35959750,"num_objects":8781,"num_object_clones":0,"num_object_copies":17562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4634},{"shards":"0,2","objects":2878}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11777","reported_seq":11924,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.244626+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:01.244626+0000","last_peered":"2024-09-16T10:28:01.244626+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:01.244626+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37353356,"num_objects":9123,"num_object_clones":0,"num_object_copies":18246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11774,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4755},{"shards":"0,2","objects":3072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":589799950,"num_objects":144011,"num_object_clones":0,"num_object_copies":288022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42735,"num_objects_misplaced":1890,"num_objects_unfound":0,"num_objects_dirty":144011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162294,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6721,"num_bytes_recovered":26624483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1258684416,"data_stored":1258593173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160829,"ondisk_log_size":160829,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018946,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717944,"kb_used_data":425452,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93653896,"statfs":{"total":96636764160,"available":95901589504,"internally_reserved":0,"allocated":435662848,"data_stored":435405142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":15.157999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":6.976}]}]},{"osd":1,"up_from":40,"seq":171798691890,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561324,"kb_used_data":304932,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93810516,"statfs":{"total":96636764160,"available":96061968384,"internally_reserved":0,"allocated":312250368,"data_stored":311920402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":262532778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.34000000000000002},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.31900000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215174,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":857096,"kb_used_data":500660,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93514744,"statfs":{"total":96636764160,"available":95759097856,"internally_reserved":0,"allocated":512675840,"data_stored":512425820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":364965544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,0,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.433},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.45700000000000002},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.39300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":511864832,"data_stored":511861219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311820288,"data_stored":311740343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":434999296,"data_stored":434991611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:04.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:04.868 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:04.870 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:04.890 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-16T10:28:04.661132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":589799969,"num_objects":144013,"num_object_clones":0,"num_object_copies":288026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42735,"num_objects_misplaced":1890,"num_objects_unfound":0,"num_objects_dirty":144013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162296,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6722,"num_bytes_recovered":26624502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160832,"ondisk_log_size":160832,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2136364,"kb_used_data":1231044,"kb_used_omap":12,"kb_used_meta":905267,"kb_avail":280979156,"statfs":{"total":289910292480,"available":287722655744,"internally_reserved":0,"allocated":1260589056,"data_stored":1259751364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":926993410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,2,7,3,1,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17432576,"num_objects":-4256,"num_object_clones":0,"num_object_copies":-8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-32,"num_objects_misplaced":-246,"num_objects_unfound":0,"num_objects_dirty":-4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4256,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003164"},"pg_stats":[{"pgid":"2.f","version":"44'11506","reported_seq":16180,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:01.270428+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:01.270428+0000","last_peered":"2024-09-16T10:28:01.270428+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:01.270428+0000","last_undegraded":"2024-09-16T10:28:01.270428+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36708352,"num_objects":8962,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1890,"num_objects_unfound":0,"num_objects_dirty":8962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2302,"num_bytes_recovered":9428992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7072},{"shards":"0,2","objects":1890}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11479","reported_seq":11635,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.222852+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:01.222852+0000","last_peered":"2024-09-16T10:28:01.222852+0000","last_clean":"2024-09-16T10:28:01.222852+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:01.222852+0000","last_undegraded":"2024-09-16T10:28:01.222852+0000","last_fullsized":"2024-09-16T10:28:01.222852+0000","mapping_epoch":34,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36794368,"num_objects":8983,"num_object_clones":0,"num_object_copies":17966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7692,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11626","reported_seq":11829,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.259460+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:01.259460+0000","last_peered":"2024-09-16T10:28:01.259460+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:01.259460+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37015576,"num_objects":9038,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9038,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7683,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3043},{"shards":"2","objects":4711}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11507","reported_seq":11564,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.221545+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:01.221545+0000","last_peered":"2024-09-16T10:28:01.221545+0000","last_clean":"2024-09-16T10:28:01.221545+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:01.221545+0000","last_undegraded":"2024-09-16T10:28:01.221545+0000","last_fullsized":"2024-09-16T10:28:01.221545+0000","mapping_epoch":34,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36679680,"num_objects":8955,"num_object_clones":0,"num_object_copies":17910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11517","reported_seq":11674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.275752+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:01.275752+0000","last_peered":"2024-09-16T10:28:01.275752+0000","last_clean":"2024-09-16T10:28:01.275752+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:01.275752+0000","last_undegraded":"2024-09-16T10:28:01.275752+0000","last_fullsized":"2024-09-16T10:28:01.275752+0000","mapping_epoch":33,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36814848,"num_objects":8988,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8159,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11410","reported_seq":11612,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.190016+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:01.190016+0000","last_peered":"2024-09-16T10:28:01.190016+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:01.190016+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36704256,"num_objects":8961,"num_object_clones":0,"num_object_copies":17922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8035,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4763},{"shards":"0,2","objects":2995}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11792","reported_seq":12039,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.244530+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:01.244530+0000","last_peered":"2024-09-16T10:28:01.244530+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:01.244530+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37388288,"num_objects":9128,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8034,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4762},{"shards":"0,2","objects":3074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11396","reported_seq":11571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.221090+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:01.221090+0000","last_peered":"2024-09-16T10:28:01.221090+0000","last_clean":"2024-09-16T10:28:01.221090+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:01.221090+0000","last_undegraded":"2024-09-16T10:28:01.221090+0000","last_fullsized":"2024-09-16T10:28:01.221090+0000","mapping_epoch":33,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36839424,"num_objects":8994,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7199,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11462","reported_seq":19965,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:59.530472+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:59.530472+0000","last_peered":"2024-09-16T10:27:59.530472+0000","last_clean":"2024-09-16T10:27:59.530472+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:59.530472+0000","last_undegraded":"2024-09-16T10:27:59.530472+0000","last_fullsized":"2024-09-16T10:27:59.530472+0000","mapping_epoch":43,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36860870,"num_objects":9001,"num_object_clones":0,"num_object_copies":18002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11884","reported_seq":12021,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.235837+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:01.235837+0000","last_peered":"2024-09-16T10:28:01.235837+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:01.235837+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37503942,"num_objects":9158,"num_object_clones":0,"num_object_copies":18316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11882,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4753},{"shards":"0,2","objects":3074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11439","reported_seq":11561,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.107007+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:01.107007+0000","last_peered":"2024-09-16T10:28:01.107007+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:01.107007+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36783046,"num_objects":8982,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11437,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3050},{"shards":"2","objects":4713}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11569","reported_seq":11673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.201756+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:01.201756+0000","last_peered":"2024-09-16T10:28:01.201756+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:01.201756+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36893638,"num_objects":9009,"num_object_clones":0,"num_object_copies":18018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11567,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2914},{"shards":"2","objects":4826}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11240","reported_seq":11375,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.169870+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:01.169870+0000","last_peered":"2024-09-16T10:28:01.169870+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:01.169870+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36020707,"num_objects":8795,"num_object_clones":0,"num_object_copies":17590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11239,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4664},{"shards":"0,2","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11764","reported_seq":11929,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.202180+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:01.202180+0000","last_peered":"2024-09-16T10:28:01.202180+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:01.202180+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37479849,"num_objects":9153,"num_object_clones":0,"num_object_copies":18306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11761,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4832},{"shards":"0,2","objects":3034}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11361","reported_seq":11482,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.194258+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:01.194258+0000","last_peered":"2024-09-16T10:28:01.194258+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:01.194258+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35959750,"num_objects":8781,"num_object_clones":0,"num_object_copies":17562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4634},{"shards":"0,2","objects":2878}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11777","reported_seq":11924,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.244626+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:01.244626+0000","last_peered":"2024-09-16T10:28:01.244626+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:01.244626+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37353356,"num_objects":9123,"num_object_clones":0,"num_object_copies":18246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11774,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4755},{"shards":"0,2","objects":3072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":589799950,"num_objects":144011,"num_object_clones":0,"num_object_copies":288022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42735,"num_objects_misplaced":1890,"num_objects_unfound":0,"num_objects_dirty":144011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162294,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6721,"num_bytes_recovered":26624483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1258684416,"data_stored":1258593173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160829,"ondisk_log_size":160829,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018946,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717944,"kb_used_data":425452,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93653896,"statfs":{"total":96636764160,"available":95901589504,"internally_reserved":0,"allocated":435662848,"data_stored":435405142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":15.157999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":6.976}]}]},{"osd":1,"up_from":40,"seq":171798691890,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561324,"kb_used_data":304932,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93810516,"statfs":{"total":96636764160,"available":96061968384,"internally_reserved":0,"allocated":312250368,"data_stored":311920402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":262532778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.34000000000000002},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.31900000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215174,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":857096,"kb_used_data":500660,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93514744,"statfs":{"total":96636764160,"available":95759097856,"internally_reserved":0,"allocated":512675840,"data_stored":512425820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":364965544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,0,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.433},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.45700000000000002},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.39300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":511864832,"data_stored":511861219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311820288,"data_stored":311740343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":434999296,"data_stored":434991611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:04.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:04.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'11506", 'reported_seq': 16180, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:01.270428+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:01.270428+0000', 'last_peered': '2024-09-16T10:28:01.270428+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:01.270428+0000', 'last_undegraded': '2024-09-16T10:28:01.270428+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'1500", 'ondisk_log_start': "35'1500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36708352, 'num_objects': 8962, 'num_object_clones': 0, 'num_object_copies': 17924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1890, 'num_objects_unfound': 0, 'num_objects_dirty': 8962, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11506, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2302, 'num_bytes_recovered': 9428992, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7072}, {'shards': '0,2', 'objects': 1890}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:04.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:04.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'11626", 'reported_seq': 11829, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:01.259460+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:01.259460+0000', 'last_peered': '2024-09-16T10:28:01.259460+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:01.259460+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'1600", 'ondisk_log_start': "35'1600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37015576, 'num_objects': 9038, 'num_object_clones': 0, 'num_object_copies': 18076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4328, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9038, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7683, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3043}, {'shards': '2', 'objects': 4711}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:04.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:04.892 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'11410", 'reported_seq': 11612, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:01.190016+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:01.190016+0000', 'last_peered': '2024-09-16T10:28:01.190016+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:01.190016+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'1400", 'ondisk_log_start': "35'1400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36704256, 'num_objects': 8961, 'num_object_clones': 0, 'num_object_copies': 17922, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8961, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8035, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4763}, {'shards': '0,2', 'objects': 2995}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'11792", 'reported_seq': 12039, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:01.244530+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:01.244530+0000', 'last_peered': '2024-09-16T10:28:01.244530+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:01.244530+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'1700", 'ondisk_log_start': "35'1700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37388288, 'num_objects': 9128, 'num_object_clones': 0, 'num_object_copies': 18256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8034, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4762}, {'shards': '0,2', 'objects': 3074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'11884", 'reported_seq': 12021, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:01.235837+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:01.235837+0000', 'last_peered': '2024-09-16T10:28:01.235837+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:01.235837+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'1800", 'ondisk_log_start': "35'1800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37503942, 'num_objects': 9158, 'num_object_clones': 0, 'num_object_copies': 18316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4407, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11882, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4753}, {'shards': '0,2', 'objects': 3074}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'11439", 'reported_seq': 11561, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:01.107007+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:01.107007+0000', 'last_peered': '2024-09-16T10:28:01.107007+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:01.107007+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'1400", 'ondisk_log_start': "35'1400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36783046, 'num_objects': 8982, 'num_object_clones': 0, 'num_object_copies': 17964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4271, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11437, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 3050}, {'shards': '2', 'objects': 4713}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'11569", 'reported_seq': 11673, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:01.201756+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:01.201756+0000', 'last_peered': '2024-09-16T10:28:01.201756+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:01.201756+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'1500", 'ondisk_log_start': "35'1500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36893638, 'num_objects': 9009, 'num_object_clones': 0, 'num_object_copies': 18018, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4185, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11567, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2914}, {'shards': '2', 'objects': 4826}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'11240", 'reported_seq': 11375, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:01.169870+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:01.169870+0000', 'last_peered': '2024-09-16T10:28:01.169870+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:01.169870+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'1200", 'ondisk_log_start': "35'1200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36020707, 'num_objects': 8795, 'num_object_clones': 0, 'num_object_copies': 17590, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8795, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11239, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4664}, {'shards': '0,2', 'objects': 2937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:04.893 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:04.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'11764", 'reported_seq': 11929, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:01.202180+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:01.202180+0000', 'last_peered': '2024-09-16T10:28:01.202180+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:01.202180+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'1700", 'ondisk_log_start': "35'1700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37479849, 'num_objects': 9153, 'num_object_clones': 0, 'num_object_copies': 18306, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4324, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11761, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4832}, {'shards': '0,2', 'objects': 3034}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:04.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:04.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'11361", 'reported_seq': 11482, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:01.194258+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:01.194258+0000', 'last_peered': '2024-09-16T10:28:01.194258+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:01.194258+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'1300", 'ondisk_log_start': "35'1300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35959750, 'num_objects': 8781, 'num_object_clones': 0, 'num_object_copies': 17562, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4149, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11359, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4634}, {'shards': '0,2', 'objects': 2878}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:04.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:04.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'11777", 'reported_seq': 11924, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:01.244626+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:01.244626+0000', 'last_peered': '2024-09-16T10:28:01.244626+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:01.244626+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'1700", 'ondisk_log_start': "35'1700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37353356, 'num_objects': 9123, 'num_object_clones': 0, 'num_object_copies': 18246, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4371, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11774, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4755}, {'shards': '0,2', 'objects': 3072}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:04.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:05.592 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:05.593 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42735/288026 objects degraded (14.837%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":374,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":144013,"data_bytes":589799969,"bytes_used":2187636736,"bytes_avail":287722655744,"bytes_total":289910292480,"degraded_objects":42735,"degraded_total":288026,"degraded_ratio":0.1483720219702388,"misplaced_objects":1890,"misplaced_total":288026,"misplaced_ratio":0.0065619076055633864,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":83945,"recovering_keys_per_sec":0,"num_objects_recovered":278,"num_bytes_recovered":1007616,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":354},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:05.613 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:05.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:06.180 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:06.180 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:06.203 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-09-16T10:28:04.661132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":589799969,"num_objects":144013,"num_object_clones":0,"num_object_copies":288026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42735,"num_objects_misplaced":1890,"num_objects_unfound":0,"num_objects_dirty":144013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162296,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6722,"num_bytes_recovered":26624502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160832,"ondisk_log_size":160832,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2136364,"kb_used_data":1231044,"kb_used_omap":12,"kb_used_meta":905267,"kb_avail":280979156,"statfs":{"total":289910292480,"available":287722655744,"internally_reserved":0,"allocated":1260589056,"data_stored":1259751364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":926993410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,2,7,3,1,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17432576,"num_objects":-4256,"num_object_clones":0,"num_object_copies":-8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-32,"num_objects_misplaced":-246,"num_objects_unfound":0,"num_objects_dirty":-4256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4256,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003164"},"pg_stats":[{"pgid":"2.f","version":"44'11506","reported_seq":16180,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:01.270428+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:01.270428+0000","last_peered":"2024-09-16T10:28:01.270428+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:01.270428+0000","last_undegraded":"2024-09-16T10:28:01.270428+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36708352,"num_objects":8962,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1890,"num_objects_unfound":0,"num_objects_dirty":8962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2302,"num_bytes_recovered":9428992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7072},{"shards":"0,2","objects":1890}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11479","reported_seq":11635,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.222852+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:01.222852+0000","last_peered":"2024-09-16T10:28:01.222852+0000","last_clean":"2024-09-16T10:28:01.222852+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:01.222852+0000","last_undegraded":"2024-09-16T10:28:01.222852+0000","last_fullsized":"2024-09-16T10:28:01.222852+0000","mapping_epoch":34,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36794368,"num_objects":8983,"num_object_clones":0,"num_object_copies":17966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7692,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11626","reported_seq":11829,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.259460+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:01.259460+0000","last_peered":"2024-09-16T10:28:01.259460+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:01.259460+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37015576,"num_objects":9038,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9038,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7683,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3043},{"shards":"2","objects":4711}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11507","reported_seq":11564,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.221545+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:01.221545+0000","last_peered":"2024-09-16T10:28:01.221545+0000","last_clean":"2024-09-16T10:28:01.221545+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:01.221545+0000","last_undegraded":"2024-09-16T10:28:01.221545+0000","last_fullsized":"2024-09-16T10:28:01.221545+0000","mapping_epoch":34,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36679680,"num_objects":8955,"num_object_clones":0,"num_object_copies":17910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11507,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11517","reported_seq":11674,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.275752+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:01.275752+0000","last_peered":"2024-09-16T10:28:01.275752+0000","last_clean":"2024-09-16T10:28:01.275752+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:01.275752+0000","last_undegraded":"2024-09-16T10:28:01.275752+0000","last_fullsized":"2024-09-16T10:28:01.275752+0000","mapping_epoch":33,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36814848,"num_objects":8988,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8159,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11410","reported_seq":11612,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.190016+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:01.190016+0000","last_peered":"2024-09-16T10:28:01.190016+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:01.190016+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36704256,"num_objects":8961,"num_object_clones":0,"num_object_copies":17922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8035,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4763},{"shards":"0,2","objects":2995}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11792","reported_seq":12039,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.244530+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:01.244530+0000","last_peered":"2024-09-16T10:28:01.244530+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:01.244530+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37388288,"num_objects":9128,"num_object_clones":0,"num_object_copies":18256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8034,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4762},{"shards":"0,2","objects":3074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11396","reported_seq":11571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:01.221090+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:01.221090+0000","last_peered":"2024-09-16T10:28:01.221090+0000","last_clean":"2024-09-16T10:28:01.221090+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:01.221090+0000","last_undegraded":"2024-09-16T10:28:01.221090+0000","last_fullsized":"2024-09-16T10:28:01.221090+0000","mapping_epoch":33,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36839424,"num_objects":8994,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7199,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11462","reported_seq":19965,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:27:59.530472+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:27:59.530472+0000","last_peered":"2024-09-16T10:27:59.530472+0000","last_clean":"2024-09-16T10:27:59.530472+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:27:59.530472+0000","last_undegraded":"2024-09-16T10:27:59.530472+0000","last_fullsized":"2024-09-16T10:27:59.530472+0000","mapping_epoch":43,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36860870,"num_objects":9001,"num_object_clones":0,"num_object_copies":18002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11460,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'11884","reported_seq":12021,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.235837+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:01.235837+0000","last_peered":"2024-09-16T10:28:01.235837+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:01.235837+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37503942,"num_objects":9158,"num_object_clones":0,"num_object_copies":18316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11882,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4753},{"shards":"0,2","objects":3074}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11439","reported_seq":11561,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.107007+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:01.107007+0000","last_peered":"2024-09-16T10:28:01.107007+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:01.107007+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36783046,"num_objects":8982,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11437,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":3050},{"shards":"2","objects":4713}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":68,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.953289+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:00.953289+0000","last_peered":"2024-09-16T10:26:00.953289+0000","last_clean":"2024-09-16T10:26:00.953289+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:00.953289+0000","last_undegraded":"2024-09-16T10:26:00.953289+0000","last_fullsized":"2024-09-16T10:26:00.953289+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11569","reported_seq":11673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.201756+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:01.201756+0000","last_peered":"2024-09-16T10:28:01.201756+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:01.201756+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36893638,"num_objects":9009,"num_object_clones":0,"num_object_copies":18018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11567,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2914},{"shards":"2","objects":4826}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11240","reported_seq":11375,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.169870+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:01.169870+0000","last_peered":"2024-09-16T10:28:01.169870+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:01.169870+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1200","ondisk_log_start":"35'1200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36020707,"num_objects":8795,"num_object_clones":0,"num_object_copies":17590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11239,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4664},{"shards":"0,2","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.948914+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:00.948914+0000","last_peered":"2024-09-16T10:26:00.948914+0000","last_clean":"2024-09-16T10:26:00.948914+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:00.948914+0000","last_undegraded":"2024-09-16T10:26:00.948914+0000","last_fullsized":"2024-09-16T10:26:00.948914+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11764","reported_seq":11929,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.202180+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:01.202180+0000","last_peered":"2024-09-16T10:28:01.202180+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:01.202180+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37479849,"num_objects":9153,"num_object_clones":0,"num_object_copies":18306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11761,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4832},{"shards":"0,2","objects":3034}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11361","reported_seq":11482,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.194258+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:01.194258+0000","last_peered":"2024-09-16T10:28:01.194258+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:01.194258+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35959750,"num_objects":8781,"num_object_clones":0,"num_object_copies":17562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4634},{"shards":"0,2","objects":2878}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-16T10:26:00.949019+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:00.949019+0000","last_peered":"2024-09-16T10:26:00.949019+0000","last_clean":"2024-09-16T10:26:00.949019+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:00.949019+0000","last_undegraded":"2024-09-16T10:26:00.949019+0000","last_fullsized":"2024-09-16T10:26:00.949019+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11777","reported_seq":11924,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:01.244626+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:01.244626+0000","last_peered":"2024-09-16T10:28:01.244626+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:01.244626+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37353356,"num_objects":9123,"num_object_clones":0,"num_object_copies":18246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11774,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4755},{"shards":"0,2","objects":3072}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":589799950,"num_objects":144011,"num_object_clones":0,"num_object_copies":288022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42735,"num_objects_misplaced":1890,"num_objects_unfound":0,"num_objects_dirty":144011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162294,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6721,"num_bytes_recovered":26624483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1258684416,"data_stored":1258593173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160829,"ondisk_log_size":160829,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018946,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717944,"kb_used_data":425452,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93653896,"statfs":{"total":96636764160,"available":95901589504,"internally_reserved":0,"allocated":435662848,"data_stored":435405142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":15.157999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":6.976}]}]},{"osd":1,"up_from":40,"seq":171798691890,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561324,"kb_used_data":304932,"kb_used_omap":4,"kb_used_meta":256379,"kb_avail":93810516,"statfs":{"total":96636764160,"available":96061968384,"internally_reserved":0,"allocated":312250368,"data_stored":311920402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":262532778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.34000000000000002},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.31900000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215174,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":857096,"kb_used_data":500660,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93514744,"statfs":{"total":96636764160,"available":95759097856,"internally_reserved":0,"allocated":512675840,"data_stored":512425820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":364965544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,0,0,0,0,9],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:26:58 2024","interfaces":[{"interface":"back","average":{"1min":2.3780000000000001,"5min":2.6829999999999998,"15min":2.7669999999999999},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":6.0609999999999999,"5min":10.272,"15min":10.272},"last":0.433},{"interface":"front","average":{"1min":2.1760000000000002,"5min":2.016,"15min":2.02},"min":{"1min":0.31,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.3479999999999999,"5min":5.96,"15min":5.96},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:27:00 2024","interfaces":[{"interface":"back","average":{"1min":3.4300000000000002,"5min":1.9390000000000001,"15min":1.048},"min":{"1min":0.499,"5min":0.182,"15min":0.182},"max":{"1min":10.919,"5min":18.093,"15min":18.093},"last":0.45700000000000002},{"interface":"front","average":{"1min":2.7519999999999998,"5min":2.0640000000000001,"15min":1.4630000000000001},"min":{"1min":0.214,"5min":0.192,"15min":0.192},"max":{"1min":9.3290000000000006,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.39300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":511864832,"data_stored":511861219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311820288,"data_stored":311740343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":434999296,"data_stored":434991611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:09.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:09.769 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:09.769 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:09.793 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-16T10:28:08.662897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":581075489,"num_objects":141883,"num_object_clones":0,"num_object_copies":283766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42715,"num_objects_misplaced":1763,"num_objects_unfound":0,"num_objects_dirty":141883,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164426,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":27144694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2139272,"kb_used_data":1220016,"kb_used_omap":12,"kb_used_meta":919219,"kb_avail":280976248,"statfs":{"total":289910292480,"available":287719677952,"internally_reserved":0,"allocated":1249296384,"data_stored":1248458692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":941280258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,2,8,3,0,0,8,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17276928,"num_objects":-4218,"num_object_clones":0,"num_object_copies":-8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-47,"num_objects_misplaced":-269,"num_objects_unfound":0,"num_objects_dirty":-4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":1101824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003910"},"pg_stats":[{"pgid":"2.f","version":"44'11637","reported_seq":16565,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:06.239478+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:06.239478+0000","last_peered":"2024-09-16T10:28:06.239478+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:06.239478+0000","last_undegraded":"2024-09-16T10:28:06.239478+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36171776,"num_objects":8831,"num_object_clones":0,"num_object_copies":17662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1763,"num_objects_unfound":0,"num_objects_dirty":8831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11637,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2429,"num_bytes_recovered":9949184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7068},{"shards":"0,2","objects":1763}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11621","reported_seq":11778,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.260196+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:06.260196+0000","last_peered":"2024-09-16T10:28:06.260196+0000","last_clean":"2024-09-16T10:28:06.260196+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:06.260196+0000","last_undegraded":"2024-09-16T10:28:06.260196+0000","last_fullsized":"2024-09-16T10:28:06.260196+0000","mapping_epoch":34,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36212736,"num_objects":8841,"num_object_clones":0,"num_object_copies":17682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7834,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11756","reported_seq":11961,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.216289+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:06.216289+0000","last_peered":"2024-09-16T10:28:06.216289+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:06.216289+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36483096,"num_objects":8908,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8908,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7813,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2913},{"shards":"2","objects":4582}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11635","reported_seq":11692,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.179548+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:06.179548+0000","last_peered":"2024-09-16T10:28:06.179548+0000","last_clean":"2024-09-16T10:28:06.179548+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:06.179548+0000","last_undegraded":"2024-09-16T10:28:06.179548+0000","last_fullsized":"2024-09-16T10:28:06.179548+0000","mapping_epoch":34,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36155392,"num_objects":8827,"num_object_clones":0,"num_object_copies":17654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11628","reported_seq":11785,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.227036+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:06.227036+0000","last_peered":"2024-09-16T10:28:06.227036+0000","last_clean":"2024-09-16T10:28:06.227036+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:06.227036+0000","last_undegraded":"2024-09-16T10:28:06.227036+0000","last_fullsized":"2024-09-16T10:28:06.227036+0000","mapping_epoch":33,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36360192,"num_objects":8877,"num_object_clones":0,"num_object_copies":17754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8270,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11542","reported_seq":11752,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.256368+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:06.256368+0000","last_peered":"2024-09-16T10:28:06.256368+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:06.256368+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36163584,"num_objects":8829,"num_object_clones":0,"num_object_copies":17658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8167,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4635},{"shards":"0,2","objects":2863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11935","reported_seq":12189,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.281304+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:06.281304+0000","last_peered":"2024-09-16T10:28:06.281304+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:06.281304+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36802560,"num_objects":8985,"num_object_clones":0,"num_object_copies":17970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8177,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4622},{"shards":"0,2","objects":2931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11542","reported_seq":11717,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.299581+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:06.299581+0000","last_peered":"2024-09-16T10:28:06.299581+0000","last_clean":"2024-09-16T10:28:06.299581+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:06.299581+0000","last_undegraded":"2024-09-16T10:28:06.299581+0000","last_fullsized":"2024-09-16T10:28:06.299581+0000","mapping_epoch":33,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36241408,"num_objects":8848,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7345,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11592","reported_seq":20094,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:04.573823+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:04.573823+0000","last_peered":"2024-09-16T10:28:04.573823+0000","last_clean":"2024-09-16T10:28:04.573823+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:04.573823+0000","last_undegraded":"2024-09-16T10:28:04.573823+0000","last_fullsized":"2024-09-16T10:28:04.573823+0000","mapping_epoch":43,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36328390,"num_objects":8871,"num_object_clones":0,"num_object_copies":17742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12007","reported_seq":12144,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.246528+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:06.246528+0000","last_peered":"2024-09-16T10:28:06.246528+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:06.246528+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37000134,"num_objects":9035,"num_object_clones":0,"num_object_copies":18070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4630},{"shards":"0,2","objects":2951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11590","reported_seq":11712,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.208051+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:06.208051+0000","last_peered":"2024-09-16T10:28:06.208051+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:06.208051+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36164550,"num_objects":8831,"num_object_clones":0,"num_object_copies":17662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11588,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2899},{"shards":"2","objects":4562}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11695","reported_seq":11800,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.025459+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:06.025459+0000","last_peered":"2024-09-16T10:28:06.025459+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:06.025459+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377542,"num_objects":8883,"num_object_clones":0,"num_object_copies":17766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11693,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2788},{"shards":"2","objects":4700}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11366","reported_seq":11504,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.225021+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:06.225021+0000","last_peered":"2024-09-16T10:28:06.225021+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:06.225021+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35504611,"num_objects":8669,"num_object_clones":0,"num_object_copies":17338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11365,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4541},{"shards":"0,2","objects":2811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11893","reported_seq":12059,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.282370+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:06.282370+0000","last_peered":"2024-09-16T10:28:06.282370+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:06.282370+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36951465,"num_objects":9024,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11890,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4704},{"shards":"0,2","objects":2905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11502","reported_seq":11635,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.284307+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:06.284307+0000","last_peered":"2024-09-16T10:28:06.284307+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:06.284307+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35382214,"num_objects":8640,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11500,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4499},{"shards":"0,2","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11918","reported_seq":12069,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.273858+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:06.273858+0000","last_peered":"2024-09-16T10:28:06.273858+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:06.273858+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36775820,"num_objects":8982,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11915,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4616},{"shards":"0,2","objects":2931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":581075470,"num_objects":141881,"num_object_clones":0,"num_object_copies":283762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42715,"num_objects_misplaced":1763,"num_objects_unfound":0,"num_objects_dirty":141881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164424,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6868,"num_bytes_recovered":27144675,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1247342592,"data_stored":1247251349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160759,"ondisk_log_size":160759,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018947,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717760,"kb_used_data":421176,"kb_used_omap":4,"kb_used_meta":296571,"kb_avail":93654080,"statfs":{"total":96636764160,"available":95901777920,"internally_reserved":0,"allocated":431284224,"data_stored":431026518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":303689392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.1339999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.4820000000000002},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":2.722}]}]},{"osd":1,"up_from":40,"seq":171798691891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566644,"kb_used_data":304492,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93805196,"statfs":{"total":96636764160,"available":96056520704,"internally_reserved":0,"allocated":311799808,"data_stored":311469842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":268431018},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.41199999999999998},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.0600000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215175,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":854868,"kb_used_data":494348,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93516972,"statfs":{"total":96636764160,"available":95761379328,"internally_reserved":0,"allocated":506212352,"data_stored":505962332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,1,3,1,0,0,8],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.5820000000000001},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.2090000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.7389999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.6739999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":505331712,"data_stored":505328099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311320576,"data_stored":311240631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":430690304,"data_stored":430682619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:09.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:10.343 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:10.365 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-16T10:28:08.662897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":581075489,"num_objects":141883,"num_object_clones":0,"num_object_copies":283766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42715,"num_objects_misplaced":1763,"num_objects_unfound":0,"num_objects_dirty":141883,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164426,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":27144694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2139272,"kb_used_data":1220016,"kb_used_omap":12,"kb_used_meta":919219,"kb_avail":280976248,"statfs":{"total":289910292480,"available":287719677952,"internally_reserved":0,"allocated":1249296384,"data_stored":1248458692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":941280258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,2,8,3,0,0,8,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17276928,"num_objects":-4218,"num_object_clones":0,"num_object_copies":-8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-47,"num_objects_misplaced":-269,"num_objects_unfound":0,"num_objects_dirty":-4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":1101824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003910"},"pg_stats":[{"pgid":"2.f","version":"44'11637","reported_seq":16565,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:06.239478+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:06.239478+0000","last_peered":"2024-09-16T10:28:06.239478+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:06.239478+0000","last_undegraded":"2024-09-16T10:28:06.239478+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36171776,"num_objects":8831,"num_object_clones":0,"num_object_copies":17662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1763,"num_objects_unfound":0,"num_objects_dirty":8831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11637,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2429,"num_bytes_recovered":9949184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7068},{"shards":"0,2","objects":1763}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11621","reported_seq":11778,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.260196+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:06.260196+0000","last_peered":"2024-09-16T10:28:06.260196+0000","last_clean":"2024-09-16T10:28:06.260196+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:06.260196+0000","last_undegraded":"2024-09-16T10:28:06.260196+0000","last_fullsized":"2024-09-16T10:28:06.260196+0000","mapping_epoch":34,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36212736,"num_objects":8841,"num_object_clones":0,"num_object_copies":17682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7834,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11756","reported_seq":11961,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.216289+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:06.216289+0000","last_peered":"2024-09-16T10:28:06.216289+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:06.216289+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36483096,"num_objects":8908,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8908,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7813,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2913},{"shards":"2","objects":4582}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11635","reported_seq":11692,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.179548+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:06.179548+0000","last_peered":"2024-09-16T10:28:06.179548+0000","last_clean":"2024-09-16T10:28:06.179548+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:06.179548+0000","last_undegraded":"2024-09-16T10:28:06.179548+0000","last_fullsized":"2024-09-16T10:28:06.179548+0000","mapping_epoch":34,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36155392,"num_objects":8827,"num_object_clones":0,"num_object_copies":17654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11628","reported_seq":11785,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.227036+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:06.227036+0000","last_peered":"2024-09-16T10:28:06.227036+0000","last_clean":"2024-09-16T10:28:06.227036+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:06.227036+0000","last_undegraded":"2024-09-16T10:28:06.227036+0000","last_fullsized":"2024-09-16T10:28:06.227036+0000","mapping_epoch":33,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36360192,"num_objects":8877,"num_object_clones":0,"num_object_copies":17754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8270,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11542","reported_seq":11752,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.256368+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:06.256368+0000","last_peered":"2024-09-16T10:28:06.256368+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:06.256368+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36163584,"num_objects":8829,"num_object_clones":0,"num_object_copies":17658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8167,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4635},{"shards":"0,2","objects":2863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11935","reported_seq":12189,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.281304+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:06.281304+0000","last_peered":"2024-09-16T10:28:06.281304+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:06.281304+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36802560,"num_objects":8985,"num_object_clones":0,"num_object_copies":17970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8177,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4622},{"shards":"0,2","objects":2931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11542","reported_seq":11717,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.299581+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:06.299581+0000","last_peered":"2024-09-16T10:28:06.299581+0000","last_clean":"2024-09-16T10:28:06.299581+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:06.299581+0000","last_undegraded":"2024-09-16T10:28:06.299581+0000","last_fullsized":"2024-09-16T10:28:06.299581+0000","mapping_epoch":33,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36241408,"num_objects":8848,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7345,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11592","reported_seq":20094,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:04.573823+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:04.573823+0000","last_peered":"2024-09-16T10:28:04.573823+0000","last_clean":"2024-09-16T10:28:04.573823+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:04.573823+0000","last_undegraded":"2024-09-16T10:28:04.573823+0000","last_fullsized":"2024-09-16T10:28:04.573823+0000","mapping_epoch":43,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36328390,"num_objects":8871,"num_object_clones":0,"num_object_copies":17742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11590,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12007","reported_seq":12144,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.246528+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:06.246528+0000","last_peered":"2024-09-16T10:28:06.246528+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:06.246528+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37000134,"num_objects":9035,"num_object_clones":0,"num_object_copies":18070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4630},{"shards":"0,2","objects":2951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11590","reported_seq":11712,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.208051+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:06.208051+0000","last_peered":"2024-09-16T10:28:06.208051+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:06.208051+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36164550,"num_objects":8831,"num_object_clones":0,"num_object_copies":17662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11588,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2899},{"shards":"2","objects":4562}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11695","reported_seq":11800,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.025459+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:06.025459+0000","last_peered":"2024-09-16T10:28:06.025459+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:06.025459+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377542,"num_objects":8883,"num_object_clones":0,"num_object_copies":17766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11693,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2788},{"shards":"2","objects":4700}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11366","reported_seq":11504,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.225021+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:06.225021+0000","last_peered":"2024-09-16T10:28:06.225021+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:06.225021+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35504611,"num_objects":8669,"num_object_clones":0,"num_object_copies":17338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11365,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4541},{"shards":"0,2","objects":2811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11893","reported_seq":12059,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.282370+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:06.282370+0000","last_peered":"2024-09-16T10:28:06.282370+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:06.282370+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36951465,"num_objects":9024,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11890,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4704},{"shards":"0,2","objects":2905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11502","reported_seq":11635,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.284307+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:06.284307+0000","last_peered":"2024-09-16T10:28:06.284307+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:06.284307+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35382214,"num_objects":8640,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11500,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4499},{"shards":"0,2","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11918","reported_seq":12069,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.273858+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:06.273858+0000","last_peered":"2024-09-16T10:28:06.273858+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:06.273858+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36775820,"num_objects":8982,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11915,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4616},{"shards":"0,2","objects":2931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":581075470,"num_objects":141881,"num_object_clones":0,"num_object_copies":283762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42715,"num_objects_misplaced":1763,"num_objects_unfound":0,"num_objects_dirty":141881,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164424,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6868,"num_bytes_recovered":27144675,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1247342592,"data_stored":1247251349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160759,"ondisk_log_size":160759,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018947,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717760,"kb_used_data":421176,"kb_used_omap":4,"kb_used_meta":296571,"kb_avail":93654080,"statfs":{"total":96636764160,"available":95901777920,"internally_reserved":0,"allocated":431284224,"data_stored":431026518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":303689392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.1339999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.4820000000000002},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":2.722}]}]},{"osd":1,"up_from":40,"seq":171798691891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566644,"kb_used_data":304492,"kb_used_omap":4,"kb_used_meta":262139,"kb_avail":93805196,"statfs":{"total":96636764160,"available":96056520704,"internally_reserved":0,"allocated":311799808,"data_stored":311469842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":268431018},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":5.0620000000000003,"5min":3.9359999999999999,"15min":3.7120000000000002},"min":{"1min":0.20999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":17.675000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":3.7669999999999999,"5min":3.0169999999999999,"15min":2.609},"min":{"1min":0.40200000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":15.837999999999999,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:27:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.4710000000000001,"15min":2.3690000000000002},"min":{"1min":0.35799999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.5500000000000007,"5min":8.5500000000000007,"15min":8.5500000000000007},"last":0.41199999999999998},{"interface":"front","average":{"1min":3.2250000000000001,"5min":2.8700000000000001,"15min":2.8849999999999998},"min":{"1min":0.313,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.0600000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215175,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":854868,"kb_used_data":494348,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93516972,"statfs":{"total":96636764160,"available":95761379328,"internally_reserved":0,"allocated":506212352,"data_stored":505962332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,1,3,1,0,0,8],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.5820000000000001},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.2090000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.7389999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.6739999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":505331712,"data_stored":505328099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":311320576,"data_stored":311240631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":430690304,"data_stored":430682619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:10.367 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:10.367 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'11637", 'reported_seq': 16565, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:06.239478+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:06.239478+0000', 'last_peered': '2024-09-16T10:28:06.239478+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:06.239478+0000', 'last_undegraded': '2024-09-16T10:28:06.239478+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'1600", 'ondisk_log_start': "35'1600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36171776, 'num_objects': 8831, 'num_object_clones': 0, 'num_object_copies': 17662, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1763, 'num_objects_unfound': 0, 'num_objects_dirty': 8831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11637, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2429, 'num_bytes_recovered': 9949184, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7068}, {'shards': '0,2', 'objects': 1763}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:10.367 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:10.367 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'11756", 'reported_seq': 11961, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:06.216289+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:06.216289+0000', 'last_peered': '2024-09-16T10:28:06.216289+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:06.216289+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'1700", 'ondisk_log_start': "35'1700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36483096, 'num_objects': 8908, 'num_object_clones': 0, 'num_object_copies': 17816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8908, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7813, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2913}, {'shards': '2', 'objects': 4582}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:10.367 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:10.367 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'11542", 'reported_seq': 11752, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:06.256368+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:06.256368+0000', 'last_peered': '2024-09-16T10:28:06.256368+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:06.256368+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'1500", 'ondisk_log_start': "35'1500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36163584, 'num_objects': 8829, 'num_object_clones': 0, 'num_object_copies': 17658, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4196, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8167, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4635}, {'shards': '0,2', 'objects': 2863}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:10.368 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:10.368 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'11935", 'reported_seq': 12189, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:06.281304+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:06.281304+0000', 'last_peered': '2024-09-16T10:28:06.281304+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:06.281304+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'1900", 'ondisk_log_start': "35'1900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36802560, 'num_objects': 8985, 'num_object_clones': 0, 'num_object_copies': 17970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4365, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8177, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4622}, {'shards': '0,2', 'objects': 2931}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:10.368 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:10.368 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'12007", 'reported_seq': 12144, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:06.246528+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:06.246528+0000', 'last_peered': '2024-09-16T10:28:06.246528+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:06.246528+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'2000", 'ondisk_log_start': "35'2000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37000134, 'num_objects': 9035, 'num_object_clones': 0, 'num_object_copies': 18070, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4407, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12005, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4630}, {'shards': '0,2', 'objects': 2951}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:10.368 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:10.368 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'11590", 'reported_seq': 11712, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:06.208051+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:06.208051+0000', 'last_peered': '2024-09-16T10:28:06.208051+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:06.208051+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'1500", 'ondisk_log_start': "35'1500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36164550, 'num_objects': 8831, 'num_object_clones': 0, 'num_object_copies': 17662, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4271, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11588, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2899}, {'shards': '2', 'objects': 4562}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:10.368 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:10.368 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'11695", 'reported_seq': 11800, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:06.025459+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:06.025459+0000', 'last_peered': '2024-09-16T10:28:06.025459+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:06.025459+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'1600", 'ondisk_log_start': "35'1600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36377542, 'num_objects': 8883, 'num_object_clones': 0, 'num_object_copies': 17766, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4185, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11693, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2788}, {'shards': '2', 'objects': 4700}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:10.368 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:10.368 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'11366", 'reported_seq': 11504, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:06.225021+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:06.225021+0000', 'last_peered': '2024-09-16T10:28:06.225021+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:06.225021+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'1300", 'ondisk_log_start': "35'1300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35504611, 'num_objects': 8669, 'num_object_clones': 0, 'num_object_copies': 17338, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11365, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4541}, {'shards': '0,2', 'objects': 2811}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:10.369 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:10.369 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'11893", 'reported_seq': 12059, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:06.282370+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:06.282370+0000', 'last_peered': '2024-09-16T10:28:06.282370+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:06.282370+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'1800", 'ondisk_log_start': "35'1800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36951465, 'num_objects': 9024, 'num_object_clones': 0, 'num_object_copies': 18048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4323, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 9024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11890, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4704}, {'shards': '0,2', 'objects': 2905}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:10.369 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:10.369 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'11502", 'reported_seq': 11635, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:06.284307+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:06.284307+0000', 'last_peered': '2024-09-16T10:28:06.284307+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:06.284307+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'1500", 'ondisk_log_start': "35'1500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35382214, 'num_objects': 8640, 'num_object_clones': 0, 'num_object_copies': 17280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4143, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11500, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4499}, {'shards': '0,2', 'objects': 2737}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:10.369 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:10.369 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'11918", 'reported_seq': 12069, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:06.273858+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:06.273858+0000', 'last_peered': '2024-09-16T10:28:06.273858+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:06.273858+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'1900", 'ondisk_log_start': "35'1900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36775820, 'num_objects': 8982, 'num_object_clones': 0, 'num_object_copies': 17964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4369, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11915, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4616}, {'shards': '0,2', 'objects': 2931}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:10.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:11.085 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:11.085 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42715/283766 objects degraded (15.053%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":379,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":141883,"data_bytes":581075489,"bytes_used":2190614528,"bytes_avail":287719677952,"bytes_total":289910292480,"degraded_objects":42715,"degraded_total":283766,"degraded_ratio":0.15052895695749316,"misplaced_objects":1763,"misplaced_total":283766,"misplaced_ratio":0.0062128655300494077,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":91788,"recovering_keys_per_sec":0,"num_objects_recovered":316,"num_bytes_recovered":1101824,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":351},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:11.106 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:11.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:11.632 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:11.653 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-16T10:28:10.663644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":580563489,"num_objects":141758,"num_object_clones":0,"num_object_copies":283516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42715,"num_objects_misplaced":1763,"num_objects_unfound":0,"num_objects_dirty":141758,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164551,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6869,"num_bytes_recovered":27144694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160687,"ondisk_log_size":160687,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2146748,"kb_used_data":1219300,"kb_used_omap":12,"kb_used_meta":927411,"kb_avail":280968772,"statfs":{"total":289910292480,"available":287712022528,"internally_reserved":0,"allocated":1248563200,"data_stored":1247725508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":949668866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,7,6,7,2,0,0,8,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17788928,"num_objects":-4343,"num_object_clones":0,"num_object_copies":-8686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-47,"num_objects_misplaced":-269,"num_objects_unfound":0,"num_objects_dirty":-4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":1101824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004138"},"pg_stats":[{"pgid":"2.f","version":"44'11637","reported_seq":16565,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:06.239478+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:06.239478+0000","last_peered":"2024-09-16T10:28:06.239478+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:06.239478+0000","last_undegraded":"2024-09-16T10:28:06.239478+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36171776,"num_objects":8831,"num_object_clones":0,"num_object_copies":17662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1763,"num_objects_unfound":0,"num_objects_dirty":8831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11637,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2429,"num_bytes_recovered":9949184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7068},{"shards":"0,2","objects":1763}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11621","reported_seq":11778,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.260196+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:06.260196+0000","last_peered":"2024-09-16T10:28:06.260196+0000","last_clean":"2024-09-16T10:28:06.260196+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:06.260196+0000","last_undegraded":"2024-09-16T10:28:06.260196+0000","last_fullsized":"2024-09-16T10:28:06.260196+0000","mapping_epoch":34,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36212736,"num_objects":8841,"num_object_clones":0,"num_object_copies":17682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7834,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11756","reported_seq":11961,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.216289+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:06.216289+0000","last_peered":"2024-09-16T10:28:06.216289+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:06.216289+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36483096,"num_objects":8908,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8908,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7813,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2913},{"shards":"2","objects":4582}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11635","reported_seq":11692,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.179548+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:06.179548+0000","last_peered":"2024-09-16T10:28:06.179548+0000","last_clean":"2024-09-16T10:28:06.179548+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:06.179548+0000","last_undegraded":"2024-09-16T10:28:06.179548+0000","last_fullsized":"2024-09-16T10:28:06.179548+0000","mapping_epoch":34,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36155392,"num_objects":8827,"num_object_clones":0,"num_object_copies":17654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11635,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11628","reported_seq":11785,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.227036+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:06.227036+0000","last_peered":"2024-09-16T10:28:06.227036+0000","last_clean":"2024-09-16T10:28:06.227036+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:06.227036+0000","last_undegraded":"2024-09-16T10:28:06.227036+0000","last_fullsized":"2024-09-16T10:28:06.227036+0000","mapping_epoch":33,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36360192,"num_objects":8877,"num_object_clones":0,"num_object_copies":17754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8270,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11542","reported_seq":11752,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.256368+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:06.256368+0000","last_peered":"2024-09-16T10:28:06.256368+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:06.256368+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36163584,"num_objects":8829,"num_object_clones":0,"num_object_copies":17658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8167,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4635},{"shards":"0,2","objects":2863}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'11935","reported_seq":12189,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.281304+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:06.281304+0000","last_peered":"2024-09-16T10:28:06.281304+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:06.281304+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36802560,"num_objects":8985,"num_object_clones":0,"num_object_copies":17970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8177,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4622},{"shards":"0,2","objects":2931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11542","reported_seq":11717,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:06.299581+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:06.299581+0000","last_peered":"2024-09-16T10:28:06.299581+0000","last_clean":"2024-09-16T10:28:06.299581+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:06.299581+0000","last_undegraded":"2024-09-16T10:28:06.299581+0000","last_fullsized":"2024-09-16T10:28:06.299581+0000","mapping_epoch":33,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36241408,"num_objects":8848,"num_object_clones":0,"num_object_copies":17696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7345,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11717","reported_seq":20221,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:09.595974+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:09.595974+0000","last_peered":"2024-09-16T10:28:09.595974+0000","last_clean":"2024-09-16T10:28:09.595974+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:09.595974+0000","last_undegraded":"2024-09-16T10:28:09.595974+0000","last_fullsized":"2024-09-16T10:28:09.595974+0000","mapping_epoch":43,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35816390,"num_objects":8746,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12007","reported_seq":12144,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.246528+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:06.246528+0000","last_peered":"2024-09-16T10:28:06.246528+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:06.246528+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37000134,"num_objects":9035,"num_object_clones":0,"num_object_copies":18070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4630},{"shards":"0,2","objects":2951}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11590","reported_seq":11712,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.208051+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:06.208051+0000","last_peered":"2024-09-16T10:28:06.208051+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:06.208051+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36164550,"num_objects":8831,"num_object_clones":0,"num_object_copies":17662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4271,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11588,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2899},{"shards":"2","objects":4562}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11695","reported_seq":11800,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.025459+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:06.025459+0000","last_peered":"2024-09-16T10:28:06.025459+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:06.025459+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36377542,"num_objects":8883,"num_object_clones":0,"num_object_copies":17766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11693,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2788},{"shards":"2","objects":4700}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11366","reported_seq":11504,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.225021+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:06.225021+0000","last_peered":"2024-09-16T10:28:06.225021+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:06.225021+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1300","ondisk_log_start":"35'1300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35504611,"num_objects":8669,"num_object_clones":0,"num_object_copies":17338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11365,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4541},{"shards":"0,2","objects":2811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'11893","reported_seq":12059,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.282370+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:06.282370+0000","last_peered":"2024-09-16T10:28:06.282370+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:06.282370+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36951465,"num_objects":9024,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11890,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4704},{"shards":"0,2","objects":2905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11502","reported_seq":11635,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.284307+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:06.284307+0000","last_peered":"2024-09-16T10:28:06.284307+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:06.284307+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1500","ondisk_log_start":"35'1500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35382214,"num_objects":8640,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11500,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4499},{"shards":"0,2","objects":2737}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'11918","reported_seq":12069,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:06.273858+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:06.273858+0000","last_peered":"2024-09-16T10:28:06.273858+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:06.273858+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36775820,"num_objects":8982,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11915,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4616},{"shards":"0,2","objects":2931}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":580563470,"num_objects":141756,"num_object_clones":0,"num_object_copies":283512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42715,"num_objects_misplaced":1763,"num_objects_unfound":0,"num_objects_dirty":141756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164549,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6868,"num_bytes_recovered":27144675,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1246715904,"data_stored":1246624661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160684,"ondisk_log_size":160684,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018947,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717760,"kb_used_data":421176,"kb_used_omap":4,"kb_used_meta":296571,"kb_avail":93654080,"statfs":{"total":96636764160,"available":95901777920,"internally_reserved":0,"allocated":431284224,"data_stored":431026518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":303689392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.1339999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.4820000000000002},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":2.722}]}]},{"osd":1,"up_from":40,"seq":171798691892,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574120,"kb_used_data":303776,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93797720,"statfs":{"total":96636764160,"available":96048865280,"internally_reserved":0,"allocated":311066624,"data_stored":310736658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":276819626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.5640000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":8.2189999999999994}]}]},{"osd":0,"up_from":24,"seq":103079215175,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":854868,"kb_used_data":494348,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93516972,"statfs":{"total":96636764160,"available":95761379328,"internally_reserved":0,"allocated":506212352,"data_stored":505962332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,1,3,1,0,0,8],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.5820000000000001},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.2090000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.7389999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.6739999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":505331712,"data_stored":505328099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":310693888,"data_stored":310613943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":430690304,"data_stored":430682619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:14.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:15.195 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:15.195 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:15.217 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-16T10:28:14.664600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":572256801,"num_objects":139730,"num_object_clones":0,"num_object_copies":279460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42673,"num_objects_misplaced":1652,"num_objects_unfound":0,"num_objects_dirty":139730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166579,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7022,"num_bytes_recovered":27599350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160915,"ondisk_log_size":160915,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2131856,"kb_used_data":1208500,"kb_used_omap":12,"kb_used_meta":923315,"kb_avail":280983664,"statfs":{"total":289910292480,"available":287727271936,"internally_reserved":0,"allocated":1237504000,"data_stored":1236666308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":945474562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,6,8,10,7,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17543168,"num_objects":-4283,"num_object_clones":0,"num_object_copies":-8566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-62,"num_objects_misplaced":-238,"num_objects_unfound":0,"num_objects_dirty":-4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4283,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":300,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004137"},"pg_stats":[{"pgid":"2.f","version":"44'11777","reported_seq":16927,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:11.368934+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:11.368934+0000","last_peered":"2024-09-16T10:28:11.368934+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:11.368934+0000","last_undegraded":"2024-09-16T10:28:11.368934+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35598336,"num_objects":8691,"num_object_clones":0,"num_object_copies":17382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1652,"num_objects_unfound":0,"num_objects_dirty":8691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2540,"num_bytes_recovered":10403840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7039},{"shards":"0,2","objects":1652}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11752","reported_seq":11909,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:11.209277+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:11.209277+0000","last_peered":"2024-09-16T10:28:11.209277+0000","last_clean":"2024-09-16T10:28:11.209277+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:11.209277+0000","last_undegraded":"2024-09-16T10:28:11.209277+0000","last_fullsized":"2024-09-16T10:28:11.209277+0000","mapping_epoch":34,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35676160,"num_objects":8710,"num_object_clones":0,"num_object_copies":17420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7965,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11890","reported_seq":12097,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.314569+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:11.314569+0000","last_peered":"2024-09-16T10:28:11.314569+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:11.314569+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35934232,"num_objects":8774,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7947,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2779},{"shards":"2","objects":4449}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11776","reported_seq":11833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:11.188095+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:11.188095+0000","last_peered":"2024-09-16T10:28:11.188095+0000","last_clean":"2024-09-16T10:28:11.188095+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:11.188095+0000","last_undegraded":"2024-09-16T10:28:11.188095+0000","last_fullsized":"2024-09-16T10:28:11.188095+0000","mapping_epoch":34,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35577856,"num_objects":8686,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11776,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11749","reported_seq":11906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:11.219227+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:11.219227+0000","last_peered":"2024-09-16T10:28:11.219227+0000","last_clean":"2024-09-16T10:28:11.219227+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:11.219227+0000","last_undegraded":"2024-09-16T10:28:11.219227+0000","last_fullsized":"2024-09-16T10:28:11.219227+0000","mapping_epoch":33,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35864576,"num_objects":8756,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11676","reported_seq":11888,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.273804+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:11.273804+0000","last_peered":"2024-09-16T10:28:11.273804+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:11.273804+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35614720,"num_objects":8695,"num_object_clones":0,"num_object_copies":17390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8301,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4503},{"shards":"0,2","objects":2729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12100","reported_seq":12370,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.345741+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:11.345741+0000","last_peered":"2024-09-16T10:28:11.345741+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:11.345741+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36126720,"num_objects":8820,"num_object_clones":0,"num_object_copies":17640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4465},{"shards":"0,2","objects":2766}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11664","reported_seq":11839,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:11.322813+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:11.322813+0000","last_peered":"2024-09-16T10:28:11.322813+0000","last_clean":"2024-09-16T10:28:11.322813+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:11.322813+0000","last_undegraded":"2024-09-16T10:28:11.322813+0000","last_fullsized":"2024-09-16T10:28:11.322813+0000","mapping_epoch":33,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35741696,"num_objects":8726,"num_object_clones":0,"num_object_copies":17452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11717","reported_seq":20221,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:09.595974+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:09.595974+0000","last_peered":"2024-09-16T10:28:09.595974+0000","last_clean":"2024-09-16T10:28:09.595974+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:09.595974+0000","last_undegraded":"2024-09-16T10:28:09.595974+0000","last_fullsized":"2024-09-16T10:28:09.595974+0000","mapping_epoch":43,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35816390,"num_objects":8746,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12138","reported_seq":12279,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.306732+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:11.306732+0000","last_peered":"2024-09-16T10:28:11.306732+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:11.306732+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36463558,"num_objects":8904,"num_object_clones":0,"num_object_copies":17808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12136,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4501},{"shards":"0,2","objects":2820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11717","reported_seq":11851,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.229865+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:11.229865+0000","last_peered":"2024-09-16T10:28:11.229865+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:11.229865+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35644358,"num_objects":8704,"num_object_clones":0,"num_object_copies":17408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2772},{"shards":"2","objects":4441}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11825","reported_seq":11930,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.310549+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:11.310549+0000","last_peered":"2024-09-16T10:28:11.310549+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:11.310549+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35845062,"num_objects":8753,"num_object_clones":0,"num_object_copies":17506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11823,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2658},{"shards":"2","objects":4570}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11492","reported_seq":11646,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.387653+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:11.387653+0000","last_peered":"2024-09-16T10:28:11.387653+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:11.387653+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34988515,"num_objects":8543,"num_object_clones":0,"num_object_copies":17086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11491,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4422},{"shards":"0,2","objects":2685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12036","reported_seq":12211,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.372629+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:11.372629+0000","last_peered":"2024-09-16T10:28:11.372629+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:11.372629+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36365737,"num_objects":8881,"num_object_clones":0,"num_object_copies":17762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4565},{"shards":"0,2","objects":2762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11633","reported_seq":11782,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.302717+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:11.302717+0000","last_peered":"2024-09-16T10:28:11.302717+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:11.302717+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34845638,"num_objects":8509,"num_object_clones":0,"num_object_copies":17018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11631,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4376},{"shards":"0,2","objects":2606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12070","reported_seq":12226,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.269447+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:11.269447+0000","last_peered":"2024-09-16T10:28:11.269447+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:11.269447+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36153228,"num_objects":8830,"num_object_clones":0,"num_object_copies":17660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12067,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4468},{"shards":"0,2","objects":2779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":572256782,"num_objects":139728,"num_object_clones":0,"num_object_copies":279456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42673,"num_objects_misplaced":1652,"num_objects_unfound":0,"num_objects_dirty":139728,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166577,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7021,"num_bytes_recovered":27599331,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1235783680,"data_stored":1235692437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160912,"ondisk_log_size":160912,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018948,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701260,"kb_used_data":416960,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93670580,"statfs":{"total":96636764160,"available":95918673920,"internally_reserved":0,"allocated":426967040,"data_stored":426709334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.8959999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.48399999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.5249999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":7.1970000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691892,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574120,"kb_used_data":303776,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93797720,"statfs":{"total":96636764160,"available":96048865280,"internally_reserved":0,"allocated":311066624,"data_stored":310736658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":276819626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.5640000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":8.2189999999999994}]}]},{"osd":0,"up_from":24,"seq":103079215176,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":856476,"kb_used_data":487764,"kb_used_omap":4,"kb_used_meta":368699,"kb_avail":93515364,"statfs":{"total":96636764160,"available":95759732736,"internally_reserved":0,"allocated":499470336,"data_stored":499220316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":377548456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,5,5,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.83299999999999996},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.173}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":498618368,"data_stored":498614755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":310693888,"data_stored":310613943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":426471424,"data_stored":426463739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:15.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:15.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-09-16T10:28:15.596+0000 7fe5e77fe640 0 --2- 172.21.15.50:0/2226466700 >> [v2:172.21.15.50:3302/0,v1:172.21.15.50:6791/0] conn(0x7fe5e812e3b0 0x7fe5e81307a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T10:28:15.978 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:15.981 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:16.004 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-16T10:28:14.664600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":572256801,"num_objects":139730,"num_object_clones":0,"num_object_copies":279460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42673,"num_objects_misplaced":1652,"num_objects_unfound":0,"num_objects_dirty":139730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166579,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7022,"num_bytes_recovered":27599350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160915,"ondisk_log_size":160915,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2131856,"kb_used_data":1208500,"kb_used_omap":12,"kb_used_meta":923315,"kb_avail":280983664,"statfs":{"total":289910292480,"available":287727271936,"internally_reserved":0,"allocated":1237504000,"data_stored":1236666308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":945474562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,6,8,10,7,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17543168,"num_objects":-4283,"num_object_clones":0,"num_object_copies":-8566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-62,"num_objects_misplaced":-238,"num_objects_unfound":0,"num_objects_dirty":-4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4283,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":300,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004137"},"pg_stats":[{"pgid":"2.f","version":"44'11777","reported_seq":16927,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:11.368934+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:11.368934+0000","last_peered":"2024-09-16T10:28:11.368934+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:11.368934+0000","last_undegraded":"2024-09-16T10:28:11.368934+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35598336,"num_objects":8691,"num_object_clones":0,"num_object_copies":17382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1652,"num_objects_unfound":0,"num_objects_dirty":8691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11777,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2540,"num_bytes_recovered":10403840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7039},{"shards":"0,2","objects":1652}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11752","reported_seq":11909,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:11.209277+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:11.209277+0000","last_peered":"2024-09-16T10:28:11.209277+0000","last_clean":"2024-09-16T10:28:11.209277+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:11.209277+0000","last_undegraded":"2024-09-16T10:28:11.209277+0000","last_fullsized":"2024-09-16T10:28:11.209277+0000","mapping_epoch":34,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35676160,"num_objects":8710,"num_object_clones":0,"num_object_copies":17420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7965,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'11890","reported_seq":12097,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.314569+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:11.314569+0000","last_peered":"2024-09-16T10:28:11.314569+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:11.314569+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35934232,"num_objects":8774,"num_object_clones":0,"num_object_copies":17548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8774,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7947,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2779},{"shards":"2","objects":4449}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11776","reported_seq":11833,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:11.188095+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:11.188095+0000","last_peered":"2024-09-16T10:28:11.188095+0000","last_clean":"2024-09-16T10:28:11.188095+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:11.188095+0000","last_undegraded":"2024-09-16T10:28:11.188095+0000","last_fullsized":"2024-09-16T10:28:11.188095+0000","mapping_epoch":34,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35577856,"num_objects":8686,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11776,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11749","reported_seq":11906,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:11.219227+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:11.219227+0000","last_peered":"2024-09-16T10:28:11.219227+0000","last_clean":"2024-09-16T10:28:11.219227+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:11.219227+0000","last_undegraded":"2024-09-16T10:28:11.219227+0000","last_fullsized":"2024-09-16T10:28:11.219227+0000","mapping_epoch":33,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35864576,"num_objects":8756,"num_object_clones":0,"num_object_copies":17512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8391,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11676","reported_seq":11888,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.273804+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:11.273804+0000","last_peered":"2024-09-16T10:28:11.273804+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:11.273804+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35614720,"num_objects":8695,"num_object_clones":0,"num_object_copies":17390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4194,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8301,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4503},{"shards":"0,2","objects":2729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12100","reported_seq":12370,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.345741+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:11.345741+0000","last_peered":"2024-09-16T10:28:11.345741+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:11.345741+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36126720,"num_objects":8820,"num_object_clones":0,"num_object_copies":17640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8342,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4465},{"shards":"0,2","objects":2766}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11664","reported_seq":11839,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:11.322813+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:11.322813+0000","last_peered":"2024-09-16T10:28:11.322813+0000","last_clean":"2024-09-16T10:28:11.322813+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:11.322813+0000","last_undegraded":"2024-09-16T10:28:11.322813+0000","last_fullsized":"2024-09-16T10:28:11.322813+0000","mapping_epoch":33,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35741696,"num_objects":8726,"num_object_clones":0,"num_object_copies":17452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11717","reported_seq":20221,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:09.595974+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:09.595974+0000","last_peered":"2024-09-16T10:28:09.595974+0000","last_clean":"2024-09-16T10:28:09.595974+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:09.595974+0000","last_undegraded":"2024-09-16T10:28:09.595974+0000","last_fullsized":"2024-09-16T10:28:09.595974+0000","mapping_epoch":43,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35816390,"num_objects":8746,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12138","reported_seq":12279,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.306732+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:11.306732+0000","last_peered":"2024-09-16T10:28:11.306732+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:11.306732+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36463558,"num_objects":8904,"num_object_clones":0,"num_object_copies":17808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12136,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4501},{"shards":"0,2","objects":2820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11717","reported_seq":11851,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.229865+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:11.229865+0000","last_peered":"2024-09-16T10:28:11.229865+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:11.229865+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35644358,"num_objects":8704,"num_object_clones":0,"num_object_copies":17408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4265,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11715,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2772},{"shards":"2","objects":4441}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11825","reported_seq":11930,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.310549+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:11.310549+0000","last_peered":"2024-09-16T10:28:11.310549+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:11.310549+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35845062,"num_objects":8753,"num_object_clones":0,"num_object_copies":17506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11823,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2658},{"shards":"2","objects":4570}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11492","reported_seq":11646,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.387653+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:11.387653+0000","last_peered":"2024-09-16T10:28:11.387653+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:11.387653+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1400","ondisk_log_start":"35'1400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34988515,"num_objects":8543,"num_object_clones":0,"num_object_copies":17086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11491,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4422},{"shards":"0,2","objects":2685}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12036","reported_seq":12211,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.372629+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:11.372629+0000","last_peered":"2024-09-16T10:28:11.372629+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:11.372629+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36365737,"num_objects":8881,"num_object_clones":0,"num_object_copies":17762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12033,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4565},{"shards":"0,2","objects":2762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11633","reported_seq":11782,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.302717+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:11.302717+0000","last_peered":"2024-09-16T10:28:11.302717+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:11.302717+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34845638,"num_objects":8509,"num_object_clones":0,"num_object_copies":17018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11631,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4376},{"shards":"0,2","objects":2606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":63,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:01.942444+0000","last_peered":"2024-09-16T10:26:01.942444+0000","last_clean":"2024-09-16T10:26:01.942444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:01.942444+0000","last_undegraded":"2024-09-16T10:26:01.942444+0000","last_fullsized":"2024-09-16T10:26:01.942444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12070","reported_seq":12226,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:11.269447+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:11.269447+0000","last_peered":"2024-09-16T10:28:11.269447+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:11.269447+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36153228,"num_objects":8830,"num_object_clones":0,"num_object_copies":17660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12067,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4468},{"shards":"0,2","objects":2779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":572256782,"num_objects":139728,"num_object_clones":0,"num_object_copies":279456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42673,"num_objects_misplaced":1652,"num_objects_unfound":0,"num_objects_dirty":139728,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166577,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7021,"num_bytes_recovered":27599331,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1235783680,"data_stored":1235692437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160912,"ondisk_log_size":160912,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018948,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701260,"kb_used_data":416960,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93670580,"statfs":{"total":96636764160,"available":95918673920,"internally_reserved":0,"allocated":426967040,"data_stored":426709334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.8959999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.48399999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.5249999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":7.1970000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691892,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574120,"kb_used_data":303776,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93797720,"statfs":{"total":96636764160,"available":96048865280,"internally_reserved":0,"allocated":311066624,"data_stored":310736658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":276819626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,4,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.5640000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":8.2189999999999994}]}]},{"osd":0,"up_from":24,"seq":103079215176,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":856476,"kb_used_data":487764,"kb_used_omap":4,"kb_used_meta":368699,"kb_avail":93515364,"statfs":{"total":96636764160,"available":95759732736,"internally_reserved":0,"allocated":499470336,"data_stored":499220316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":377548456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,5,5,1,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.83299999999999996},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.173}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":498618368,"data_stored":498614755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":310693888,"data_stored":310613943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":426471424,"data_stored":426463739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:16.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:16.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'11777", 'reported_seq': 16927, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:11.368934+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:11.368934+0000', 'last_peered': '2024-09-16T10:28:11.368934+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:11.368934+0000', 'last_undegraded': '2024-09-16T10:28:11.368934+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'1700", 'ondisk_log_start': "35'1700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35598336, 'num_objects': 8691, 'num_object_clones': 0, 'num_object_copies': 17382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1652, 'num_objects_unfound': 0, 'num_objects_dirty': 8691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11777, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2540, 'num_bytes_recovered': 10403840, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7039}, {'shards': '0,2', 'objects': 1652}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:16.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:16.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'11890", 'reported_seq': 12097, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:11.314569+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:11.314569+0000', 'last_peered': '2024-09-16T10:28:11.314569+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:11.314569+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'1800", 'ondisk_log_start': "35'1800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35934232, 'num_objects': 8774, 'num_object_clones': 0, 'num_object_copies': 17548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8774, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7947, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 10, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2779}, {'shards': '2', 'objects': 4449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:16.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:16.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'11676", 'reported_seq': 11888, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:11.273804+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:11.273804+0000', 'last_peered': '2024-09-16T10:28:11.273804+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:11.273804+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'1600", 'ondisk_log_start': "35'1600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35614720, 'num_objects': 8695, 'num_object_clones': 0, 'num_object_copies': 17390, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4194, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8301, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 25, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4503}, {'shards': '0,2', 'objects': 2729}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:16.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:16.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'12100", 'reported_seq': 12370, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:11.345741+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:11.345741+0000', 'last_peered': '2024-09-16T10:28:11.345741+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:11.345741+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'2000", 'ondisk_log_start': "35'2000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36126720, 'num_objects': 8820, 'num_object_clones': 0, 'num_object_copies': 17640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4357, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8342, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 48, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4465}, {'shards': '0,2', 'objects': 2766}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:16.006 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'12138", 'reported_seq': 12279, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:11.306732+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:11.306732+0000', 'last_peered': '2024-09-16T10:28:11.306732+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:11.306732+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'2100", 'ondisk_log_start': "35'2100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36463558, 'num_objects': 8904, 'num_object_clones': 0, 'num_object_copies': 17808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4405, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12136, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4501}, {'shards': '0,2', 'objects': 2820}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'11717", 'reported_seq': 11851, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:11.229865+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:11.229865+0000', 'last_peered': '2024-09-16T10:28:11.229865+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:11.229865+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'1700", 'ondisk_log_start': "35'1700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35644358, 'num_objects': 8704, 'num_object_clones': 0, 'num_object_copies': 17408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4265, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8704, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11715, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2772}, {'shards': '2', 'objects': 4441}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'11825", 'reported_seq': 11930, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:11.310549+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:11.310549+0000', 'last_peered': '2024-09-16T10:28:11.310549+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:11.310549+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'1800", 'ondisk_log_start': "35'1800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35845062, 'num_objects': 8753, 'num_object_clones': 0, 'num_object_copies': 17506, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4185, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8753, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11823, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 11, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2658}, {'shards': '2', 'objects': 4570}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'11492", 'reported_seq': 11646, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:11.387653+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:11.387653+0000', 'last_peered': '2024-09-16T10:28:11.387653+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:11.387653+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'1400", 'ondisk_log_start': "35'1400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34988515, 'num_objects': 8543, 'num_object_clones': 0, 'num_object_copies': 17086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4122, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11491, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4422}, {'shards': '0,2', 'objects': 2685}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'12036", 'reported_seq': 12211, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:11.372629+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:11.372629+0000', 'last_peered': '2024-09-16T10:28:11.372629+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:11.372629+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'2000", 'ondisk_log_start': "35'2000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36365737, 'num_objects': 8881, 'num_object_clones': 0, 'num_object_copies': 17762, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12033, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4565}, {'shards': '0,2', 'objects': 2762}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'11633", 'reported_seq': 11782, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:11.302717+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:11.302717+0000', 'last_peered': '2024-09-16T10:28:11.302717+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:11.302717+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'1600", 'ondisk_log_start': "35'1600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34845638, 'num_objects': 8509, 'num_object_clones': 0, 'num_object_copies': 17018, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4135, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11631, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4376}, {'shards': '0,2', 'objects': 2606}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:16.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:16.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'12070", 'reported_seq': 12226, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:11.269447+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:11.269447+0000', 'last_peered': '2024-09-16T10:28:11.269447+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:11.269447+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'2000", 'ondisk_log_start': "35'2000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36153228, 'num_objects': 8830, 'num_object_clones': 0, 'num_object_copies': 17660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4365, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8830, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12067, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4468}, {'shards': '0,2', 'objects': 2779}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:16.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:16.685 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:16.685 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42673/279460 objects degraded (15.270%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":385,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":139730,"data_bytes":572256801,"bytes_used":2183020544,"bytes_avail":287727271936,"bytes_total":289910292480,"degraded_objects":42673,"degraded_total":279460,"degraded_ratio":0.15269806054533744,"misplaced_objects":1652,"misplaced_total":279460,"misplaced_ratio":0.0059114005582194228,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":81209,"recovering_keys_per_sec":0,"num_objects_recovered":300,"num_bytes_recovered":974848,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":356},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:16.717 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:16.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:17.272 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:17.292 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-16T10:28:16.665053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":563720737,"num_objects":137646,"num_object_clones":0,"num_object_copies":275292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42638,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":137646,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168663,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7203,"num_bytes_recovered":28197366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2112688,"kb_used_data":1197520,"kb_used_omap":12,"kb_used_meta":915123,"kb_avail":281002832,"statfs":{"total":289910292480,"available":287746899968,"internally_reserved":0,"allocated":1226260480,"data_stored":1225422788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":937085954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,0,1,7,9,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26079232,"num_objects":-6367,"num_object_clones":0,"num_object_copies":-12734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":-384,"num_objects_unfound":0,"num_objects_dirty":-6367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6367,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":481,"num_bytes_recovered":1572864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003921"},"pg_stats":[{"pgid":"2.f","version":"44'11906","reported_seq":17348,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:16.462424+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:16.462424+0000","last_peered":"2024-09-16T10:28:16.462424+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:16.462424+0000","last_undegraded":"2024-09-16T10:28:16.462424+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35069952,"num_objects":8562,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":8562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2686,"num_bytes_recovered":11001856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7056},{"shards":"0,2","objects":1506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11884","reported_seq":12041,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.331871+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:16.331871+0000","last_peered":"2024-09-16T10:28:16.331871+0000","last_clean":"2024-09-16T10:28:16.331871+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:16.331871+0000","last_undegraded":"2024-09-16T10:28:16.331871+0000","last_fullsized":"2024-09-16T10:28:16.331871+0000","mapping_epoch":34,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35135488,"num_objects":8578,"num_object_clones":0,"num_object_copies":17156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8097,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12013","reported_seq":12224,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.277337+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:16.277337+0000","last_peered":"2024-09-16T10:28:16.277337+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:16.277337+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35430424,"num_objects":8651,"num_object_clones":0,"num_object_copies":17302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8651,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8070,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2656},{"shards":"2","objects":4328}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11921","reported_seq":11978,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.332284+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:16.332284+0000","last_peered":"2024-09-16T10:28:16.332284+0000","last_clean":"2024-09-16T10:28:16.332284+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:16.332284+0000","last_undegraded":"2024-09-16T10:28:16.332284+0000","last_fullsized":"2024-09-16T10:28:16.332284+0000","mapping_epoch":34,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34983936,"num_objects":8541,"num_object_clones":0,"num_object_copies":17082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11921,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11878","reported_seq":12035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.384381+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:16.384381+0000","last_peered":"2024-09-16T10:28:16.384381+0000","last_clean":"2024-09-16T10:28:16.384381+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:16.384381+0000","last_undegraded":"2024-09-16T10:28:16.384381+0000","last_fullsized":"2024-09-16T10:28:16.384381+0000","mapping_epoch":33,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35336192,"num_objects":8627,"num_object_clones":0,"num_object_copies":17254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8520,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11788","reported_seq":12004,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.358864+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:16.358864+0000","last_peered":"2024-09-16T10:28:16.358864+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:16.358864+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35155968,"num_objects":8583,"num_object_clones":0,"num_object_copies":17166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4392},{"shards":"0,2","objects":2617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12231","reported_seq":12533,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.403650+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:16.403650+0000","last_peered":"2024-09-16T10:28:16.403650+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:16.403650+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35590144,"num_objects":8689,"num_object_clones":0,"num_object_copies":17378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4350},{"shards":"0,2","objects":2635}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11806","reported_seq":11980,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.441869+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:16.441869+0000","last_peered":"2024-09-16T10:28:16.441869+0000","last_clean":"2024-09-16T10:28:16.441869+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:16.441869+0000","last_undegraded":"2024-09-16T10:28:16.441869+0000","last_fullsized":"2024-09-16T10:28:16.441869+0000","mapping_epoch":33,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35160064,"num_objects":8584,"num_object_clones":0,"num_object_copies":17168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7609,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11838","reported_seq":20342,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:14.677192+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:14.677192+0000","last_peered":"2024-09-16T10:28:14.677192+0000","last_clean":"2024-09-16T10:28:14.677192+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:14.677192+0000","last_undegraded":"2024-09-16T10:28:14.677192+0000","last_fullsized":"2024-09-16T10:28:14.677192+0000","mapping_epoch":43,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35320774,"num_objects":8625,"num_object_clones":0,"num_object_copies":17250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11836,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12266","reported_seq":12413,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.442913+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:16.442913+0000","last_peered":"2024-09-16T10:28:16.442913+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:16.442913+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35939270,"num_objects":8776,"num_object_clones":0,"num_object_copies":17552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12264,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4376},{"shards":"0,2","objects":2692}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11858","reported_seq":11994,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.187307+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:16.187307+0000","last_peered":"2024-09-16T10:28:16.187307+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:16.187307+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35066822,"num_objects":8563,"num_object_clones":0,"num_object_copies":17126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11856,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2631},{"shards":"2","objects":4301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11958","reported_seq":12069,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.166080+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:16.166080+0000","last_peered":"2024-09-16T10:28:16.166080+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:16.166080+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35300294,"num_objects":8620,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11956,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2525},{"shards":"2","objects":4440}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11601","reported_seq":11759,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.349214+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:16.349214+0000","last_peered":"2024-09-16T10:28:16.349214+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:16.349214+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34542051,"num_objects":8434,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4314},{"shards":"0,2","objects":2576}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12176","reported_seq":12352,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.462032+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:16.462032+0000","last_peered":"2024-09-16T10:28:16.462032+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:16.462032+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35792297,"num_objects":8741,"num_object_clones":0,"num_object_copies":17482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12173,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4425},{"shards":"0,2","objects":2622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11773","reported_seq":11930,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.398612+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:16.398612+0000","last_peered":"2024-09-16T10:28:16.398612+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:16.398612+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34272198,"num_objects":8369,"num_object_clones":0,"num_object_copies":16738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4240},{"shards":"0,2","objects":2466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12199","reported_seq":12365,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.381945+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:16.381945+0000","last_peered":"2024-09-16T10:28:16.381945+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:16.381945+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35624844,"num_objects":8701,"num_object_clones":0,"num_object_copies":17402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12196,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4343},{"shards":"0,2","objects":2650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":563720718,"num_objects":137644,"num_object_clones":0,"num_object_copies":275288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42638,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":137644,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168661,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7202,"num_bytes_recovered":28197347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1224470528,"data_stored":1224379285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160796,"ondisk_log_size":160796,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018949,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696968,"kb_used_data":412672,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93674872,"statfs":{"total":96636764160,"available":95923068928,"internally_reserved":0,"allocated":422576128,"data_stored":422318422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.27},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.304}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.286},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":0.997}]}]},{"osd":1,"up_from":40,"seq":171798691893,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577812,"kb_used_data":303372,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93794028,"statfs":{"total":96636764160,"available":96045084672,"internally_reserved":0,"allocated":310652928,"data_stored":310322962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":281013930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.9139999999999999},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.8839999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":2.54},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.26500000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215177,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":837908,"kb_used_data":481476,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93533932,"statfs":{"total":96636764160,"available":95778746368,"internally_reserved":0,"allocated":493031424,"data_stored":492781404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":364965544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,5,1,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.056},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.2960000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":6.0830000000000002},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":492138496,"data_stored":492134883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":310267904,"data_stored":310187959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":422064128,"data_stored":422056443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:20.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:20.847 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:20.875 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-16T10:28:20.666020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":563196449,"num_objects":137518,"num_object_clones":0,"num_object_copies":275036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42638,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":137518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168791,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7203,"num_bytes_recovered":28197366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2103984,"kb_used_data":1197008,"kb_used_omap":12,"kb_used_meta":906931,"kb_avail":281011536,"statfs":{"total":289910292480,"available":287755812864,"internally_reserved":0,"allocated":1225736192,"data_stored":1224898500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":928697346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,8,2,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17879040,"num_objects":-4365,"num_object_clones":0,"num_object_copies":-8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":-257,"num_objects_unfound":0,"num_objects_dirty":-4365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":1052672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003122"},"pg_stats":[{"pgid":"2.f","version":"44'11906","reported_seq":17348,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:16.462424+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:16.462424+0000","last_peered":"2024-09-16T10:28:16.462424+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:16.462424+0000","last_undegraded":"2024-09-16T10:28:16.462424+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35069952,"num_objects":8562,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":8562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2686,"num_bytes_recovered":11001856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7056},{"shards":"0,2","objects":1506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11884","reported_seq":12041,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.331871+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:16.331871+0000","last_peered":"2024-09-16T10:28:16.331871+0000","last_clean":"2024-09-16T10:28:16.331871+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:16.331871+0000","last_undegraded":"2024-09-16T10:28:16.331871+0000","last_fullsized":"2024-09-16T10:28:16.331871+0000","mapping_epoch":34,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35135488,"num_objects":8578,"num_object_clones":0,"num_object_copies":17156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8097,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12013","reported_seq":12224,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.277337+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:16.277337+0000","last_peered":"2024-09-16T10:28:16.277337+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:16.277337+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35430424,"num_objects":8651,"num_object_clones":0,"num_object_copies":17302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8651,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8070,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2656},{"shards":"2","objects":4328}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11921","reported_seq":11978,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.332284+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:16.332284+0000","last_peered":"2024-09-16T10:28:16.332284+0000","last_clean":"2024-09-16T10:28:16.332284+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:16.332284+0000","last_undegraded":"2024-09-16T10:28:16.332284+0000","last_fullsized":"2024-09-16T10:28:16.332284+0000","mapping_epoch":34,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34983936,"num_objects":8541,"num_object_clones":0,"num_object_copies":17082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11921,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11878","reported_seq":12035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.384381+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:16.384381+0000","last_peered":"2024-09-16T10:28:16.384381+0000","last_clean":"2024-09-16T10:28:16.384381+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:16.384381+0000","last_undegraded":"2024-09-16T10:28:16.384381+0000","last_fullsized":"2024-09-16T10:28:16.384381+0000","mapping_epoch":33,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35336192,"num_objects":8627,"num_object_clones":0,"num_object_copies":17254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8520,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11788","reported_seq":12004,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.358864+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:16.358864+0000","last_peered":"2024-09-16T10:28:16.358864+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:16.358864+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35155968,"num_objects":8583,"num_object_clones":0,"num_object_copies":17166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4392},{"shards":"0,2","objects":2617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12231","reported_seq":12533,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.403650+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:16.403650+0000","last_peered":"2024-09-16T10:28:16.403650+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:16.403650+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35590144,"num_objects":8689,"num_object_clones":0,"num_object_copies":17378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4350},{"shards":"0,2","objects":2635}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11806","reported_seq":11980,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.441869+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:16.441869+0000","last_peered":"2024-09-16T10:28:16.441869+0000","last_clean":"2024-09-16T10:28:16.441869+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:16.441869+0000","last_undegraded":"2024-09-16T10:28:16.441869+0000","last_fullsized":"2024-09-16T10:28:16.441869+0000","mapping_epoch":33,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35160064,"num_objects":8584,"num_object_clones":0,"num_object_copies":17168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7609,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11966","reported_seq":20470,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:19.806747+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:19.806747+0000","last_peered":"2024-09-16T10:28:19.806747+0000","last_clean":"2024-09-16T10:28:19.806747+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:19.806747+0000","last_undegraded":"2024-09-16T10:28:19.806747+0000","last_fullsized":"2024-09-16T10:28:19.806747+0000","mapping_epoch":43,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34796486,"num_objects":8497,"num_object_clones":0,"num_object_copies":16994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12266","reported_seq":12413,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.442913+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:16.442913+0000","last_peered":"2024-09-16T10:28:16.442913+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:16.442913+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35939270,"num_objects":8776,"num_object_clones":0,"num_object_copies":17552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12264,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4376},{"shards":"0,2","objects":2692}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11858","reported_seq":11994,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.187307+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:16.187307+0000","last_peered":"2024-09-16T10:28:16.187307+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:16.187307+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35066822,"num_objects":8563,"num_object_clones":0,"num_object_copies":17126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11856,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2631},{"shards":"2","objects":4301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11958","reported_seq":12069,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.166080+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:16.166080+0000","last_peered":"2024-09-16T10:28:16.166080+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:16.166080+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35300294,"num_objects":8620,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11956,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2525},{"shards":"2","objects":4440}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11601","reported_seq":11759,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.349214+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:16.349214+0000","last_peered":"2024-09-16T10:28:16.349214+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:16.349214+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34542051,"num_objects":8434,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4314},{"shards":"0,2","objects":2576}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12176","reported_seq":12352,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.462032+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:16.462032+0000","last_peered":"2024-09-16T10:28:16.462032+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:16.462032+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35792297,"num_objects":8741,"num_object_clones":0,"num_object_copies":17482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12173,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4425},{"shards":"0,2","objects":2622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11773","reported_seq":11930,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.398612+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:16.398612+0000","last_peered":"2024-09-16T10:28:16.398612+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:16.398612+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34272198,"num_objects":8369,"num_object_clones":0,"num_object_copies":16738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4240},{"shards":"0,2","objects":2466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12199","reported_seq":12365,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.381945+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:16.381945+0000","last_peered":"2024-09-16T10:28:16.381945+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:16.381945+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35624844,"num_objects":8701,"num_object_clones":0,"num_object_copies":17402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12196,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4343},{"shards":"0,2","objects":2650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":563196430,"num_objects":137516,"num_object_clones":0,"num_object_copies":275032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42638,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":137516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168789,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7202,"num_bytes_recovered":28197347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1223946240,"data_stored":1223854997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018949,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696968,"kb_used_data":412672,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93674872,"statfs":{"total":96636764160,"available":95923068928,"internally_reserved":0,"allocated":422576128,"data_stored":422318422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.27},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.304}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.286},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":0.997}]}]},{"osd":1,"up_from":40,"seq":171798691894,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569108,"kb_used_data":302860,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93802732,"statfs":{"total":96636764160,"available":96053997568,"internally_reserved":0,"allocated":310128640,"data_stored":309798674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":272625322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.224},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.78200000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":3.4710000000000001},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":3.4460000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215177,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":837908,"kb_used_data":481476,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93533932,"statfs":{"total":96636764160,"available":95778746368,"internally_reserved":0,"allocated":493031424,"data_stored":492781404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":364965544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,5,1,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.056},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.2960000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":6.0830000000000002},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":492138496,"data_stored":492134883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309743616,"data_stored":309663671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":422064128,"data_stored":422056443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:20.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:21.415 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:21.417 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:21.436 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-16T10:28:20.666020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":563196449,"num_objects":137518,"num_object_clones":0,"num_object_copies":275036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42638,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":137518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168791,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7203,"num_bytes_recovered":28197366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2103984,"kb_used_data":1197008,"kb_used_omap":12,"kb_used_meta":906931,"kb_avail":281011536,"statfs":{"total":289910292480,"available":287755812864,"internally_reserved":0,"allocated":1225736192,"data_stored":1224898500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":928697346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,8,2,6,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17879040,"num_objects":-4365,"num_object_clones":0,"num_object_copies":-8730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":-257,"num_objects_unfound":0,"num_objects_dirty":-4365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":1052672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003122"},"pg_stats":[{"pgid":"2.f","version":"44'11906","reported_seq":17348,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:16.462424+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:16.462424+0000","last_peered":"2024-09-16T10:28:16.462424+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:16.462424+0000","last_undegraded":"2024-09-16T10:28:16.462424+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35069952,"num_objects":8562,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":8562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2686,"num_bytes_recovered":11001856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7056},{"shards":"0,2","objects":1506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'11884","reported_seq":12041,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.331871+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:16.331871+0000","last_peered":"2024-09-16T10:28:16.331871+0000","last_clean":"2024-09-16T10:28:16.331871+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:16.331871+0000","last_undegraded":"2024-09-16T10:28:16.331871+0000","last_fullsized":"2024-09-16T10:28:16.331871+0000","mapping_epoch":34,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35135488,"num_objects":8578,"num_object_clones":0,"num_object_copies":17156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8097,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12013","reported_seq":12224,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.277337+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:16.277337+0000","last_peered":"2024-09-16T10:28:16.277337+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:16.277337+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35430424,"num_objects":8651,"num_object_clones":0,"num_object_copies":17302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8651,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8070,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2656},{"shards":"2","objects":4328}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'11921","reported_seq":11978,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.332284+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:16.332284+0000","last_peered":"2024-09-16T10:28:16.332284+0000","last_clean":"2024-09-16T10:28:16.332284+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:16.332284+0000","last_undegraded":"2024-09-16T10:28:16.332284+0000","last_fullsized":"2024-09-16T10:28:16.332284+0000","mapping_epoch":34,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34983936,"num_objects":8541,"num_object_clones":0,"num_object_copies":17082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11921,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'11878","reported_seq":12035,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.384381+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:16.384381+0000","last_peered":"2024-09-16T10:28:16.384381+0000","last_clean":"2024-09-16T10:28:16.384381+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:16.384381+0000","last_undegraded":"2024-09-16T10:28:16.384381+0000","last_fullsized":"2024-09-16T10:28:16.384381+0000","mapping_epoch":33,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35336192,"num_objects":8627,"num_object_clones":0,"num_object_copies":17254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8520,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11788","reported_seq":12004,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.358864+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:16.358864+0000","last_peered":"2024-09-16T10:28:16.358864+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:16.358864+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35155968,"num_objects":8583,"num_object_clones":0,"num_object_copies":17166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4392},{"shards":"0,2","objects":2617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12231","reported_seq":12533,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.403650+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:16.403650+0000","last_peered":"2024-09-16T10:28:16.403650+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:16.403650+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35590144,"num_objects":8689,"num_object_clones":0,"num_object_copies":17378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8473,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4350},{"shards":"0,2","objects":2635}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11806","reported_seq":11980,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:16.441869+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:16.441869+0000","last_peered":"2024-09-16T10:28:16.441869+0000","last_clean":"2024-09-16T10:28:16.441869+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:16.441869+0000","last_undegraded":"2024-09-16T10:28:16.441869+0000","last_fullsized":"2024-09-16T10:28:16.441869+0000","mapping_epoch":33,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35160064,"num_objects":8584,"num_object_clones":0,"num_object_copies":17168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7609,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11966","reported_seq":20470,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:19.806747+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:19.806747+0000","last_peered":"2024-09-16T10:28:19.806747+0000","last_clean":"2024-09-16T10:28:19.806747+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:19.806747+0000","last_undegraded":"2024-09-16T10:28:19.806747+0000","last_fullsized":"2024-09-16T10:28:19.806747+0000","mapping_epoch":43,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34796486,"num_objects":8497,"num_object_clones":0,"num_object_copies":16994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12266","reported_seq":12413,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.442913+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:16.442913+0000","last_peered":"2024-09-16T10:28:16.442913+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:16.442913+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35939270,"num_objects":8776,"num_object_clones":0,"num_object_copies":17552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4402,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12264,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4376},{"shards":"0,2","objects":2692}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11858","reported_seq":11994,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.187307+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:16.187307+0000","last_peered":"2024-09-16T10:28:16.187307+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:16.187307+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35066822,"num_objects":8563,"num_object_clones":0,"num_object_copies":17126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11856,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2631},{"shards":"2","objects":4301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'11958","reported_seq":12069,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.166080+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:16.166080+0000","last_peered":"2024-09-16T10:28:16.166080+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:16.166080+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35300294,"num_objects":8620,"num_object_clones":0,"num_object_copies":17240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11956,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2525},{"shards":"2","objects":4440}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11601","reported_seq":11759,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.349214+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:16.349214+0000","last_peered":"2024-09-16T10:28:16.349214+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:16.349214+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1600","ondisk_log_start":"35'1600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34542051,"num_objects":8434,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11600,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4314},{"shards":"0,2","objects":2576}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12176","reported_seq":12352,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.462032+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:16.462032+0000","last_peered":"2024-09-16T10:28:16.462032+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:16.462032+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35792297,"num_objects":8741,"num_object_clones":0,"num_object_copies":17482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12173,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4425},{"shards":"0,2","objects":2622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11773","reported_seq":11930,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.398612+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:16.398612+0000","last_peered":"2024-09-16T10:28:16.398612+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:16.398612+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34272198,"num_objects":8369,"num_object_clones":0,"num_object_copies":16738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11771,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4240},{"shards":"0,2","objects":2466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12199","reported_seq":12365,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:16.381945+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:16.381945+0000","last_peered":"2024-09-16T10:28:16.381945+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:16.381945+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35624844,"num_objects":8701,"num_object_clones":0,"num_object_copies":17402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12196,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4343},{"shards":"0,2","objects":2650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":563196430,"num_objects":137516,"num_object_clones":0,"num_object_copies":275032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42638,"num_objects_misplaced":1506,"num_objects_unfound":0,"num_objects_dirty":137516,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168789,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7202,"num_bytes_recovered":28197347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1223946240,"data_stored":1223854997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018949,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696968,"kb_used_data":412672,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93674872,"statfs":{"total":96636764160,"available":95923068928,"internally_reserved":0,"allocated":422576128,"data_stored":422318422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.27},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.304}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.286},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":0.997}]}]},{"osd":1,"up_from":40,"seq":171798691894,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569108,"kb_used_data":302860,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93802732,"statfs":{"total":96636764160,"available":96053997568,"internally_reserved":0,"allocated":310128640,"data_stored":309798674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":272625322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.224},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.78200000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":3.4710000000000001},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":3.4460000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215177,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":837908,"kb_used_data":481476,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93533932,"statfs":{"total":96636764160,"available":95778746368,"internally_reserved":0,"allocated":493031424,"data_stored":492781404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":364965544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,5,1,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.056},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.2960000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":6.0830000000000002},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":492138496,"data_stored":492134883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309743616,"data_stored":309663671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":422064128,"data_stored":422056443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:21.437 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:21.437 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'11906", 'reported_seq': 17348, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:16.462424+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:16.462424+0000', 'last_peered': '2024-09-16T10:28:16.462424+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:16.462424+0000', 'last_undegraded': '2024-09-16T10:28:16.462424+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'1900", 'ondisk_log_start': "35'1900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35069952, 'num_objects': 8562, 'num_object_clones': 0, 'num_object_copies': 17124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1506, 'num_objects_unfound': 0, 'num_objects_dirty': 8562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11906, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2686, 'num_bytes_recovered': 11001856, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7056}, {'shards': '0,2', 'objects': 1506}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:21.438 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:21.438 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'12013", 'reported_seq': 12224, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:16.277337+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:16.277337+0000', 'last_peered': '2024-09-16T10:28:16.277337+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:16.277337+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'2000", 'ondisk_log_start': "35'2000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35430424, 'num_objects': 8651, 'num_object_clones': 0, 'num_object_copies': 17302, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4324, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8651, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8070, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2656}, {'shards': '2', 'objects': 4328}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:21.438 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:21.438 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'11788", 'reported_seq': 12004, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:16.358864+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:16.358864+0000', 'last_peered': '2024-09-16T10:28:16.358864+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:16.358864+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'1700", 'ondisk_log_start': "35'1700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35155968, 'num_objects': 8583, 'num_object_clones': 0, 'num_object_copies': 17166, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4193, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8413, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4392}, {'shards': '0,2', 'objects': 2617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:21.438 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:21.438 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'12231", 'reported_seq': 12533, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:16.403650+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:16.403650+0000', 'last_peered': '2024-09-16T10:28:16.403650+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:16.403650+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'2200", 'ondisk_log_start': "35'2200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35590144, 'num_objects': 8689, 'num_object_clones': 0, 'num_object_copies': 17378, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4341, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8473, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 64, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4350}, {'shards': '0,2', 'objects': 2635}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:21.438 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:21.439 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'12266", 'reported_seq': 12413, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:16.442913+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:16.442913+0000', 'last_peered': '2024-09-16T10:28:16.442913+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:16.442913+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'2200", 'ondisk_log_start': "35'2200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35939270, 'num_objects': 8776, 'num_object_clones': 0, 'num_object_copies': 17552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4402, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12264, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 36, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4376}, {'shards': '0,2', 'objects': 2692}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:21.439 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:21.439 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'11858", 'reported_seq': 11994, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:16.187307+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:16.187307+0000', 'last_peered': '2024-09-16T10:28:16.187307+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:16.187307+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'1800", 'ondisk_log_start': "35'1800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35066822, 'num_objects': 8563, 'num_object_clones': 0, 'num_object_copies': 17126, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4264, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11856, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2631}, {'shards': '2', 'objects': 4301}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:21.439 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:21.439 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'11958", 'reported_seq': 12069, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:16.166080+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:16.166080+0000', 'last_peered': '2024-09-16T10:28:16.166080+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:16.166080+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'1900", 'ondisk_log_start': "35'1900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35300294, 'num_objects': 8620, 'num_object_clones': 0, 'num_object_copies': 17240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8620, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11956, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2525}, {'shards': '2', 'objects': 4440}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:21.439 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:21.439 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'11601", 'reported_seq': 11759, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:16.349214+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:16.349214+0000', 'last_peered': '2024-09-16T10:28:16.349214+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:16.349214+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'1600", 'ondisk_log_start': "35'1600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34542051, 'num_objects': 8434, 'num_object_clones': 0, 'num_object_copies': 16868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11600, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4314}, {'shards': '0,2', 'objects': 2576}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:21.439 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:21.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'12176", 'reported_seq': 12352, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:16.462032+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:16.462032+0000', 'last_peered': '2024-09-16T10:28:16.462032+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:16.462032+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'2100", 'ondisk_log_start': "35'2100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35792297, 'num_objects': 8741, 'num_object_clones': 0, 'num_object_copies': 17482, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4319, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12173, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4425}, {'shards': '0,2', 'objects': 2622}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:21.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:21.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'11773", 'reported_seq': 11930, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:16.398612+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:16.398612+0000', 'last_peered': '2024-09-16T10:28:16.398612+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:16.398612+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'1700", 'ondisk_log_start': "35'1700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34272198, 'num_objects': 8369, 'num_object_clones': 0, 'num_object_copies': 16738, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4131, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11771, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4240}, {'shards': '0,2', 'objects': 2466}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:21.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:21.440 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'12199", 'reported_seq': 12365, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:16.381945+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:16.381945+0000', 'last_peered': '2024-09-16T10:28:16.381945+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:16.381945+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'2100", 'ondisk_log_start': "35'2100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35624844, 'num_objects': 8701, 'num_object_clones': 0, 'num_object_copies': 17402, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4361, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8701, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12196, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4343}, {'shards': '0,2', 'objects': 2650}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:21.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:22.118 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:22.119 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42638/275036 objects degraded (15.503%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":390,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":137518,"data_bytes":563196449,"bytes_used":2154479616,"bytes_avail":287755812864,"bytes_total":289910292480,"degraded_objects":42638,"degraded_total":275036,"degraded_ratio":0.15502697828647885,"misplaced_objects":1506,"misplaced_total":275036,"misplaced_ratio":0.0054756468244157135,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":87699,"recovering_keys_per_sec":0,"num_objects_recovered":334,"num_bytes_recovered":1052672,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":363},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:22.144 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:22.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:22.731 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:22.733 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:22.755 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-16T10:28:22.667486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":554889761,"num_objects":135490,"num_object_clones":0,"num_object_copies":270980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42624,"num_objects_misplaced":1381,"num_objects_unfound":0,"num_objects_dirty":135490,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170819,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7342,"num_bytes_recovered":28709366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160755,"ondisk_log_size":160755,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2101372,"kb_used_data":1186208,"kb_used_omap":12,"kb_used_meta":915123,"kb_avail":281014148,"statfs":{"total":289910292480,"available":287758487552,"internally_reserved":0,"allocated":1214676992,"data_stored":1213839300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":937085954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,10,3,8,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25673728,"num_objects":-6268,"num_object_clones":0,"num_object_copies":-12536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":-382,"num_objects_unfound":0,"num_objects_dirty":-6268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6268,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":1564672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003842"},"pg_stats":[{"pgid":"2.f","version":"44'12031","reported_seq":17723,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:21.472378+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:21.472378+0000","last_peered":"2024-09-16T10:28:21.472378+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:21.472378+0000","last_undegraded":"2024-09-16T10:28:21.472378+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34557952,"num_objects":8437,"num_object_clones":0,"num_object_copies":16874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1381,"num_objects_unfound":0,"num_objects_dirty":8437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":11513856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7056},{"shards":"0,2","objects":1381}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12026","reported_seq":12183,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:21.351037+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:21.351037+0000","last_peered":"2024-09-16T10:28:21.351037+0000","last_clean":"2024-09-16T10:28:21.351037+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:21.351037+0000","last_undegraded":"2024-09-16T10:28:21.351037+0000","last_fullsized":"2024-09-16T10:28:21.351037+0000","mapping_epoch":34,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34553856,"num_objects":8436,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8239,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12135","reported_seq":12346,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.295686+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:21.295686+0000","last_peered":"2024-09-16T10:28:21.295686+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:21.295686+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34930712,"num_objects":8529,"num_object_clones":0,"num_object_copies":17058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8192,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2534},{"shards":"2","objects":4206}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12058","reported_seq":12115,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:21.360280+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:21.360280+0000","last_peered":"2024-09-16T10:28:21.360280+0000","last_clean":"2024-09-16T10:28:21.360280+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:21.360280+0000","last_undegraded":"2024-09-16T10:28:21.360280+0000","last_fullsized":"2024-09-16T10:28:21.360280+0000","mapping_epoch":34,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34422784,"num_objects":8404,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12058,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12036","reported_seq":12193,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:21.383444+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:21.383444+0000","last_peered":"2024-09-16T10:28:21.383444+0000","last_clean":"2024-09-16T10:28:21.383444+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:21.383444+0000","last_undegraded":"2024-09-16T10:28:21.383444+0000","last_fullsized":"2024-09-16T10:28:21.383444+0000","mapping_epoch":33,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34689024,"num_objects":8469,"num_object_clones":0,"num_object_copies":16938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8678,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11923","reported_seq":12143,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.473046+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:21.473046+0000","last_peered":"2024-09-16T10:28:21.473046+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:21.473046+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34603008,"num_objects":8448,"num_object_clones":0,"num_object_copies":16896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8548,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4259},{"shards":"0,2","objects":2482}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12366","reported_seq":12676,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.431388+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:21.431388+0000","last_peered":"2024-09-16T10:28:21.431388+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:21.431388+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35037184,"num_objects":8554,"num_object_clones":0,"num_object_copies":17108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8608,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4219},{"shards":"0,2","objects":2500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11922","reported_seq":12098,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:21.186894+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:21.186894+0000","last_peered":"2024-09-16T10:28:21.186894+0000","last_clean":"2024-09-16T10:28:21.186894+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:21.186894+0000","last_undegraded":"2024-09-16T10:28:21.186894+0000","last_fullsized":"2024-09-16T10:28:21.186894+0000","mapping_epoch":33,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34684928,"num_objects":8468,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11966","reported_seq":20470,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:19.806747+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:19.806747+0000","last_peered":"2024-09-16T10:28:19.806747+0000","last_clean":"2024-09-16T10:28:19.806747+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:19.806747+0000","last_undegraded":"2024-09-16T10:28:19.806747+0000","last_fullsized":"2024-09-16T10:28:19.806747+0000","mapping_epoch":43,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34796486,"num_objects":8497,"num_object_clones":0,"num_object_copies":16994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12416","reported_seq":12565,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.436355+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:21.436355+0000","last_peered":"2024-09-16T10:28:21.436355+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:21.436355+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35324870,"num_objects":8626,"num_object_clones":0,"num_object_copies":17252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12414,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4227},{"shards":"0,2","objects":2542}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11994","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.336477+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:21.336477+0000","last_peered":"2024-09-16T10:28:21.336477+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:21.336477+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34509766,"num_objects":8427,"num_object_clones":0,"num_object_copies":16854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11992,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2495},{"shards":"2","objects":4165}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12089","reported_seq":12200,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.353606+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:21.353606+0000","last_peered":"2024-09-16T10:28:21.353606+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:21.353606+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34763718,"num_objects":8489,"num_object_clones":0,"num_object_copies":16978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2394},{"shards":"2","objects":4309}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11729","reported_seq":11890,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.435315+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:21.435315+0000","last_peered":"2024-09-16T10:28:21.435315+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:21.435315+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34017763,"num_objects":8306,"num_object_clones":0,"num_object_copies":16612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11728,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4188},{"shards":"0,2","objects":2448}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12330","reported_seq":12512,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.449659+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:21.449659+0000","last_peered":"2024-09-16T10:28:21.449659+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:21.449659+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35161513,"num_objects":8587,"num_object_clones":0,"num_object_copies":17174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12327,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4274},{"shards":"0,2","objects":2468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11895","reported_seq":12054,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.311410+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:21.311410+0000","last_peered":"2024-09-16T10:28:21.311410+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:21.311410+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33772486,"num_objects":8247,"num_object_clones":0,"num_object_copies":16494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11893,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4119},{"shards":"0,2","objects":2344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12336","reported_seq":12505,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.482104+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:21.482104+0000","last_peered":"2024-09-16T10:28:21.482104+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:21.482104+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35063692,"num_objects":8564,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12333,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4207},{"shards":"0,2","objects":2513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":554889742,"num_objects":135488,"num_object_clones":0,"num_object_copies":270976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42624,"num_objects_misplaced":1381,"num_objects_unfound":0,"num_objects_dirty":135488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170817,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7341,"num_bytes_recovered":28709347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1212923904,"data_stored":1212832661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160752,"ondisk_log_size":160752,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018950,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696824,"kb_used_data":408428,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93675016,"statfs":{"total":96636764160,"available":95923216384,"internally_reserved":0,"allocated":418230272,"data_stored":417972566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.1539999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.75}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":1.716}]}]},{"osd":1,"up_from":40,"seq":171798691894,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569108,"kb_used_data":302860,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93802732,"statfs":{"total":96636764160,"available":96053997568,"internally_reserved":0,"allocated":310128640,"data_stored":309798674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":272625322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.224},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.78200000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":3.4710000000000001},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":3.4460000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215178,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":835440,"kb_used_data":474920,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93536400,"statfs":{"total":96636764160,"available":95781273600,"internally_reserved":0,"allocated":486318080,"data_stored":486068060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,5,1,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.056},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.2960000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":6.0830000000000002},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":485478400,"data_stored":485474787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309743616,"data_stored":309663671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417701888,"data_stored":417694203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:25.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:26.325 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:26.326 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:26.359 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-16T10:28:24.668050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":554889761,"num_objects":135490,"num_object_clones":0,"num_object_copies":270980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42624,"num_objects_misplaced":1381,"num_objects_unfound":0,"num_objects_dirty":135490,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170819,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7342,"num_bytes_recovered":28709366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160755,"ondisk_log_size":160755,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2101372,"kb_used_data":1186208,"kb_used_omap":12,"kb_used_meta":915123,"kb_avail":281014148,"statfs":{"total":289910292480,"available":287758487552,"internally_reserved":0,"allocated":1214676992,"data_stored":1213839300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":937085954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,10,3,8,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17367040,"num_objects":-4240,"num_object_clones":0,"num_object_copies":-8480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-49,"num_objects_misplaced":-271,"num_objects_unfound":0,"num_objects_dirty":-4240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4240,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":320,"num_bytes_recovered":1110016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003842"},"pg_stats":[{"pgid":"2.f","version":"44'12031","reported_seq":17723,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:21.472378+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:21.472378+0000","last_peered":"2024-09-16T10:28:21.472378+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:21.472378+0000","last_undegraded":"2024-09-16T10:28:21.472378+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34557952,"num_objects":8437,"num_object_clones":0,"num_object_copies":16874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1381,"num_objects_unfound":0,"num_objects_dirty":8437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12031,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2811,"num_bytes_recovered":11513856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7056},{"shards":"0,2","objects":1381}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12026","reported_seq":12183,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:21.351037+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:21.351037+0000","last_peered":"2024-09-16T10:28:21.351037+0000","last_clean":"2024-09-16T10:28:21.351037+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:21.351037+0000","last_undegraded":"2024-09-16T10:28:21.351037+0000","last_fullsized":"2024-09-16T10:28:21.351037+0000","mapping_epoch":34,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34553856,"num_objects":8436,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8239,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12135","reported_seq":12346,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.295686+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:21.295686+0000","last_peered":"2024-09-16T10:28:21.295686+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:21.295686+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34930712,"num_objects":8529,"num_object_clones":0,"num_object_copies":17058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8192,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2534},{"shards":"2","objects":4206}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12058","reported_seq":12115,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:21.360280+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:21.360280+0000","last_peered":"2024-09-16T10:28:21.360280+0000","last_clean":"2024-09-16T10:28:21.360280+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:21.360280+0000","last_undegraded":"2024-09-16T10:28:21.360280+0000","last_fullsized":"2024-09-16T10:28:21.360280+0000","mapping_epoch":34,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34422784,"num_objects":8404,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12058,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12036","reported_seq":12193,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:21.383444+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:21.383444+0000","last_peered":"2024-09-16T10:28:21.383444+0000","last_clean":"2024-09-16T10:28:21.383444+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:21.383444+0000","last_undegraded":"2024-09-16T10:28:21.383444+0000","last_fullsized":"2024-09-16T10:28:21.383444+0000","mapping_epoch":33,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34689024,"num_objects":8469,"num_object_clones":0,"num_object_copies":16938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8678,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'11923","reported_seq":12143,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.473046+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:21.473046+0000","last_peered":"2024-09-16T10:28:21.473046+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:21.473046+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34603008,"num_objects":8448,"num_object_clones":0,"num_object_copies":16896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8548,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4259},{"shards":"0,2","objects":2482}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12366","reported_seq":12676,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.431388+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:21.431388+0000","last_peered":"2024-09-16T10:28:21.431388+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:21.431388+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35037184,"num_objects":8554,"num_object_clones":0,"num_object_copies":17108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8608,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4219},{"shards":"0,2","objects":2500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'11922","reported_seq":12098,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:21.186894+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:21.186894+0000","last_peered":"2024-09-16T10:28:21.186894+0000","last_clean":"2024-09-16T10:28:21.186894+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:21.186894+0000","last_undegraded":"2024-09-16T10:28:21.186894+0000","last_fullsized":"2024-09-16T10:28:21.186894+0000","mapping_epoch":33,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34684928,"num_objects":8468,"num_object_clones":0,"num_object_copies":16936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'11966","reported_seq":20470,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:19.806747+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:19.806747+0000","last_peered":"2024-09-16T10:28:19.806747+0000","last_clean":"2024-09-16T10:28:19.806747+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:19.806747+0000","last_undegraded":"2024-09-16T10:28:19.806747+0000","last_fullsized":"2024-09-16T10:28:19.806747+0000","mapping_epoch":43,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34796486,"num_objects":8497,"num_object_clones":0,"num_object_copies":16994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12416","reported_seq":12565,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.436355+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:21.436355+0000","last_peered":"2024-09-16T10:28:21.436355+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:21.436355+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35324870,"num_objects":8626,"num_object_clones":0,"num_object_copies":17252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12414,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4227},{"shards":"0,2","objects":2542}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'11994","reported_seq":12130,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.336477+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:21.336477+0000","last_peered":"2024-09-16T10:28:21.336477+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:21.336477+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'1900","ondisk_log_start":"35'1900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34509766,"num_objects":8427,"num_object_clones":0,"num_object_copies":16854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4264,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11992,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2495},{"shards":"2","objects":4165}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12089","reported_seq":12200,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.353606+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:21.353606+0000","last_peered":"2024-09-16T10:28:21.353606+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:21.353606+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34763718,"num_objects":8489,"num_object_clones":0,"num_object_copies":16978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2394},{"shards":"2","objects":4309}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11729","reported_seq":11890,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.435315+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:21.435315+0000","last_peered":"2024-09-16T10:28:21.435315+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:21.435315+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1700","ondisk_log_start":"35'1700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34017763,"num_objects":8306,"num_object_clones":0,"num_object_copies":16612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11728,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4188},{"shards":"0,2","objects":2448}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12330","reported_seq":12512,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.449659+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:21.449659+0000","last_peered":"2024-09-16T10:28:21.449659+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:21.449659+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35161513,"num_objects":8587,"num_object_clones":0,"num_object_copies":17174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12327,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4274},{"shards":"0,2","objects":2468}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'11895","reported_seq":12054,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.311410+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:21.311410+0000","last_peered":"2024-09-16T10:28:21.311410+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:21.311410+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33772486,"num_objects":8247,"num_object_clones":0,"num_object_copies":16494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11893,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4119},{"shards":"0,2","objects":2344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12336","reported_seq":12505,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:21.482104+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:21.482104+0000","last_peered":"2024-09-16T10:28:21.482104+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:21.482104+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35063692,"num_objects":8564,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12333,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4207},{"shards":"0,2","objects":2513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":554889742,"num_objects":135488,"num_object_clones":0,"num_object_copies":270976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42624,"num_objects_misplaced":1381,"num_objects_unfound":0,"num_objects_dirty":135488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170817,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7341,"num_bytes_recovered":28709347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1212923904,"data_stored":1212832661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160752,"ondisk_log_size":160752,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018950,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696824,"kb_used_data":408428,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93675016,"statfs":{"total":96636764160,"available":95923216384,"internally_reserved":0,"allocated":418230272,"data_stored":417972566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.1539999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.75}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":1.716}]}]},{"osd":1,"up_from":40,"seq":171798691894,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569108,"kb_used_data":302860,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93802732,"statfs":{"total":96636764160,"available":96053997568,"internally_reserved":0,"allocated":310128640,"data_stored":309798674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":272625322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.224},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.78200000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":3.4710000000000001},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":3.4460000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215178,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":835440,"kb_used_data":474920,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93536400,"statfs":{"total":96636764160,"available":95781273600,"internally_reserved":0,"allocated":486318080,"data_stored":486068060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,5,1,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.056},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.2960000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":6.0830000000000002},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":485478400,"data_stored":485474787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309743616,"data_stored":309663671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":417701888,"data_stored":417694203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:26.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:26.899 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:26.900 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:26.921 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-16T10:28:26.668503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":546054689,"num_objects":133333,"num_object_clones":0,"num_object_copies":266666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42615,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":133333,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172976,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7493,"num_bytes_recovered":29290998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2102516,"kb_used_data":1175060,"kb_used_omap":12,"kb_used_meta":927411,"kb_avail":281013004,"statfs":{"total":289910292480,"available":287757316096,"internally_reserved":0,"allocated":1203261440,"data_stored":1202423748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":949668866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,6,4,14,7,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26202112,"num_objects":-6397,"num_object_clones":0,"num_object_copies":-12794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-58,"num_objects_misplaced":-413,"num_objects_unfound":0,"num_objects_dirty":-6397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6397,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":471,"num_bytes_recovered":1691648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003902"},"pg_stats":[{"pgid":"2.f","version":"44'12138","reported_seq":18113,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:26.533366+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:26.533366+0000","last_peered":"2024-09-16T10:28:26.533366+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:26.533366+0000","last_undegraded":"2024-09-16T10:28:26.533366+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34119680,"num_objects":8330,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":8330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12138,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2953,"num_bytes_recovered":12095488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7091},{"shards":"0,2","objects":1239}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12158","reported_seq":12315,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.333827+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:26.333827+0000","last_peered":"2024-09-16T10:28:26.333827+0000","last_clean":"2024-09-16T10:28:26.333827+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:26.333827+0000","last_undegraded":"2024-09-16T10:28:26.333827+0000","last_fullsized":"2024-09-16T10:28:26.333827+0000","mapping_epoch":34,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":8304,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8371,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12277","reported_seq":12491,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.375936+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:26.375936+0000","last_peered":"2024-09-16T10:28:26.375936+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:26.375936+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34349080,"num_objects":8387,"num_object_clones":0,"num_object_copies":16774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8334,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2392},{"shards":"2","objects":4066}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12198","reported_seq":12255,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.403612+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:26.403612+0000","last_peered":"2024-09-16T10:28:26.403612+0000","last_clean":"2024-09-16T10:28:26.403612+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:26.403612+0000","last_undegraded":"2024-09-16T10:28:26.403612+0000","last_fullsized":"2024-09-16T10:28:26.403612+0000","mapping_epoch":34,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33849344,"num_objects":8264,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12198,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12179","reported_seq":12336,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.339756+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:26.339756+0000","last_peered":"2024-09-16T10:28:26.339756+0000","last_clean":"2024-09-16T10:28:26.339756+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:26.339756+0000","last_undegraded":"2024-09-16T10:28:26.339756+0000","last_fullsized":"2024-09-16T10:28:26.339756+0000","mapping_epoch":33,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34103296,"num_objects":8326,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8821,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12071","reported_seq":12293,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.472871+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:26.472871+0000","last_peered":"2024-09-16T10:28:26.472871+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:26.472871+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33996800,"num_objects":8300,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4112},{"shards":"0,2","objects":2334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12514","reported_seq":12828,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.473854+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:26.473854+0000","last_peered":"2024-09-16T10:28:26.473854+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:26.473854+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34430976,"num_objects":8406,"num_object_clones":0,"num_object_copies":16812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4073},{"shards":"0,2","objects":2352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12038","reported_seq":12214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.477549+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:26.477549+0000","last_peered":"2024-09-16T10:28:26.477549+0000","last_clean":"2024-09-16T10:28:26.477549+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:26.477549+0000","last_undegraded":"2024-09-16T10:28:26.477549+0000","last_fullsized":"2024-09-16T10:28:26.477549+0000","mapping_epoch":33,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209792,"num_objects":8352,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12101","reported_seq":20605,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:24.906783+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:24.906783+0000","last_peered":"2024-09-16T10:28:24.906783+0000","last_clean":"2024-09-16T10:28:24.906783+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:24.906783+0000","last_undegraded":"2024-09-16T10:28:24.906783+0000","last_fullsized":"2024-09-16T10:28:24.906783+0000","mapping_epoch":43,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34243526,"num_objects":8362,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12099,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12562","reported_seq":12713,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.363870+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:26.363870+0000","last_peered":"2024-09-16T10:28:26.363870+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:26.363870+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34726854,"num_objects":8480,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12560,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4082},{"shards":"0,2","objects":2396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12128","reported_seq":12266,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.310080+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:26.310080+0000","last_peered":"2024-09-16T10:28:26.310080+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:26.310080+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33960902,"num_objects":8293,"num_object_clones":0,"num_object_copies":16586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2361},{"shards":"2","objects":4032}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12225","reported_seq":12336,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.351877+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:26.351877+0000","last_peered":"2024-09-16T10:28:26.351877+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:26.351877+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34206662,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2258},{"shards":"2","objects":4173}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11865","reported_seq":12029,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.527323+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:26.527323+0000","last_peered":"2024-09-16T10:28:26.527323+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:26.527323+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33460707,"num_objects":8170,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11864,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4053},{"shards":"0,2","objects":2312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12451","reported_seq":12635,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.526046+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:26.526046+0000","last_peered":"2024-09-16T10:28:26.526046+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:26.526046+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665897,"num_objects":8466,"num_object_clones":0,"num_object_copies":16932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4154},{"shards":"0,2","objects":2347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12029","reported_seq":12188,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.460893+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:26.460893+0000","last_peered":"2024-09-16T10:28:26.460893+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:26.460893+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223622,"num_objects":8113,"num_object_clones":0,"num_object_copies":16226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3985},{"shards":"0,2","objects":2210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12475","reported_seq":12644,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.462563+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:26.462563+0000","last_peered":"2024-09-16T10:28:26.462563+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:26.462563+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34494348,"num_objects":8425,"num_object_clones":0,"num_object_copies":16850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12472,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4068},{"shards":"0,2","objects":2374}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":546054670,"num_objects":133331,"num_object_clones":0,"num_object_copies":266662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42615,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":133331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172974,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":29290979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1201332224,"data_stored":1201240981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018951,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696604,"kb_used_data":404112,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93675236,"statfs":{"total":96636764160,"available":95923441664,"internally_reserved":0,"allocated":413810688,"data_stored":413552982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.5800000000000001},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.17899999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":9.2059999999999995},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":3.6970000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572708,"kb_used_data":302364,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93799132,"statfs":{"total":96636764160,"available":96050311168,"internally_reserved":0,"allocated":309620736,"data_stored":309290770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":276819626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.39700000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833204,"kb_used_data":468584,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93538636,"statfs":{"total":96636764160,"available":95783563264,"internally_reserved":0,"allocated":479830016,"data_stored":479579996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":373354152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,4,4,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.6679999999999999},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.246}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":7.5069999999999997},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.641}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":478838784,"data_stored":478835171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309223424,"data_stored":309143479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413270016,"data_stored":413262331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:26.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:26.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'12138", 'reported_seq': 18113, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:26.533366+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:26.533366+0000', 'last_peered': '2024-09-16T10:28:26.533366+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:26.533366+0000', 'last_undegraded': '2024-09-16T10:28:26.533366+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'2100", 'ondisk_log_start': "35'2100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34119680, 'num_objects': 8330, 'num_object_clones': 0, 'num_object_copies': 16660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1239, 'num_objects_unfound': 0, 'num_objects_dirty': 8330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12138, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2953, 'num_bytes_recovered': 12095488, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7091}, {'shards': '0,2', 'objects': 1239}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:26.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:26.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'12277", 'reported_seq': 12491, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.375936+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:26.375936+0000', 'last_peered': '2024-09-16T10:28:26.375936+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:26.375936+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'2200", 'ondisk_log_start': "35'2200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34349080, 'num_objects': 8387, 'num_object_clones': 0, 'num_object_copies': 16774, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8387, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8334, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2392}, {'shards': '2', 'objects': 4066}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:26.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:26.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'12071", 'reported_seq': 12293, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.472871+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:26.472871+0000', 'last_peered': '2024-09-16T10:28:26.472871+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:26.472871+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'2000", 'ondisk_log_start': "35'2000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33996800, 'num_objects': 8300, 'num_object_clones': 0, 'num_object_copies': 16600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4190, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8696, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4112}, {'shards': '0,2', 'objects': 2334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:26.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:26.923 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'12514", 'reported_seq': 12828, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.473854+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:26.473854+0000', 'last_peered': '2024-09-16T10:28:26.473854+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:26.473854+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'2500", 'ondisk_log_start': "35'2500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34430976, 'num_objects': 8406, 'num_object_clones': 0, 'num_object_copies': 16812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8756, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 70, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4073}, {'shards': '0,2', 'objects': 2352}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'12562", 'reported_seq': 12713, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.363870+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:26.363870+0000', 'last_peered': '2024-09-16T10:28:26.363870+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:26.363870+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'2500", 'ondisk_log_start': "35'2500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34726854, 'num_objects': 8480, 'num_object_clones': 0, 'num_object_copies': 16960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4400, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12560, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4082}, {'shards': '0,2', 'objects': 2396}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'12128", 'reported_seq': 12266, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.310080+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:26.310080+0000', 'last_peered': '2024-09-16T10:28:26.310080+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:26.310080+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'2100", 'ondisk_log_start': "35'2100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33960902, 'num_objects': 8293, 'num_object_clones': 0, 'num_object_copies': 16586, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12126, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2361}, {'shards': '2', 'objects': 4032}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'12225", 'reported_seq': 12336, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.351877+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:26.351877+0000', 'last_peered': '2024-09-16T10:28:26.351877+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:26.351877+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'2200", 'ondisk_log_start': "35'2200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34206662, 'num_objects': 8353, 'num_object_clones': 0, 'num_object_copies': 16706, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12223, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2258}, {'shards': '2', 'objects': 4173}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'11865", 'reported_seq': 12029, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.527323+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:26.527323+0000', 'last_peered': '2024-09-16T10:28:26.527323+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:26.527323+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'1800", 'ondisk_log_start': "35'1800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33460707, 'num_objects': 8170, 'num_object_clones': 0, 'num_object_copies': 16340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11864, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4053}, {'shards': '0,2', 'objects': 2312}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'12451", 'reported_seq': 12635, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.526046+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:26.526046+0000', 'last_peered': '2024-09-16T10:28:26.526046+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:26.526046+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'2400", 'ondisk_log_start': "35'2400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34665897, 'num_objects': 8466, 'num_object_clones': 0, 'num_object_copies': 16932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12448, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4154}, {'shards': '0,2', 'objects': 2347}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:26.924 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:26.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'12029", 'reported_seq': 12188, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.460893+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:26.460893+0000', 'last_peered': '2024-09-16T10:28:26.460893+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:26.460893+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'2000", 'ondisk_log_start': "35'2000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33223622, 'num_objects': 8113, 'num_object_clones': 0, 'num_object_copies': 16226, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12027, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3985}, {'shards': '0,2', 'objects': 2210}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:26.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:26.925 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'12475", 'reported_seq': 12644, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.462563+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:26.462563+0000', 'last_peered': '2024-09-16T10:28:26.462563+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:26.462563+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'2400", 'ondisk_log_start': "35'2400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34494348, 'num_objects': 8425, 'num_object_clones': 0, 'num_object_copies': 16850, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4360, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12472, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4068}, {'shards': '0,2', 'objects': 2374}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:26.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:27.577 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42615/266666 objects degraded (15.981%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":396,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":133333,"data_bytes":546054689,"bytes_used":2152976384,"bytes_avail":287757316096,"bytes_total":289910292480,"degraded_objects":42615,"degraded_total":266666,"degraded_ratio":0.15980664951662379,"misplaced_objects":1239,"misplaced_total":266666,"misplaced_ratio":0.0046462616156540395,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":140924,"recovering_keys_per_sec":0,"num_objects_recovered":471,"num_bytes_recovered":1691648,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":532},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:27.602 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:27.602 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:28.167 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:28.167 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:28.185 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-16T10:28:26.668503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":546054689,"num_objects":133333,"num_object_clones":0,"num_object_copies":266666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42615,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":133333,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172976,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7493,"num_bytes_recovered":29290998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2102516,"kb_used_data":1175060,"kb_used_omap":12,"kb_used_meta":927411,"kb_avail":281013004,"statfs":{"total":289910292480,"available":287757316096,"internally_reserved":0,"allocated":1203261440,"data_stored":1202423748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":949668866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,6,4,14,7,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26202112,"num_objects":-6397,"num_object_clones":0,"num_object_copies":-12794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-58,"num_objects_misplaced":-413,"num_objects_unfound":0,"num_objects_dirty":-6397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6397,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":471,"num_bytes_recovered":1691648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003902"},"pg_stats":[{"pgid":"2.f","version":"44'12138","reported_seq":18113,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:26.533366+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:26.533366+0000","last_peered":"2024-09-16T10:28:26.533366+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:26.533366+0000","last_undegraded":"2024-09-16T10:28:26.533366+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34119680,"num_objects":8330,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":8330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12138,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2953,"num_bytes_recovered":12095488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7091},{"shards":"0,2","objects":1239}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12158","reported_seq":12315,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.333827+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:26.333827+0000","last_peered":"2024-09-16T10:28:26.333827+0000","last_clean":"2024-09-16T10:28:26.333827+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:26.333827+0000","last_undegraded":"2024-09-16T10:28:26.333827+0000","last_fullsized":"2024-09-16T10:28:26.333827+0000","mapping_epoch":34,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":8304,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8371,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12277","reported_seq":12491,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.375936+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:26.375936+0000","last_peered":"2024-09-16T10:28:26.375936+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:26.375936+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34349080,"num_objects":8387,"num_object_clones":0,"num_object_copies":16774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8334,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2392},{"shards":"2","objects":4066}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12198","reported_seq":12255,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.403612+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:26.403612+0000","last_peered":"2024-09-16T10:28:26.403612+0000","last_clean":"2024-09-16T10:28:26.403612+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:26.403612+0000","last_undegraded":"2024-09-16T10:28:26.403612+0000","last_fullsized":"2024-09-16T10:28:26.403612+0000","mapping_epoch":34,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33849344,"num_objects":8264,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12198,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12179","reported_seq":12336,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.339756+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:26.339756+0000","last_peered":"2024-09-16T10:28:26.339756+0000","last_clean":"2024-09-16T10:28:26.339756+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:26.339756+0000","last_undegraded":"2024-09-16T10:28:26.339756+0000","last_fullsized":"2024-09-16T10:28:26.339756+0000","mapping_epoch":33,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34103296,"num_objects":8326,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8821,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12071","reported_seq":12293,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.472871+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:26.472871+0000","last_peered":"2024-09-16T10:28:26.472871+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:26.472871+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33996800,"num_objects":8300,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4112},{"shards":"0,2","objects":2334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12514","reported_seq":12828,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.473854+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:26.473854+0000","last_peered":"2024-09-16T10:28:26.473854+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:26.473854+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34430976,"num_objects":8406,"num_object_clones":0,"num_object_copies":16812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4073},{"shards":"0,2","objects":2352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12038","reported_seq":12214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.477549+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:26.477549+0000","last_peered":"2024-09-16T10:28:26.477549+0000","last_clean":"2024-09-16T10:28:26.477549+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:26.477549+0000","last_undegraded":"2024-09-16T10:28:26.477549+0000","last_fullsized":"2024-09-16T10:28:26.477549+0000","mapping_epoch":33,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209792,"num_objects":8352,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12101","reported_seq":20605,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:24.906783+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:24.906783+0000","last_peered":"2024-09-16T10:28:24.906783+0000","last_clean":"2024-09-16T10:28:24.906783+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:24.906783+0000","last_undegraded":"2024-09-16T10:28:24.906783+0000","last_fullsized":"2024-09-16T10:28:24.906783+0000","mapping_epoch":43,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34243526,"num_objects":8362,"num_object_clones":0,"num_object_copies":16724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12099,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12562","reported_seq":12713,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.363870+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:26.363870+0000","last_peered":"2024-09-16T10:28:26.363870+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:26.363870+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34726854,"num_objects":8480,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12560,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4082},{"shards":"0,2","objects":2396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12128","reported_seq":12266,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.310080+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:26.310080+0000","last_peered":"2024-09-16T10:28:26.310080+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:26.310080+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33960902,"num_objects":8293,"num_object_clones":0,"num_object_copies":16586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2361},{"shards":"2","objects":4032}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12225","reported_seq":12336,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.351877+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:26.351877+0000","last_peered":"2024-09-16T10:28:26.351877+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:26.351877+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34206662,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2258},{"shards":"2","objects":4173}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11865","reported_seq":12029,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.527323+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:26.527323+0000","last_peered":"2024-09-16T10:28:26.527323+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:26.527323+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33460707,"num_objects":8170,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11864,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4053},{"shards":"0,2","objects":2312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12451","reported_seq":12635,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.526046+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:26.526046+0000","last_peered":"2024-09-16T10:28:26.526046+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:26.526046+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665897,"num_objects":8466,"num_object_clones":0,"num_object_copies":16932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4154},{"shards":"0,2","objects":2347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12029","reported_seq":12188,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.460893+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:26.460893+0000","last_peered":"2024-09-16T10:28:26.460893+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:26.460893+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223622,"num_objects":8113,"num_object_clones":0,"num_object_copies":16226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3985},{"shards":"0,2","objects":2210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12475","reported_seq":12644,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.462563+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:26.462563+0000","last_peered":"2024-09-16T10:28:26.462563+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:26.462563+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34494348,"num_objects":8425,"num_object_clones":0,"num_object_copies":16850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12472,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4068},{"shards":"0,2","objects":2374}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":546054670,"num_objects":133331,"num_object_clones":0,"num_object_copies":266662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42615,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":133331,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172974,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":29290979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1201332224,"data_stored":1201240981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018951,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696604,"kb_used_data":404112,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93675236,"statfs":{"total":96636764160,"available":95923441664,"internally_reserved":0,"allocated":413810688,"data_stored":413552982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.5800000000000001},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.17899999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":9.2059999999999995},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":3.6970000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572708,"kb_used_data":302364,"kb_used_omap":4,"kb_used_meta":270331,"kb_avail":93799132,"statfs":{"total":96636764160,"available":96050311168,"internally_reserved":0,"allocated":309620736,"data_stored":309290770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":276819626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.92100000000000004},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.39700000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833204,"kb_used_data":468584,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93538636,"statfs":{"total":96636764160,"available":95783563264,"internally_reserved":0,"allocated":479830016,"data_stored":479579996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":373354152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,4,4,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.6679999999999999},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.246}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":7.5069999999999997},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.641}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":478838784,"data_stored":478835171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":309223424,"data_stored":309143479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413270016,"data_stored":413262331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:31.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:31.747 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:31.750 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:31.773 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-16T10:28:30.669464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":545485345,"num_objects":133194,"num_object_clones":0,"num_object_copies":266388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42615,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":133194,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173115,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7493,"num_bytes_recovered":29290998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2106072,"kb_used_data":1174520,"kb_used_omap":12,"kb_used_meta":931507,"kb_avail":281009448,"statfs":{"total":289910292480,"available":287753674752,"internally_reserved":0,"allocated":1202708480,"data_stored":1201870788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":953863170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,4,9,7,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18235392,"num_objects":-4452,"num_object_clones":0,"num_object_copies":-8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-23,"num_objects_misplaced":-267,"num_objects_unfound":0,"num_objects_dirty":-4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4452,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1093632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003894"},"pg_stats":[{"pgid":"2.f","version":"44'12138","reported_seq":18113,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:26.533366+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:26.533366+0000","last_peered":"2024-09-16T10:28:26.533366+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:26.533366+0000","last_undegraded":"2024-09-16T10:28:26.533366+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34119680,"num_objects":8330,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":8330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12138,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2953,"num_bytes_recovered":12095488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7091},{"shards":"0,2","objects":1239}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12158","reported_seq":12315,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.333827+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:26.333827+0000","last_peered":"2024-09-16T10:28:26.333827+0000","last_clean":"2024-09-16T10:28:26.333827+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:26.333827+0000","last_undegraded":"2024-09-16T10:28:26.333827+0000","last_fullsized":"2024-09-16T10:28:26.333827+0000","mapping_epoch":34,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":8304,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8371,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12277","reported_seq":12491,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.375936+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:26.375936+0000","last_peered":"2024-09-16T10:28:26.375936+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:26.375936+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34349080,"num_objects":8387,"num_object_clones":0,"num_object_copies":16774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8334,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2392},{"shards":"2","objects":4066}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12198","reported_seq":12255,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.403612+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:26.403612+0000","last_peered":"2024-09-16T10:28:26.403612+0000","last_clean":"2024-09-16T10:28:26.403612+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:26.403612+0000","last_undegraded":"2024-09-16T10:28:26.403612+0000","last_fullsized":"2024-09-16T10:28:26.403612+0000","mapping_epoch":34,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33849344,"num_objects":8264,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12198,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12179","reported_seq":12336,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.339756+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:26.339756+0000","last_peered":"2024-09-16T10:28:26.339756+0000","last_clean":"2024-09-16T10:28:26.339756+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:26.339756+0000","last_undegraded":"2024-09-16T10:28:26.339756+0000","last_fullsized":"2024-09-16T10:28:26.339756+0000","mapping_epoch":33,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34103296,"num_objects":8326,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8821,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12071","reported_seq":12293,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.472871+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:26.472871+0000","last_peered":"2024-09-16T10:28:26.472871+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:26.472871+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33996800,"num_objects":8300,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4112},{"shards":"0,2","objects":2334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12514","reported_seq":12828,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.473854+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:26.473854+0000","last_peered":"2024-09-16T10:28:26.473854+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:26.473854+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34430976,"num_objects":8406,"num_object_clones":0,"num_object_copies":16812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4073},{"shards":"0,2","objects":2352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12038","reported_seq":12214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.477549+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:26.477549+0000","last_peered":"2024-09-16T10:28:26.477549+0000","last_clean":"2024-09-16T10:28:26.477549+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:26.477549+0000","last_undegraded":"2024-09-16T10:28:26.477549+0000","last_fullsized":"2024-09-16T10:28:26.477549+0000","mapping_epoch":33,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209792,"num_objects":8352,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12240","reported_seq":20744,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:29.812226+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:29.812226+0000","last_peered":"2024-09-16T10:28:29.812226+0000","last_clean":"2024-09-16T10:28:29.812226+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:29.812226+0000","last_undegraded":"2024-09-16T10:28:29.812226+0000","last_fullsized":"2024-09-16T10:28:29.812226+0000","mapping_epoch":43,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33674182,"num_objects":8223,"num_object_clones":0,"num_object_copies":16446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12238,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12562","reported_seq":12713,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.363870+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:26.363870+0000","last_peered":"2024-09-16T10:28:26.363870+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:26.363870+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34726854,"num_objects":8480,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12560,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4082},{"shards":"0,2","objects":2396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12128","reported_seq":12266,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.310080+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:26.310080+0000","last_peered":"2024-09-16T10:28:26.310080+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:26.310080+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33960902,"num_objects":8293,"num_object_clones":0,"num_object_copies":16586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2361},{"shards":"2","objects":4032}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12225","reported_seq":12336,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.351877+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:26.351877+0000","last_peered":"2024-09-16T10:28:26.351877+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:26.351877+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34206662,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2258},{"shards":"2","objects":4173}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11865","reported_seq":12029,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.527323+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:26.527323+0000","last_peered":"2024-09-16T10:28:26.527323+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:26.527323+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33460707,"num_objects":8170,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11864,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4053},{"shards":"0,2","objects":2312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12451","reported_seq":12635,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.526046+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:26.526046+0000","last_peered":"2024-09-16T10:28:26.526046+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:26.526046+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665897,"num_objects":8466,"num_object_clones":0,"num_object_copies":16932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4154},{"shards":"0,2","objects":2347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12029","reported_seq":12188,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.460893+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:26.460893+0000","last_peered":"2024-09-16T10:28:26.460893+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:26.460893+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223622,"num_objects":8113,"num_object_clones":0,"num_object_copies":16226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3985},{"shards":"0,2","objects":2210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12475","reported_seq":12644,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.462563+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:26.462563+0000","last_peered":"2024-09-16T10:28:26.462563+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:26.462563+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34494348,"num_objects":8425,"num_object_clones":0,"num_object_copies":16850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12472,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4068},{"shards":"0,2","objects":2374}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":545485326,"num_objects":133192,"num_object_clones":0,"num_object_copies":266384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42615,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":133192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173113,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":29290979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1200816128,"data_stored":1200724885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018951,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696604,"kb_used_data":404112,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93675236,"statfs":{"total":96636764160,"available":95923441664,"internally_reserved":0,"allocated":413810688,"data_stored":413552982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.5800000000000001},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.17899999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":9.2059999999999995},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":3.6970000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576264,"kb_used_data":301824,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93795576,"statfs":{"total":96636764160,"available":96046669824,"internally_reserved":0,"allocated":309067776,"data_stored":308737810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":281013930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":8.0190000000000001},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.4530000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.312},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.6850000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833204,"kb_used_data":468584,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93538636,"statfs":{"total":96636764160,"available":95783563264,"internally_reserved":0,"allocated":479830016,"data_stored":479579996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":373354152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,4,4,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.6679999999999999},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.246}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":7.5069999999999997},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.641}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":478838784,"data_stored":478835171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308707328,"data_stored":308627383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413270016,"data_stored":413262331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:31.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:32.292 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:32.295 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:32.321 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-16T10:28:30.669464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":545485345,"num_objects":133194,"num_object_clones":0,"num_object_copies":266388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42615,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":133194,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173115,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7493,"num_bytes_recovered":29290998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2106072,"kb_used_data":1174520,"kb_used_omap":12,"kb_used_meta":931507,"kb_avail":281009448,"statfs":{"total":289910292480,"available":287753674752,"internally_reserved":0,"allocated":1202708480,"data_stored":1201870788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":953863170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,5,4,9,7,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18235392,"num_objects":-4452,"num_object_clones":0,"num_object_copies":-8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-23,"num_objects_misplaced":-267,"num_objects_unfound":0,"num_objects_dirty":-4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4452,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1093632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003894"},"pg_stats":[{"pgid":"2.f","version":"44'12138","reported_seq":18113,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:26.533366+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:26.533366+0000","last_peered":"2024-09-16T10:28:26.533366+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:26.533366+0000","last_undegraded":"2024-09-16T10:28:26.533366+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34119680,"num_objects":8330,"num_object_clones":0,"num_object_copies":16660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":8330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12138,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2953,"num_bytes_recovered":12095488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7091},{"shards":"0,2","objects":1239}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12158","reported_seq":12315,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.333827+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:26.333827+0000","last_peered":"2024-09-16T10:28:26.333827+0000","last_clean":"2024-09-16T10:28:26.333827+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:26.333827+0000","last_undegraded":"2024-09-16T10:28:26.333827+0000","last_fullsized":"2024-09-16T10:28:26.333827+0000","mapping_epoch":34,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":8304,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8371,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12277","reported_seq":12491,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.375936+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:26.375936+0000","last_peered":"2024-09-16T10:28:26.375936+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:26.375936+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34349080,"num_objects":8387,"num_object_clones":0,"num_object_copies":16774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8334,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2392},{"shards":"2","objects":4066}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12198","reported_seq":12255,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.403612+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:26.403612+0000","last_peered":"2024-09-16T10:28:26.403612+0000","last_clean":"2024-09-16T10:28:26.403612+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:26.403612+0000","last_undegraded":"2024-09-16T10:28:26.403612+0000","last_fullsized":"2024-09-16T10:28:26.403612+0000","mapping_epoch":34,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33849344,"num_objects":8264,"num_object_clones":0,"num_object_copies":16528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12198,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12179","reported_seq":12336,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.339756+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:26.339756+0000","last_peered":"2024-09-16T10:28:26.339756+0000","last_clean":"2024-09-16T10:28:26.339756+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:26.339756+0000","last_undegraded":"2024-09-16T10:28:26.339756+0000","last_fullsized":"2024-09-16T10:28:26.339756+0000","mapping_epoch":33,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34103296,"num_objects":8326,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8821,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12071","reported_seq":12293,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.472871+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:26.472871+0000","last_peered":"2024-09-16T10:28:26.472871+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:26.472871+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33996800,"num_objects":8300,"num_object_clones":0,"num_object_copies":16600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8696,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4112},{"shards":"0,2","objects":2334}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12514","reported_seq":12828,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.473854+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:26.473854+0000","last_peered":"2024-09-16T10:28:26.473854+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:26.473854+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34430976,"num_objects":8406,"num_object_clones":0,"num_object_copies":16812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4073},{"shards":"0,2","objects":2352}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12038","reported_seq":12214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:26.477549+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:26.477549+0000","last_peered":"2024-09-16T10:28:26.477549+0000","last_clean":"2024-09-16T10:28:26.477549+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:26.477549+0000","last_undegraded":"2024-09-16T10:28:26.477549+0000","last_fullsized":"2024-09-16T10:28:26.477549+0000","mapping_epoch":33,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209792,"num_objects":8352,"num_object_clones":0,"num_object_copies":16704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7841,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12240","reported_seq":20744,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:29.812226+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:29.812226+0000","last_peered":"2024-09-16T10:28:29.812226+0000","last_clean":"2024-09-16T10:28:29.812226+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:29.812226+0000","last_undegraded":"2024-09-16T10:28:29.812226+0000","last_fullsized":"2024-09-16T10:28:29.812226+0000","mapping_epoch":43,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33674182,"num_objects":8223,"num_object_clones":0,"num_object_copies":16446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12238,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12562","reported_seq":12713,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.363870+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:26.363870+0000","last_peered":"2024-09-16T10:28:26.363870+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:26.363870+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34726854,"num_objects":8480,"num_object_clones":0,"num_object_copies":16960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12560,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4082},{"shards":"0,2","objects":2396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12128","reported_seq":12266,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.310080+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:26.310080+0000","last_peered":"2024-09-16T10:28:26.310080+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:26.310080+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33960902,"num_objects":8293,"num_object_clones":0,"num_object_copies":16586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2361},{"shards":"2","objects":4032}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12225","reported_seq":12336,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.351877+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:26.351877+0000","last_peered":"2024-09-16T10:28:26.351877+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:26.351877+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34206662,"num_objects":8353,"num_object_clones":0,"num_object_copies":16706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12223,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2258},{"shards":"2","objects":4173}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'11865","reported_seq":12029,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.527323+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:26.527323+0000","last_peered":"2024-09-16T10:28:26.527323+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:26.527323+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'1800","ondisk_log_start":"35'1800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33460707,"num_objects":8170,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11864,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4053},{"shards":"0,2","objects":2312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12451","reported_seq":12635,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.526046+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:26.526046+0000","last_peered":"2024-09-16T10:28:26.526046+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:26.526046+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665897,"num_objects":8466,"num_object_clones":0,"num_object_copies":16932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12448,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4154},{"shards":"0,2","objects":2347}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12029","reported_seq":12188,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.460893+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:26.460893+0000","last_peered":"2024-09-16T10:28:26.460893+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:26.460893+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33223622,"num_objects":8113,"num_object_clones":0,"num_object_copies":16226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12027,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3985},{"shards":"0,2","objects":2210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12475","reported_seq":12644,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:26.462563+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:26.462563+0000","last_peered":"2024-09-16T10:28:26.462563+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:26.462563+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34494348,"num_objects":8425,"num_object_clones":0,"num_object_copies":16850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12472,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4068},{"shards":"0,2","objects":2374}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":545485326,"num_objects":133192,"num_object_clones":0,"num_object_copies":266384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42615,"num_objects_misplaced":1239,"num_objects_unfound":0,"num_objects_dirty":133192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173113,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7492,"num_bytes_recovered":29290979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1200816128,"data_stored":1200724885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018951,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696604,"kb_used_data":404112,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93675236,"statfs":{"total":96636764160,"available":95923441664,"internally_reserved":0,"allocated":413810688,"data_stored":413552982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.5800000000000001},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.17899999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":9.2059999999999995},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":3.6970000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576264,"kb_used_data":301824,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93795576,"statfs":{"total":96636764160,"available":96046669824,"internally_reserved":0,"allocated":309067776,"data_stored":308737810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":281013930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":8.0190000000000001},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.4530000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.312},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.6850000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215179,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833204,"kb_used_data":468584,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93538636,"statfs":{"total":96636764160,"available":95783563264,"internally_reserved":0,"allocated":479830016,"data_stored":479579996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":373354152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,4,4,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.6679999999999999},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.246}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":7.5069999999999997},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.641}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":478838784,"data_stored":478835171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308707328,"data_stored":308627383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":413270016,"data_stored":413262331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:32.323 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:32.323 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'12138", 'reported_seq': 18113, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:26.533366+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:26.533366+0000', 'last_peered': '2024-09-16T10:28:26.533366+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:26.533366+0000', 'last_undegraded': '2024-09-16T10:28:26.533366+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'2100", 'ondisk_log_start': "35'2100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34119680, 'num_objects': 8330, 'num_object_clones': 0, 'num_object_copies': 16660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1239, 'num_objects_unfound': 0, 'num_objects_dirty': 8330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12138, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2953, 'num_bytes_recovered': 12095488, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7091}, {'shards': '0,2', 'objects': 1239}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:32.323 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:32.323 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'12277", 'reported_seq': 12491, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.375936+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:26.375936+0000', 'last_peered': '2024-09-16T10:28:26.375936+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:26.375936+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'2200", 'ondisk_log_start': "35'2200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34349080, 'num_objects': 8387, 'num_object_clones': 0, 'num_object_copies': 16774, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8387, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8334, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2392}, {'shards': '2', 'objects': 4066}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:32.323 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:32.323 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'12071", 'reported_seq': 12293, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.472871+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:26.472871+0000', 'last_peered': '2024-09-16T10:28:26.472871+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:26.472871+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'2000", 'ondisk_log_start': "35'2000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33996800, 'num_objects': 8300, 'num_object_clones': 0, 'num_object_copies': 16600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4190, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8696, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4112}, {'shards': '0,2', 'objects': 2334}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:32.323 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:32.324 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'12514", 'reported_seq': 12828, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.473854+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:26.473854+0000', 'last_peered': '2024-09-16T10:28:26.473854+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:26.473854+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'2500", 'ondisk_log_start': "35'2500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34430976, 'num_objects': 8406, 'num_object_clones': 0, 'num_object_copies': 16812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8406, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8756, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 70, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4073}, {'shards': '0,2', 'objects': 2352}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:32.324 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:32.324 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'12562", 'reported_seq': 12713, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.363870+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:26.363870+0000', 'last_peered': '2024-09-16T10:28:26.363870+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:26.363870+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'2500", 'ondisk_log_start': "35'2500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34726854, 'num_objects': 8480, 'num_object_clones': 0, 'num_object_copies': 16960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4400, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12560, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4082}, {'shards': '0,2', 'objects': 2396}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:32.324 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:32.324 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'12128", 'reported_seq': 12266, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.310080+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:26.310080+0000', 'last_peered': '2024-09-16T10:28:26.310080+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:26.310080+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'2100", 'ondisk_log_start': "35'2100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33960902, 'num_objects': 8293, 'num_object_clones': 0, 'num_object_copies': 16586, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12126, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2361}, {'shards': '2', 'objects': 4032}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:32.324 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:32.324 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'12225", 'reported_seq': 12336, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.351877+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:26.351877+0000', 'last_peered': '2024-09-16T10:28:26.351877+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:26.351877+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'2200", 'ondisk_log_start': "35'2200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34206662, 'num_objects': 8353, 'num_object_clones': 0, 'num_object_copies': 16706, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12223, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2258}, {'shards': '2', 'objects': 4173}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:32.324 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:32.324 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'11865", 'reported_seq': 12029, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.527323+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:26.527323+0000', 'last_peered': '2024-09-16T10:28:26.527323+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:26.527323+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'1800", 'ondisk_log_start': "35'1800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33460707, 'num_objects': 8170, 'num_object_clones': 0, 'num_object_copies': 16340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8170, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11864, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4053}, {'shards': '0,2', 'objects': 2312}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:32.325 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:32.325 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'12451", 'reported_seq': 12635, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.526046+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:26.526046+0000', 'last_peered': '2024-09-16T10:28:26.526046+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:26.526046+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'2400", 'ondisk_log_start': "35'2400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34665897, 'num_objects': 8466, 'num_object_clones': 0, 'num_object_copies': 16932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4315, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12448, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 28, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4154}, {'shards': '0,2', 'objects': 2347}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:32.325 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:32.325 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'12029", 'reported_seq': 12188, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.460893+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:26.460893+0000', 'last_peered': '2024-09-16T10:28:26.460893+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:26.460893+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'2000", 'ondisk_log_start': "35'2000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33223622, 'num_objects': 8113, 'num_object_clones': 0, 'num_object_copies': 16226, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12027, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 41, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3985}, {'shards': '0,2', 'objects': 2210}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:32.325 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:32.325 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'12475", 'reported_seq': 12644, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:26.462563+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:26.462563+0000', 'last_peered': '2024-09-16T10:28:26.462563+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:26.462563+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'2400", 'ondisk_log_start': "35'2400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34494348, 'num_objects': 8425, 'num_object_clones': 0, 'num_object_copies': 16850, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4360, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12472, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 43, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 4068}, {'shards': '0,2', 'objects': 2374}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:32.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:33.008 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:33.009 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42591/262302 objects degraded (16.237%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":401,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":131151,"data_bytes":537117217,"bytes_used":2124664832,"bytes_avail":287785627648,"bytes_total":289910292480,"degraded_objects":42591,"degraded_total":262302,"degraded_ratio":0.16237390488825856,"misplaced_objects":1106,"misplaced_total":262302,"misplaced_ratio":0.0042165137894488031,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":136488,"recovering_keys_per_sec":0,"num_objects_recovered":447,"num_bytes_recovered":1638400,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":530},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:33.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:33.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:33.592 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:33.617 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-16T10:28:32.669951+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":537117217,"num_objects":131151,"num_object_clones":0,"num_object_copies":262302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42591,"num_objects_misplaced":1106,"num_objects_unfound":0,"num_objects_dirty":131151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175158,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7650,"num_bytes_recovered":29835766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160794,"ondisk_log_size":160794,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2074868,"kb_used_data":1163792,"kb_used_omap":12,"kb_used_meta":911027,"kb_avail":281040652,"statfs":{"total":289910292480,"available":287785627648,"internally_reserved":0,"allocated":1191723008,"data_stored":1190885316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":932891650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,1,2,5,9,1,0,2,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26079232,"num_objects":-6367,"num_object_clones":0,"num_object_copies":-12734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-47,"num_objects_misplaced":-400,"num_objects_unfound":0,"num_objects_dirty":-6367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6367,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":447,"num_bytes_recovered":1638400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003931"},"pg_stats":[{"pgid":"2.f","version":"44'12268","reported_seq":18509,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:31.598017+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:31.598017+0000","last_peered":"2024-09-16T10:28:31.598017+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:31.598017+0000","last_undegraded":"2024-09-16T10:28:31.598017+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33587200,"num_objects":8200,"num_object_clones":0,"num_object_copies":16400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1106,"num_objects_unfound":0,"num_objects_dirty":8200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12268,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3086,"num_bytes_recovered":12640256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7094},{"shards":"0,2","objects":1106}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12282","reported_seq":12440,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:31.417138+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:31.417138+0000","last_peered":"2024-09-16T10:28:31.417138+0000","last_clean":"2024-09-16T10:28:31.417138+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:31.417138+0000","last_undegraded":"2024-09-16T10:28:31.417138+0000","last_fullsized":"2024-09-16T10:28:31.417138+0000","mapping_epoch":34,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33505280,"num_objects":8180,"num_object_clones":0,"num_object_copies":16360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8495,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12406","reported_seq":12621,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:31.213079+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:31.213079+0000","last_peered":"2024-09-16T10:28:31.213079+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:31.213079+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33820696,"num_objects":8258,"num_object_clones":0,"num_object_copies":16516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8258,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8463,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2263},{"shards":"2","objects":3937}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12324","reported_seq":12381,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:31.318878+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:31.318878+0000","last_peered":"2024-09-16T10:28:31.318878+0000","last_clean":"2024-09-16T10:28:31.318878+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:31.318878+0000","last_undegraded":"2024-09-16T10:28:31.318878+0000","last_fullsized":"2024-09-16T10:28:31.318878+0000","mapping_epoch":34,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33333248,"num_objects":8138,"num_object_clones":0,"num_object_copies":16276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12324,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12314","reported_seq":12470,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:31.590642+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:31.590642+0000","last_peered":"2024-09-16T10:28:31.590642+0000","last_clean":"2024-09-16T10:28:31.590642+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:31.590642+0000","last_undegraded":"2024-09-16T10:28:31.590642+0000","last_fullsized":"2024-09-16T10:28:31.590642+0000","mapping_epoch":33,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33550336,"num_objects":8191,"num_object_clones":0,"num_object_copies":16382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8956,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12194","reported_seq":12418,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:31.579104+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:31.579104+0000","last_peered":"2024-09-16T10:28:31.579104+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:31.579104+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33492992,"num_objects":8177,"num_object_clones":0,"num_object_copies":16354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8819,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3990},{"shards":"0,2","objects":2211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12644","reported_seq":12964,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:31.545236+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:31.545236+0000","last_peered":"2024-09-16T10:28:31.545236+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:31.545236+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33898496,"num_objects":8276,"num_object_clones":0,"num_object_copies":16552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8886,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3946},{"shards":"0,2","objects":2222}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12186","reported_seq":12362,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:31.581517+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:31.581517+0000","last_peered":"2024-09-16T10:28:31.581517+0000","last_clean":"2024-09-16T10:28:31.581517+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:31.581517+0000","last_undegraded":"2024-09-16T10:28:31.581517+0000","last_fullsized":"2024-09-16T10:28:31.581517+0000","mapping_epoch":33,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33603584,"num_objects":8204,"num_object_clones":0,"num_object_copies":16408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7989,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12240","reported_seq":20744,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:29.812226+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:29.812226+0000","last_peered":"2024-09-16T10:28:29.812226+0000","last_clean":"2024-09-16T10:28:29.812226+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:29.812226+0000","last_undegraded":"2024-09-16T10:28:29.812226+0000","last_fullsized":"2024-09-16T10:28:29.812226+0000","mapping_epoch":43,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33674182,"num_objects":8223,"num_object_clones":0,"num_object_copies":16446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12238,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12705","reported_seq":12868,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:31.539221+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:31.539221+0000","last_peered":"2024-09-16T10:28:31.539221+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:31.539221+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34141126,"num_objects":8337,"num_object_clones":0,"num_object_copies":16674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12703,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3945},{"shards":"0,2","objects":2253}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12284","reported_seq":12420,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:31.403893+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:31.403893+0000","last_peered":"2024-09-16T10:28:31.403893+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:31.403893+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33321926,"num_objects":8137,"num_object_clones":0,"num_object_copies":16274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2205},{"shards":"2","objects":3876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12364","reported_seq":12475,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:31.414773+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:31.414773+0000","last_peered":"2024-09-16T10:28:31.414773+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:31.414773+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33637318,"num_objects":8214,"num_object_clones":0,"num_object_copies":16428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12362,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2119},{"shards":"2","objects":4034}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12006","reported_seq":12176,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:31.589934+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:31.589934+0000","last_peered":"2024-09-16T10:28:31.589934+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:31.589934+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2000","ondisk_log_start":"35'2000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32883171,"num_objects":8029,"num_object_clones":0,"num_object_copies":16058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3915},{"shards":"0,2","objects":2171}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12594","reported_seq":12786,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:31.563566+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:31.563566+0000","last_peered":"2024-09-16T10:28:31.563566+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:31.563566+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34080169,"num_objects":8323,"num_object_clones":0,"num_object_copies":16646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12591,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":4015},{"shards":"0,2","objects":2204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12151","reported_seq":12316,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:31.571812+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:31.571812+0000","last_peered":"2024-09-16T10:28:31.571812+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:31.571812+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32723910,"num_objects":7991,"num_object_clones":0,"num_object_copies":15982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12149,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3866},{"shards":"0,2","objects":2088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12629","reported_seq":12804,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:31.492358+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:31.492358+0000","last_peered":"2024-09-16T10:28:31.492358+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:31.492358+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33863564,"num_objects":8271,"num_object_clones":0,"num_object_copies":16542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12626,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3918},{"shards":"0,2","objects":2220}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":537117198,"num_objects":131149,"num_object_clones":0,"num_object_copies":262298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42591,"num_objects_misplaced":1106,"num_objects_unfound":0,"num_objects_dirty":131149,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175156,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7649,"num_bytes_recovered":29835747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1189744640,"data_stored":1189653397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160791,"ondisk_log_size":160791,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018952,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679888,"kb_used_data":399688,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93691952,"statfs":{"total":96636764160,"available":95940558848,"internally_reserved":0,"allocated":409280512,"data_stored":409022806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":286912176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,4,1,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.6789999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.2509999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":0.27800000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691896,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576264,"kb_used_data":301824,"kb_used_omap":4,"kb_used_meta":274427,"kb_avail":93795576,"statfs":{"total":96636764160,"available":96046669824,"internally_reserved":0,"allocated":309067776,"data_stored":308737810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":281013930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":8.0190000000000001},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.4530000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.312},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.6850000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215180,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":818716,"kb_used_data":462280,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93553124,"statfs":{"total":96636764160,"available":95798398976,"internally_reserved":0,"allocated":473374720,"data_stored":473124700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":364965544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,2,5,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.61699999999999999},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":0.44800000000000001},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.4359999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":472244224,"data_stored":472240611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308707328,"data_stored":308627383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":408793088,"data_stored":408785403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:36.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:37.169 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:37.170 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:37.190 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-16T10:28:36.670966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":528319009,"num_objects":129003,"num_object_clones":0,"num_object_copies":258006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42566,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":129003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177306,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":30409206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160642,"ondisk_log_size":160642,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2063876,"kb_used_data":1152808,"kb_used_omap":12,"kb_used_meta":911027,"kb_avail":281051644,"statfs":{"total":289910292480,"available":287796883456,"internally_reserved":0,"allocated":1180475392,"data_stored":1179637700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":932891650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,5,9,2,3,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26570752,"num_objects":-6487,"num_object_clones":0,"num_object_copies":-12974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-58,"num_objects_misplaced":-415,"num_objects_unfound":0,"num_objects_dirty":-6487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":1699840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002915"},"pg_stats":[{"pgid":"2.f","version":"44'12406","reported_seq":18928,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:36.620627+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:36.620627+0000","last_peered":"2024-09-16T10:28:36.620627+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:36.620627+0000","last_undegraded":"2024-09-16T10:28:36.620627+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33021952,"num_objects":8062,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":8062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3226,"num_bytes_recovered":13213696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7096},{"shards":"0,2","objects":966}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12405","reported_seq":12563,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.400243+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:36.400243+0000","last_peered":"2024-09-16T10:28:36.400243+0000","last_clean":"2024-09-16T10:28:36.400243+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:36.400243+0000","last_undegraded":"2024-09-16T10:28:36.400243+0000","last_fullsized":"2024-09-16T10:28:36.400243+0000","mapping_epoch":34,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33001472,"num_objects":8057,"num_object_clones":0,"num_object_copies":16114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12549","reported_seq":12764,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.451366+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:36.451366+0000","last_peered":"2024-09-16T10:28:36.451366+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:36.451366+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33234968,"num_objects":8115,"num_object_clones":0,"num_object_copies":16230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8606,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2120},{"shards":"2","objects":3794}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12461","reported_seq":12518,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.425716+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:36.425716+0000","last_peered":"2024-09-16T10:28:36.425716+0000","last_clean":"2024-09-16T10:28:36.425716+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:36.425716+0000","last_undegraded":"2024-09-16T10:28:36.425716+0000","last_fullsized":"2024-09-16T10:28:36.425716+0000","mapping_epoch":34,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32772096,"num_objects":8001,"num_object_clones":0,"num_object_copies":16002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12435","reported_seq":12592,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.583534+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:36.583534+0000","last_peered":"2024-09-16T10:28:36.583534+0000","last_clean":"2024-09-16T10:28:36.583534+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:36.583534+0000","last_undegraded":"2024-09-16T10:28:36.583534+0000","last_fullsized":"2024-09-16T10:28:36.583534+0000","mapping_epoch":33,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33054720,"num_objects":8070,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12315","reported_seq":12545,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.509330+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:36.509330+0000","last_peered":"2024-09-16T10:28:36.509330+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:36.509330+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32997376,"num_objects":8056,"num_object_clones":0,"num_object_copies":16112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8940,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3872},{"shards":"0,2","objects":2090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12780","reported_seq":13100,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.621142+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:36.621142+0000","last_peered":"2024-09-16T10:28:36.621142+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:36.621142+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33341440,"num_objects":8140,"num_object_clones":0,"num_object_copies":16280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3810},{"shards":"0,2","objects":2086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12331","reported_seq":12507,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.613393+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:36.613393+0000","last_peered":"2024-09-16T10:28:36.613393+0000","last_clean":"2024-09-16T10:28:36.613393+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:36.613393+0000","last_undegraded":"2024-09-16T10:28:36.613393+0000","last_fullsized":"2024-09-16T10:28:36.613393+0000","mapping_epoch":33,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33009664,"num_objects":8059,"num_object_clones":0,"num_object_copies":16118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8134,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12381","reported_seq":20885,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:35.072820+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:35.072820+0000","last_peered":"2024-09-16T10:28:35.072820+0000","last_clean":"2024-09-16T10:28:35.072820+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:35.072820+0000","last_undegraded":"2024-09-16T10:28:35.072820+0000","last_fullsized":"2024-09-16T10:28:35.072820+0000","mapping_epoch":43,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33096646,"num_objects":8082,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12379,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12866","reported_seq":13037,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.567772+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:36.567772+0000","last_peered":"2024-09-16T10:28:36.567772+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:36.567772+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33481670,"num_objects":8176,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12864,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3788},{"shards":"0,2","objects":2092}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12405","reported_seq":12542,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.458179+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:36.458179+0000","last_peered":"2024-09-16T10:28:36.458179+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:36.458179+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32826310,"num_objects":8016,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12403,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2084},{"shards":"2","objects":3755}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12501","reported_seq":12612,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.337063+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:36.337063+0000","last_peered":"2024-09-16T10:28:36.337063+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:36.337063+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076166,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1982},{"shards":"2","objects":3897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12135","reported_seq":12320,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.629678+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:36.629678+0000","last_peered":"2024-09-16T10:28:36.629678+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:36.629678+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32354787,"num_objects":7900,"num_object_clones":0,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12134,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3794},{"shards":"0,2","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12734","reported_seq":12936,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.535968+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:36.535968+0000","last_peered":"2024-09-16T10:28:36.535968+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:36.535968+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33506729,"num_objects":8183,"num_object_clones":0,"num_object_copies":16366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12731,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3880},{"shards":"0,2","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12263","reported_seq":12430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.572420+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:36.572420+0000","last_peered":"2024-09-16T10:28:36.572420+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:36.572420+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32265158,"num_objects":7879,"num_object_clones":0,"num_object_copies":15758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12261,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3755},{"shards":"0,2","objects":1976}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12772","reported_seq":12957,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.588679+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:36.588679+0000","last_peered":"2024-09-16T10:28:36.588679+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:36.588679+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33277836,"num_objects":8128,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12769,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3779},{"shards":"0,2","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":528318990,"num_objects":129001,"num_object_clones":0,"num_object_copies":258002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42566,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":129001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177304,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7814,"num_bytes_recovered":30409187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1178304512,"data_stored":1178213269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160639,"ondisk_log_size":160639,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018953,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679792,"kb_used_data":395496,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93692048,"statfs":{"total":96636764160,"available":95940657152,"internally_reserved":0,"allocated":404987904,"data_stored":404730198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.248},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":2.2599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691898,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567640,"kb_used_data":301388,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93804200,"statfs":{"total":96636764160,"available":96055500800,"internally_reserved":0,"allocated":308621312,"data_stored":308291346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":272625322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.143}]}]},{"osd":0,"up_from":24,"seq":103079215181,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":816444,"kb_used_data":455924,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93555396,"statfs":{"total":96636764160,"available":95800725504,"internally_reserved":0,"allocated":466866176,"data_stored":466616156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.88},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":3.6579999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.0269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":465645568,"data_stored":465641955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308244480,"data_stored":308164535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404414464,"data_stored":404406779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:37.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:37.738 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:37.739 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:37.766 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-16T10:28:36.670966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":528319009,"num_objects":129003,"num_object_clones":0,"num_object_copies":258006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42566,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":129003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177306,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":30409206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160642,"ondisk_log_size":160642,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2063876,"kb_used_data":1152808,"kb_used_omap":12,"kb_used_meta":911027,"kb_avail":281051644,"statfs":{"total":289910292480,"available":287796883456,"internally_reserved":0,"allocated":1180475392,"data_stored":1179637700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":932891650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,5,9,2,3,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26570752,"num_objects":-6487,"num_object_clones":0,"num_object_copies":-12974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-58,"num_objects_misplaced":-415,"num_objects_unfound":0,"num_objects_dirty":-6487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6487,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":1699840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002915"},"pg_stats":[{"pgid":"2.f","version":"44'12406","reported_seq":18928,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:36.620627+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:36.620627+0000","last_peered":"2024-09-16T10:28:36.620627+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:36.620627+0000","last_undegraded":"2024-09-16T10:28:36.620627+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33021952,"num_objects":8062,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":8062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3226,"num_bytes_recovered":13213696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7096},{"shards":"0,2","objects":966}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12405","reported_seq":12563,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.400243+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:36.400243+0000","last_peered":"2024-09-16T10:28:36.400243+0000","last_clean":"2024-09-16T10:28:36.400243+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:36.400243+0000","last_undegraded":"2024-09-16T10:28:36.400243+0000","last_fullsized":"2024-09-16T10:28:36.400243+0000","mapping_epoch":34,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33001472,"num_objects":8057,"num_object_clones":0,"num_object_copies":16114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12549","reported_seq":12764,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.451366+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:36.451366+0000","last_peered":"2024-09-16T10:28:36.451366+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:36.451366+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33234968,"num_objects":8115,"num_object_clones":0,"num_object_copies":16230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8606,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2120},{"shards":"2","objects":3794}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12461","reported_seq":12518,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.425716+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:36.425716+0000","last_peered":"2024-09-16T10:28:36.425716+0000","last_clean":"2024-09-16T10:28:36.425716+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:36.425716+0000","last_undegraded":"2024-09-16T10:28:36.425716+0000","last_fullsized":"2024-09-16T10:28:36.425716+0000","mapping_epoch":34,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32772096,"num_objects":8001,"num_object_clones":0,"num_object_copies":16002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12435","reported_seq":12592,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.583534+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:36.583534+0000","last_peered":"2024-09-16T10:28:36.583534+0000","last_clean":"2024-09-16T10:28:36.583534+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:36.583534+0000","last_undegraded":"2024-09-16T10:28:36.583534+0000","last_fullsized":"2024-09-16T10:28:36.583534+0000","mapping_epoch":33,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33054720,"num_objects":8070,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12315","reported_seq":12545,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.509330+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:36.509330+0000","last_peered":"2024-09-16T10:28:36.509330+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:36.509330+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32997376,"num_objects":8056,"num_object_clones":0,"num_object_copies":16112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8940,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3872},{"shards":"0,2","objects":2090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12780","reported_seq":13100,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.621142+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:36.621142+0000","last_peered":"2024-09-16T10:28:36.621142+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:36.621142+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33341440,"num_objects":8140,"num_object_clones":0,"num_object_copies":16280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3810},{"shards":"0,2","objects":2086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12331","reported_seq":12507,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.613393+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:36.613393+0000","last_peered":"2024-09-16T10:28:36.613393+0000","last_clean":"2024-09-16T10:28:36.613393+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:36.613393+0000","last_undegraded":"2024-09-16T10:28:36.613393+0000","last_fullsized":"2024-09-16T10:28:36.613393+0000","mapping_epoch":33,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33009664,"num_objects":8059,"num_object_clones":0,"num_object_copies":16118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8134,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12381","reported_seq":20885,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:35.072820+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:35.072820+0000","last_peered":"2024-09-16T10:28:35.072820+0000","last_clean":"2024-09-16T10:28:35.072820+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:35.072820+0000","last_undegraded":"2024-09-16T10:28:35.072820+0000","last_fullsized":"2024-09-16T10:28:35.072820+0000","mapping_epoch":43,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33096646,"num_objects":8082,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12379,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12866","reported_seq":13037,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.567772+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:36.567772+0000","last_peered":"2024-09-16T10:28:36.567772+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:36.567772+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33481670,"num_objects":8176,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12864,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3788},{"shards":"0,2","objects":2092}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12405","reported_seq":12542,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.458179+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:36.458179+0000","last_peered":"2024-09-16T10:28:36.458179+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:36.458179+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32826310,"num_objects":8016,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12403,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2084},{"shards":"2","objects":3755}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12501","reported_seq":12612,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.337063+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:36.337063+0000","last_peered":"2024-09-16T10:28:36.337063+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:36.337063+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076166,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1982},{"shards":"2","objects":3897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12135","reported_seq":12320,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.629678+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:36.629678+0000","last_peered":"2024-09-16T10:28:36.629678+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:36.629678+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32354787,"num_objects":7900,"num_object_clones":0,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12134,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3794},{"shards":"0,2","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12734","reported_seq":12936,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.535968+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:36.535968+0000","last_peered":"2024-09-16T10:28:36.535968+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:36.535968+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33506729,"num_objects":8183,"num_object_clones":0,"num_object_copies":16366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12731,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3880},{"shards":"0,2","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12263","reported_seq":12430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.572420+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:36.572420+0000","last_peered":"2024-09-16T10:28:36.572420+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:36.572420+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32265158,"num_objects":7879,"num_object_clones":0,"num_object_copies":15758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12261,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3755},{"shards":"0,2","objects":1976}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12772","reported_seq":12957,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.588679+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:36.588679+0000","last_peered":"2024-09-16T10:28:36.588679+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:36.588679+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33277836,"num_objects":8128,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12769,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3779},{"shards":"0,2","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":528318990,"num_objects":129001,"num_object_clones":0,"num_object_copies":258002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42566,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":129001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177304,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7814,"num_bytes_recovered":30409187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1178304512,"data_stored":1178213269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160639,"ondisk_log_size":160639,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018953,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679792,"kb_used_data":395496,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93692048,"statfs":{"total":96636764160,"available":95940657152,"internally_reserved":0,"allocated":404987904,"data_stored":404730198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.248},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":2.2599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691898,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567640,"kb_used_data":301388,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93804200,"statfs":{"total":96636764160,"available":96055500800,"internally_reserved":0,"allocated":308621312,"data_stored":308291346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":272625322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.143}]}]},{"osd":0,"up_from":24,"seq":103079215181,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":816444,"kb_used_data":455924,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93555396,"statfs":{"total":96636764160,"available":95800725504,"internally_reserved":0,"allocated":466866176,"data_stored":466616156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.88},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":3.6579999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.0269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":465645568,"data_stored":465641955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308244480,"data_stored":308164535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404414464,"data_stored":404406779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:37.767 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:37.767 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'12406", 'reported_seq': 18928, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:36.620627+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:36.620627+0000', 'last_peered': '2024-09-16T10:28:36.620627+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:36.620627+0000', 'last_undegraded': '2024-09-16T10:28:36.620627+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'2400", 'ondisk_log_start': "35'2400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33021952, 'num_objects': 8062, 'num_object_clones': 0, 'num_object_copies': 16124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 966, 'num_objects_unfound': 0, 'num_objects_dirty': 8062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12406, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3226, 'num_bytes_recovered': 13213696, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7096}, {'shards': '0,2', 'objects': 966}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:37.767 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:37.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'12549", 'reported_seq': 12764, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:36.451366+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:36.451366+0000', 'last_peered': '2024-09-16T10:28:36.451366+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:36.451366+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'2500", 'ondisk_log_start': "35'2500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33234968, 'num_objects': 8115, 'num_object_clones': 0, 'num_object_copies': 16230, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8115, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8606, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2120}, {'shards': '2', 'objects': 3794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:37.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:37.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'12315", 'reported_seq': 12545, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:36.509330+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:36.509330+0000', 'last_peered': '2024-09-16T10:28:36.509330+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:36.509330+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'2300", 'ondisk_log_start': "35'2300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32997376, 'num_objects': 8056, 'num_object_clones': 0, 'num_object_copies': 16112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4186, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8940, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 33, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3872}, {'shards': '0,2', 'objects': 2090}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:37.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:37.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'12780", 'reported_seq': 13100, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:36.621142+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:36.621142+0000', 'last_peered': '2024-09-16T10:28:36.621142+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:36.621142+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'2700", 'ondisk_log_start': "35'2700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33341440, 'num_objects': 8140, 'num_object_clones': 0, 'num_object_copies': 16280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4332, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9022, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3810}, {'shards': '0,2', 'objects': 2086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:37.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:37.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'12866", 'reported_seq': 13037, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:36.567772+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:36.567772+0000', 'last_peered': '2024-09-16T10:28:36.567772+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:36.567772+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'2800", 'ondisk_log_start': "35'2800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33481670, 'num_objects': 8176, 'num_object_clones': 0, 'num_object_copies': 16352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4390, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8176, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12864, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 48, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3788}, {'shards': '0,2', 'objects': 2092}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:37.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:37.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'12405", 'reported_seq': 12542, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:36.458179+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:36.458179+0000', 'last_peered': '2024-09-16T10:28:36.458179+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:36.458179+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'2400", 'ondisk_log_start': "35'2400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32826310, 'num_objects': 8016, 'num_object_clones': 0, 'num_object_copies': 16032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4263, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8016, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12403, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 2084}, {'shards': '2', 'objects': 3755}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:37.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:37.769 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'12501", 'reported_seq': 12612, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:36.337063+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:36.337063+0000', 'last_peered': '2024-09-16T10:28:36.337063+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:36.337063+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'2500", 'ondisk_log_start': "35'2500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33076166, 'num_objects': 8077, 'num_object_clones': 0, 'num_object_copies': 16154, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12499, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1982}, {'shards': '2', 'objects': 3897}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:37.769 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:37.769 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'12135", 'reported_seq': 12320, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:36.629678+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:36.629678+0000', 'last_peered': '2024-09-16T10:28:36.629678+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:36.629678+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'2100", 'ondisk_log_start': "35'2100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32354787, 'num_objects': 7900, 'num_object_clones': 0, 'num_object_copies': 15800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4107, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12134, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 54, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3794}, {'shards': '0,2', 'objects': 2042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:37.769 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:37.769 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'12734", 'reported_seq': 12936, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:36.535968+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:36.535968+0000', 'last_peered': '2024-09-16T10:28:36.535968+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:36.535968+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'2700", 'ondisk_log_start': "35'2700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33506729, 'num_objects': 8183, 'num_object_clones': 0, 'num_object_copies': 16366, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12731, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 37, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3880}, {'shards': '0,2', 'objects': 2064}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:37.769 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:37.769 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'12263", 'reported_seq': 12430, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:36.572420+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:36.572420+0000', 'last_peered': '2024-09-16T10:28:36.572420+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:36.572420+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'2200", 'ondisk_log_start': "35'2200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32265158, 'num_objects': 7879, 'num_object_clones': 0, 'num_object_copies': 15758, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4126, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12261, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3755}, {'shards': '0,2', 'objects': 1976}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:37.769 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:37.769 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'12772", 'reported_seq': 12957, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:36.588679+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:36.588679+0000', 'last_peered': '2024-09-16T10:28:36.588679+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:36.588679+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'2700", 'ondisk_log_start': "35'2700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33277836, 'num_objects': 8128, 'num_object_clones': 0, 'num_object_copies': 16256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4352, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12769, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3779}, {'shards': '0,2', 'objects': 2077}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:37.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:38.445 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:38.445 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42566/258006 objects degraded (16.498%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":407,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":129003,"data_bytes":528319009,"bytes_used":2113409024,"bytes_avail":287796883456,"bytes_total":289910292480,"degraded_objects":42566,"degraded_total":258006,"degraded_ratio":0.16498065936451092,"misplaced_objects":966,"misplaced_total":258006,"misplaced_ratio":0.0037440989744424546,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":141618,"recovering_keys_per_sec":0,"num_objects_recovered":473,"num_bytes_recovered":1699840,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":540},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:38.468 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:38.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:39.007 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:39.009 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:39.031 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-09-16T10:28:38.672251+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":528319009,"num_objects":129003,"num_object_clones":0,"num_object_copies":258006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42566,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":129003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177306,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":30409206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160642,"ondisk_log_size":160642,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2063876,"kb_used_data":1152808,"kb_used_omap":12,"kb_used_meta":911027,"kb_avail":281051644,"statfs":{"total":289910292480,"available":287796883456,"internally_reserved":0,"allocated":1180475392,"data_stored":1179637700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":932891650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,5,5,9,2,3,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17735680,"num_objects":-4330,"num_object_clones":0,"num_object_copies":-8660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-49,"num_objects_misplaced":-273,"num_objects_unfound":0,"num_objects_dirty":-4330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4330,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":1118208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003748"},"pg_stats":[{"pgid":"2.f","version":"44'12406","reported_seq":18928,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:36.620627+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:36.620627+0000","last_peered":"2024-09-16T10:28:36.620627+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:36.620627+0000","last_undegraded":"2024-09-16T10:28:36.620627+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33021952,"num_objects":8062,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":8062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3226,"num_bytes_recovered":13213696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7096},{"shards":"0,2","objects":966}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12405","reported_seq":12563,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.400243+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:36.400243+0000","last_peered":"2024-09-16T10:28:36.400243+0000","last_clean":"2024-09-16T10:28:36.400243+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:36.400243+0000","last_undegraded":"2024-09-16T10:28:36.400243+0000","last_fullsized":"2024-09-16T10:28:36.400243+0000","mapping_epoch":34,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33001472,"num_objects":8057,"num_object_clones":0,"num_object_copies":16114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12549","reported_seq":12764,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.451366+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:36.451366+0000","last_peered":"2024-09-16T10:28:36.451366+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:36.451366+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33234968,"num_objects":8115,"num_object_clones":0,"num_object_copies":16230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8606,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2120},{"shards":"2","objects":3794}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12461","reported_seq":12518,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.425716+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:36.425716+0000","last_peered":"2024-09-16T10:28:36.425716+0000","last_clean":"2024-09-16T10:28:36.425716+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:36.425716+0000","last_undegraded":"2024-09-16T10:28:36.425716+0000","last_fullsized":"2024-09-16T10:28:36.425716+0000","mapping_epoch":34,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32772096,"num_objects":8001,"num_object_clones":0,"num_object_copies":16002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12435","reported_seq":12592,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.583534+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:36.583534+0000","last_peered":"2024-09-16T10:28:36.583534+0000","last_clean":"2024-09-16T10:28:36.583534+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:36.583534+0000","last_undegraded":"2024-09-16T10:28:36.583534+0000","last_fullsized":"2024-09-16T10:28:36.583534+0000","mapping_epoch":33,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33054720,"num_objects":8070,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12315","reported_seq":12545,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.509330+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:36.509330+0000","last_peered":"2024-09-16T10:28:36.509330+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:36.509330+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32997376,"num_objects":8056,"num_object_clones":0,"num_object_copies":16112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8940,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3872},{"shards":"0,2","objects":2090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12780","reported_seq":13100,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.621142+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:36.621142+0000","last_peered":"2024-09-16T10:28:36.621142+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:36.621142+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33341440,"num_objects":8140,"num_object_clones":0,"num_object_copies":16280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3810},{"shards":"0,2","objects":2086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12331","reported_seq":12507,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.613393+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:36.613393+0000","last_peered":"2024-09-16T10:28:36.613393+0000","last_clean":"2024-09-16T10:28:36.613393+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:36.613393+0000","last_undegraded":"2024-09-16T10:28:36.613393+0000","last_fullsized":"2024-09-16T10:28:36.613393+0000","mapping_epoch":33,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33009664,"num_objects":8059,"num_object_clones":0,"num_object_copies":16118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8134,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12381","reported_seq":20885,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:35.072820+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:35.072820+0000","last_peered":"2024-09-16T10:28:35.072820+0000","last_clean":"2024-09-16T10:28:35.072820+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:35.072820+0000","last_undegraded":"2024-09-16T10:28:35.072820+0000","last_fullsized":"2024-09-16T10:28:35.072820+0000","mapping_epoch":43,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33096646,"num_objects":8082,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12379,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12866","reported_seq":13037,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.567772+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:36.567772+0000","last_peered":"2024-09-16T10:28:36.567772+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:36.567772+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33481670,"num_objects":8176,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12864,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3788},{"shards":"0,2","objects":2092}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12405","reported_seq":12542,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.458179+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:36.458179+0000","last_peered":"2024-09-16T10:28:36.458179+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:36.458179+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32826310,"num_objects":8016,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12403,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2084},{"shards":"2","objects":3755}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12501","reported_seq":12612,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.337063+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:36.337063+0000","last_peered":"2024-09-16T10:28:36.337063+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:36.337063+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076166,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1982},{"shards":"2","objects":3897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12135","reported_seq":12320,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.629678+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:36.629678+0000","last_peered":"2024-09-16T10:28:36.629678+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:36.629678+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32354787,"num_objects":7900,"num_object_clones":0,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12134,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3794},{"shards":"0,2","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12734","reported_seq":12936,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.535968+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:36.535968+0000","last_peered":"2024-09-16T10:28:36.535968+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:36.535968+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33506729,"num_objects":8183,"num_object_clones":0,"num_object_copies":16366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12731,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3880},{"shards":"0,2","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12263","reported_seq":12430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.572420+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:36.572420+0000","last_peered":"2024-09-16T10:28:36.572420+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:36.572420+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32265158,"num_objects":7879,"num_object_clones":0,"num_object_copies":15758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12261,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3755},{"shards":"0,2","objects":1976}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12772","reported_seq":12957,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.588679+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:36.588679+0000","last_peered":"2024-09-16T10:28:36.588679+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:36.588679+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33277836,"num_objects":8128,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12769,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3779},{"shards":"0,2","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":528318990,"num_objects":129001,"num_object_clones":0,"num_object_copies":258002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42566,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":129001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177304,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7814,"num_bytes_recovered":30409187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1178304512,"data_stored":1178213269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160639,"ondisk_log_size":160639,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018953,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679792,"kb_used_data":395496,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93692048,"statfs":{"total":96636764160,"available":95940657152,"internally_reserved":0,"allocated":404987904,"data_stored":404730198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.248},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":2.2599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691898,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567640,"kb_used_data":301388,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93804200,"statfs":{"total":96636764160,"available":96055500800,"internally_reserved":0,"allocated":308621312,"data_stored":308291346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":272625322},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.27000000000000002},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.143}]}]},{"osd":0,"up_from":24,"seq":103079215181,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":816444,"kb_used_data":455924,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93555396,"statfs":{"total":96636764160,"available":95800725504,"internally_reserved":0,"allocated":466866176,"data_stored":466616156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.88},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":3.6579999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.0269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":465645568,"data_stored":465641955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":308244480,"data_stored":308164535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404414464,"data_stored":404406779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:42.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:42.546 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:42.547 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:42.570 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-16T10:28:40.672688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":527720993,"num_objects":128857,"num_object_clones":0,"num_object_copies":257714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42566,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":128857,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177452,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7815,"num_bytes_recovered":30409206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160588,"ondisk_log_size":160588,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2085580,"kb_used_data":1152372,"kb_used_omap":12,"kb_used_meta":933171,"kb_avail":281029940,"statfs":{"total":289910292480,"available":287774658560,"internally_reserved":0,"allocated":1180028928,"data_stored":1179191236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":955567106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,3,7,1,3,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18333696,"num_objects":-4476,"num_object_clones":0,"num_object_copies":-8952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-49,"num_objects_misplaced":-273,"num_objects_unfound":0,"num_objects_dirty":-4476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4476,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":1118208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003670"},"pg_stats":[{"pgid":"2.f","version":"44'12406","reported_seq":18928,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:36.620627+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:36.620627+0000","last_peered":"2024-09-16T10:28:36.620627+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:36.620627+0000","last_undegraded":"2024-09-16T10:28:36.620627+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33021952,"num_objects":8062,"num_object_clones":0,"num_object_copies":16124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":8062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3226,"num_bytes_recovered":13213696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7096},{"shards":"0,2","objects":966}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12405","reported_seq":12563,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.400243+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:36.400243+0000","last_peered":"2024-09-16T10:28:36.400243+0000","last_clean":"2024-09-16T10:28:36.400243+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:36.400243+0000","last_undegraded":"2024-09-16T10:28:36.400243+0000","last_fullsized":"2024-09-16T10:28:36.400243+0000","mapping_epoch":34,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33001472,"num_objects":8057,"num_object_clones":0,"num_object_copies":16114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12549","reported_seq":12764,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.451366+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:36.451366+0000","last_peered":"2024-09-16T10:28:36.451366+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:36.451366+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33234968,"num_objects":8115,"num_object_clones":0,"num_object_copies":16230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8606,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2120},{"shards":"2","objects":3794}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12461","reported_seq":12518,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.425716+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:36.425716+0000","last_peered":"2024-09-16T10:28:36.425716+0000","last_clean":"2024-09-16T10:28:36.425716+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:36.425716+0000","last_undegraded":"2024-09-16T10:28:36.425716+0000","last_fullsized":"2024-09-16T10:28:36.425716+0000","mapping_epoch":34,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32772096,"num_objects":8001,"num_object_clones":0,"num_object_copies":16002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12435","reported_seq":12592,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.583534+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:36.583534+0000","last_peered":"2024-09-16T10:28:36.583534+0000","last_clean":"2024-09-16T10:28:36.583534+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:36.583534+0000","last_undegraded":"2024-09-16T10:28:36.583534+0000","last_fullsized":"2024-09-16T10:28:36.583534+0000","mapping_epoch":33,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33054720,"num_objects":8070,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9077,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12315","reported_seq":12545,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.509330+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:36.509330+0000","last_peered":"2024-09-16T10:28:36.509330+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:36.509330+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32997376,"num_objects":8056,"num_object_clones":0,"num_object_copies":16112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8940,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3872},{"shards":"0,2","objects":2090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12780","reported_seq":13100,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.621142+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:36.621142+0000","last_peered":"2024-09-16T10:28:36.621142+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:36.621142+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33341440,"num_objects":8140,"num_object_clones":0,"num_object_copies":16280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3810},{"shards":"0,2","objects":2086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12331","reported_seq":12507,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:36.613393+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:36.613393+0000","last_peered":"2024-09-16T10:28:36.613393+0000","last_clean":"2024-09-16T10:28:36.613393+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:36.613393+0000","last_undegraded":"2024-09-16T10:28:36.613393+0000","last_fullsized":"2024-09-16T10:28:36.613393+0000","mapping_epoch":33,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33009664,"num_objects":8059,"num_object_clones":0,"num_object_copies":16118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8134,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12527","reported_seq":21031,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:40.084486+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:40.084486+0000","last_peered":"2024-09-16T10:28:40.084486+0000","last_clean":"2024-09-16T10:28:40.084486+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:40.084486+0000","last_undegraded":"2024-09-16T10:28:40.084486+0000","last_fullsized":"2024-09-16T10:28:40.084486+0000","mapping_epoch":43,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32498630,"num_objects":7936,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12525,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'12866","reported_seq":13037,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.567772+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:36.567772+0000","last_peered":"2024-09-16T10:28:36.567772+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:36.567772+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33481670,"num_objects":8176,"num_object_clones":0,"num_object_copies":16352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12864,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3788},{"shards":"0,2","objects":2092}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12405","reported_seq":12542,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.458179+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:36.458179+0000","last_peered":"2024-09-16T10:28:36.458179+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:36.458179+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32826310,"num_objects":8016,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12403,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":2084},{"shards":"2","objects":3755}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12501","reported_seq":12612,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.337063+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:36.337063+0000","last_peered":"2024-09-16T10:28:36.337063+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:36.337063+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33076166,"num_objects":8077,"num_object_clones":0,"num_object_copies":16154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1982},{"shards":"2","objects":3897}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12135","reported_seq":12320,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.629678+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:36.629678+0000","last_peered":"2024-09-16T10:28:36.629678+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:36.629678+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2100","ondisk_log_start":"35'2100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32354787,"num_objects":7900,"num_object_clones":0,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12134,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3794},{"shards":"0,2","objects":2042}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12734","reported_seq":12936,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.535968+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:36.535968+0000","last_peered":"2024-09-16T10:28:36.535968+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:36.535968+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33506729,"num_objects":8183,"num_object_clones":0,"num_object_copies":16366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12731,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3880},{"shards":"0,2","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12263","reported_seq":12430,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.572420+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:36.572420+0000","last_peered":"2024-09-16T10:28:36.572420+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:36.572420+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32265158,"num_objects":7879,"num_object_clones":0,"num_object_copies":15758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12261,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3755},{"shards":"0,2","objects":1976}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12772","reported_seq":12957,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:36.588679+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:36.588679+0000","last_peered":"2024-09-16T10:28:36.588679+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:36.588679+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33277836,"num_objects":8128,"num_object_clones":0,"num_object_copies":16256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12769,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3779},{"shards":"0,2","objects":2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":527720974,"num_objects":128855,"num_object_clones":0,"num_object_copies":257710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42566,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":128855,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177450,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7814,"num_bytes_recovered":30409187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1177833472,"data_stored":1177742229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160585,"ondisk_log_size":160585,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018953,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679792,"kb_used_data":395496,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93692048,"statfs":{"total":96636764160,"available":95940657152,"internally_reserved":0,"allocated":404987904,"data_stored":404730198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,0,0,0,0,0,0,0,0,1],"upper_bound":32768},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.248},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.30299999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":2.2599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691899,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589344,"kb_used_data":300952,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93782496,"statfs":{"total":96636764160,"available":96033275904,"internally_reserved":0,"allocated":308174848,"data_stored":307844882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":295300778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.921},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.9550000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.877}]}]},{"osd":0,"up_from":24,"seq":103079215181,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":816444,"kb_used_data":455924,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93555396,"statfs":{"total":96636764160,"available":95800725504,"internally_reserved":0,"allocated":466866176,"data_stored":466616156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,3,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.88},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":3.6579999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.0269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":465645568,"data_stored":465641955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307773440,"data_stored":307693495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":404414464,"data_stored":404406779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:42.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:43.120 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:43.144 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-16T10:28:42.673195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":519516705,"num_objects":126854,"num_object_clones":0,"num_object_copies":253708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42548,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":126854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179455,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7995,"num_bytes_recovered":31072758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160691,"ondisk_log_size":160691,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2083052,"kb_used_data":1141652,"kb_used_omap":12,"kb_used_meta":941363,"kb_avail":281032468,"statfs":{"total":289910292480,"available":287777247232,"internally_reserved":0,"allocated":1169051648,"data_stored":1168213956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":963955714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,7,6,4,4,0,0,0,0,5,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25968640,"num_objects":-6340,"num_object_clones":0,"num_object_copies":-12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-67,"num_objects_misplaced":-435,"num_objects_unfound":0,"num_objects_dirty":-6340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6340,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":502,"num_bytes_recovered":1781760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003731"},"pg_stats":[{"pgid":"2.f","version":"44'12518","reported_seq":19364,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:41.658163+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:41.658163+0000","last_peered":"2024-09-16T10:28:41.658163+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:41.658163+0000","last_undegraded":"2024-09-16T10:28:41.658163+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32563200,"num_objects":7950,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":7950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3388,"num_bytes_recovered":13877248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7146},{"shards":"0,2","objects":804}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12543","reported_seq":12701,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.247225+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:41.247225+0000","last_peered":"2024-09-16T10:28:41.247225+0000","last_clean":"2024-09-16T10:28:41.247225+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:41.247225+0000","last_undegraded":"2024-09-16T10:28:41.247225+0000","last_fullsized":"2024-09-16T10:28:41.247225+0000","mapping_epoch":34,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32436224,"num_objects":7919,"num_object_clones":0,"num_object_copies":15838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12684","reported_seq":12915,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.251318+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:41.251318+0000","last_peered":"2024-09-16T10:28:41.251318+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:41.251318+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32682008,"num_objects":7980,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7980,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8741,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1985},{"shards":"2","objects":3667}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12603","reported_seq":12660,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.476528+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:41.476528+0000","last_peered":"2024-09-16T10:28:41.476528+0000","last_clean":"2024-09-16T10:28:41.476528+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:41.476528+0000","last_undegraded":"2024-09-16T10:28:41.476528+0000","last_fullsized":"2024-09-16T10:28:41.476528+0000","mapping_epoch":34,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32190464,"num_objects":7859,"num_object_clones":0,"num_object_copies":15718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12603,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12576","reported_seq":12733,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.661650+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:41.661650+0000","last_peered":"2024-09-16T10:28:41.661650+0000","last_clean":"2024-09-16T10:28:41.661650+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:41.661650+0000","last_undegraded":"2024-09-16T10:28:41.661650+0000","last_fullsized":"2024-09-16T10:28:41.661650+0000","mapping_epoch":33,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32477184,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9218,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12449","reported_seq":12680,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.667141+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:41.667141+0000","last_peered":"2024-09-16T10:28:41.667141+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:41.667141+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32448512,"num_objects":7922,"num_object_clones":0,"num_object_copies":15844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3740},{"shards":"0,2","objects":1956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12909","reported_seq":13229,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.616181+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:41.616181+0000","last_peered":"2024-09-16T10:28:41.616181+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:41.616181+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32813056,"num_objects":8011,"num_object_clones":0,"num_object_copies":16022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3681},{"shards":"0,2","objects":1957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12467","reported_seq":12643,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.605153+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:41.605153+0000","last_peered":"2024-09-16T10:28:41.605153+0000","last_clean":"2024-09-16T10:28:41.605153+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:41.605153+0000","last_undegraded":"2024-09-16T10:28:41.605153+0000","last_fullsized":"2024-09-16T10:28:41.605153+0000","mapping_epoch":33,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32452608,"num_objects":7923,"num_object_clones":0,"num_object_copies":15846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8270,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12527","reported_seq":21031,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:40.084486+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:40.084486+0000","last_peered":"2024-09-16T10:28:40.084486+0000","last_clean":"2024-09-16T10:28:40.084486+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:40.084486+0000","last_undegraded":"2024-09-16T10:28:40.084486+0000","last_fullsized":"2024-09-16T10:28:40.084486+0000","mapping_epoch":43,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32498630,"num_objects":7936,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12525,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13004","reported_seq":13176,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.615217+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:41.615217+0000","last_peered":"2024-09-16T10:28:41.615217+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:41.615217+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32916422,"num_objects":8038,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13002,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3652},{"shards":"0,2","objects":1954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12533","reported_seq":12673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.435955+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:41.435955+0000","last_peered":"2024-09-16T10:28:41.435955+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:41.435955+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32302022,"num_objects":7888,"num_object_clones":0,"num_object_copies":15776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1956},{"shards":"2","objects":3628}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12633","reported_seq":12746,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.428453+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:41.428453+0000","last_peered":"2024-09-16T10:28:41.428453+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:41.428453+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535494,"num_objects":7945,"num_object_clones":0,"num_object_copies":15890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12631,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1850},{"shards":"2","objects":3766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12270","reported_seq":12458,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.603022+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:41.603022+0000","last_peered":"2024-09-16T10:28:41.603022+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:41.603022+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31801827,"num_objects":7765,"num_object_clones":0,"num_object_copies":15530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12269,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3660},{"shards":"0,2","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12870","reported_seq":13074,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.552342+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:41.552342+0000","last_peered":"2024-09-16T10:28:41.552342+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:41.552342+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32949673,"num_objects":8047,"num_object_clones":0,"num_object_copies":16094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12867,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3745},{"shards":"0,2","objects":1928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12387","reported_seq":12556,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.581598+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:41.581598+0000","last_peered":"2024-09-16T10:28:41.581598+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:41.581598+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31757254,"num_objects":7755,"num_object_clones":0,"num_object_copies":15510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3632},{"shards":"0,2","objects":1852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12915","reported_seq":13102,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.561028+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:41.561028+0000","last_peered":"2024-09-16T10:28:41.561028+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:41.561028+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32692108,"num_objects":7985,"num_object_clones":0,"num_object_copies":15970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12912,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3637},{"shards":"0,2","objects":1934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":519516686,"num_objects":126852,"num_object_clones":0,"num_object_copies":253704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42548,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":126852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179453,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7994,"num_bytes_recovered":31072739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1166798848,"data_stored":1166707605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160688,"ondisk_log_size":160688,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018954,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679556,"kb_used_data":391164,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93692284,"statfs":{"total":96636764160,"available":95940898816,"internally_reserved":0,"allocated":400551936,"data_stored":400294230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.476}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":0.76900000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691899,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589344,"kb_used_data":300952,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93782496,"statfs":{"total":96636764160,"available":96033275904,"internally_reserved":0,"allocated":308174848,"data_stored":307844882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":295300778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.921},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.9550000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.877}]}]},{"osd":0,"up_from":24,"seq":103079215182,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":814152,"kb_used_data":449536,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93557688,"statfs":{"total":96636764160,"available":95803072512,"internally_reserved":0,"allocated":460324864,"data_stored":460074844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":373354152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,0,2,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.431},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.432}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.9119999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.6970000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":459063296,"data_stored":459059683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307773440,"data_stored":307693495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399962112,"data_stored":399954427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:43.146 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:43.146 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'12518", 'reported_seq': 19364, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:41.658163+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:41.658163+0000', 'last_peered': '2024-09-16T10:28:41.658163+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:41.658163+0000', 'last_undegraded': '2024-09-16T10:28:41.658163+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'2500", 'ondisk_log_start': "35'2500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32563200, 'num_objects': 7950, 'num_object_clones': 0, 'num_object_copies': 15900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 804, 'num_objects_unfound': 0, 'num_objects_dirty': 7950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12518, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3388, 'num_bytes_recovered': 13877248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7146}, {'shards': '0,2', 'objects': 804}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:43.146 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:43.146 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'12684", 'reported_seq': 12915, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.251318+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:41.251318+0000', 'last_peered': '2024-09-16T10:28:41.251318+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:41.251318+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'2600", 'ondisk_log_start': "35'2600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32682008, 'num_objects': 7980, 'num_object_clones': 0, 'num_object_copies': 15960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7980, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8741, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1985}, {'shards': '2', 'objects': 3667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:43.146 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:43.146 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'12449", 'reported_seq': 12680, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.667141+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:41.667141+0000', 'last_peered': '2024-09-16T10:28:41.667141+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:41.667141+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'2400", 'ondisk_log_start': "35'2400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32448512, 'num_objects': 7922, 'num_object_clones': 0, 'num_object_copies': 15844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4184, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9074, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3740}, {'shards': '0,2', 'objects': 1956}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:43.146 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:43.146 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'12909", 'reported_seq': 13229, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.616181+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:41.616181+0000', 'last_peered': '2024-09-16T10:28:41.616181+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:41.616181+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'2900", 'ondisk_log_start': "35'2900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32813056, 'num_objects': 8011, 'num_object_clones': 0, 'num_object_copies': 16022, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4332, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9151, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3681}, {'shards': '0,2', 'objects': 1957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:43.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:43.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'13004", 'reported_seq': 13176, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.615217+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:41.615217+0000', 'last_peered': '2024-09-16T10:28:41.615217+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:41.615217+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'3000", 'ondisk_log_start': "35'3000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32916422, 'num_objects': 8038, 'num_object_clones': 0, 'num_object_copies': 16076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4388, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13002, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 50, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3652}, {'shards': '0,2', 'objects': 1954}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:43.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:43.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'12533", 'reported_seq': 12673, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.435955+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:41.435955+0000', 'last_peered': '2024-09-16T10:28:41.435955+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:41.435955+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'2500", 'ondisk_log_start': "35'2500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32302022, 'num_objects': 7888, 'num_object_clones': 0, 'num_object_copies': 15776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12531, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1956}, {'shards': '2', 'objects': 3628}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:43.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:43.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'12633", 'reported_seq': 12746, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.428453+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:41.428453+0000', 'last_peered': '2024-09-16T10:28:41.428453+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:41.428453+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'2600", 'ondisk_log_start': "35'2600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32535494, 'num_objects': 7945, 'num_object_clones': 0, 'num_object_copies': 15890, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4181, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7945, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12631, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1850}, {'shards': '2', 'objects': 3766}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:43.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:43.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'12270", 'reported_seq': 12458, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.603022+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:41.603022+0000', 'last_peered': '2024-09-16T10:28:41.603022+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:41.603022+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'2200", 'ondisk_log_start': "35'2200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31801827, 'num_objects': 7765, 'num_object_clones': 0, 'num_object_copies': 15530, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12269, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3660}, {'shards': '0,2', 'objects': 1907}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:43.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:43.147 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'12870", 'reported_seq': 13074, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.552342+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:41.552342+0000', 'last_peered': '2024-09-16T10:28:41.552342+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:41.552342+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'2800", 'ondisk_log_start': "35'2800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32949673, 'num_objects': 8047, 'num_object_clones': 0, 'num_object_copies': 16094, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12867, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3745}, {'shards': '0,2', 'objects': 1928}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:43.148 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:43.148 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'12387", 'reported_seq': 12556, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.581598+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:41.581598+0000', 'last_peered': '2024-09-16T10:28:41.581598+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:41.581598+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'2300", 'ondisk_log_start': "35'2300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31757254, 'num_objects': 7755, 'num_object_clones': 0, 'num_object_copies': 15510, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12385, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3632}, {'shards': '0,2', 'objects': 1852}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:43.148 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:43.148 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'12915", 'reported_seq': 13102, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.561028+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:41.561028+0000', 'last_peered': '2024-09-16T10:28:41.561028+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:41.561028+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'2900", 'ondisk_log_start': "35'2900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32692108, 'num_objects': 7985, 'num_object_clones': 0, 'num_object_copies': 15970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4351, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12912, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3637}, {'shards': '0,2', 'objects': 1934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:43.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:43.787 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:43.787 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42548/253708 objects degraded (16.770%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":412,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":126854,"data_bytes":519516705,"bytes_used":2133045248,"bytes_avail":287777247232,"bytes_total":289910292480,"degraded_objects":42548,"degraded_total":253708,"degraded_ratio":0.16770460529427531,"misplaced_objects":804,"misplaced_total":253708,"misplaced_ratio":0.0031689974301165121,"recovering_objects_per_sec":41,"recovering_bytes_per_sec":148433,"recovering_keys_per_sec":0,"num_objects_recovered":502,"num_bytes_recovered":1781760,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":528},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:43.817 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:43.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:44.358 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:44.358 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:44.383 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-16T10:28:42.673195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":519516705,"num_objects":126854,"num_object_clones":0,"num_object_copies":253708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42548,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":126854,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179455,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7995,"num_bytes_recovered":31072758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160691,"ondisk_log_size":160691,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2083052,"kb_used_data":1141652,"kb_used_omap":12,"kb_used_meta":941363,"kb_avail":281032468,"statfs":{"total":289910292480,"available":287777247232,"internally_reserved":0,"allocated":1169051648,"data_stored":1168213956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":963955714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,7,6,4,4,0,0,0,0,5,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25968640,"num_objects":-6340,"num_object_clones":0,"num_object_copies":-12680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-67,"num_objects_misplaced":-435,"num_objects_unfound":0,"num_objects_dirty":-6340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6340,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":502,"num_bytes_recovered":1781760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003731"},"pg_stats":[{"pgid":"2.f","version":"44'12518","reported_seq":19364,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:41.658163+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:41.658163+0000","last_peered":"2024-09-16T10:28:41.658163+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:41.658163+0000","last_undegraded":"2024-09-16T10:28:41.658163+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32563200,"num_objects":7950,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":7950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3388,"num_bytes_recovered":13877248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7146},{"shards":"0,2","objects":804}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12543","reported_seq":12701,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.247225+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:41.247225+0000","last_peered":"2024-09-16T10:28:41.247225+0000","last_clean":"2024-09-16T10:28:41.247225+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:41.247225+0000","last_undegraded":"2024-09-16T10:28:41.247225+0000","last_fullsized":"2024-09-16T10:28:41.247225+0000","mapping_epoch":34,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32436224,"num_objects":7919,"num_object_clones":0,"num_object_copies":15838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12684","reported_seq":12915,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.251318+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:41.251318+0000","last_peered":"2024-09-16T10:28:41.251318+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:41.251318+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32682008,"num_objects":7980,"num_object_clones":0,"num_object_copies":15960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7980,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8741,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1985},{"shards":"2","objects":3667}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12603","reported_seq":12660,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.476528+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:41.476528+0000","last_peered":"2024-09-16T10:28:41.476528+0000","last_clean":"2024-09-16T10:28:41.476528+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:41.476528+0000","last_undegraded":"2024-09-16T10:28:41.476528+0000","last_fullsized":"2024-09-16T10:28:41.476528+0000","mapping_epoch":34,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32190464,"num_objects":7859,"num_object_clones":0,"num_object_copies":15718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12603,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12576","reported_seq":12733,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.661650+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:41.661650+0000","last_peered":"2024-09-16T10:28:41.661650+0000","last_clean":"2024-09-16T10:28:41.661650+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:41.661650+0000","last_undegraded":"2024-09-16T10:28:41.661650+0000","last_fullsized":"2024-09-16T10:28:41.661650+0000","mapping_epoch":33,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32477184,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9218,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12449","reported_seq":12680,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.667141+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:41.667141+0000","last_peered":"2024-09-16T10:28:41.667141+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:41.667141+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32448512,"num_objects":7922,"num_object_clones":0,"num_object_copies":15844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3740},{"shards":"0,2","objects":1956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12909","reported_seq":13229,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.616181+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:41.616181+0000","last_peered":"2024-09-16T10:28:41.616181+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:41.616181+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32813056,"num_objects":8011,"num_object_clones":0,"num_object_copies":16022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3681},{"shards":"0,2","objects":1957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12467","reported_seq":12643,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.605153+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:41.605153+0000","last_peered":"2024-09-16T10:28:41.605153+0000","last_clean":"2024-09-16T10:28:41.605153+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:41.605153+0000","last_undegraded":"2024-09-16T10:28:41.605153+0000","last_fullsized":"2024-09-16T10:28:41.605153+0000","mapping_epoch":33,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32452608,"num_objects":7923,"num_object_clones":0,"num_object_copies":15846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8270,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12527","reported_seq":21031,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:40.084486+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:40.084486+0000","last_peered":"2024-09-16T10:28:40.084486+0000","last_clean":"2024-09-16T10:28:40.084486+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:40.084486+0000","last_undegraded":"2024-09-16T10:28:40.084486+0000","last_fullsized":"2024-09-16T10:28:40.084486+0000","mapping_epoch":43,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32498630,"num_objects":7936,"num_object_clones":0,"num_object_copies":15872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12525,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13004","reported_seq":13176,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.615217+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:41.615217+0000","last_peered":"2024-09-16T10:28:41.615217+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:41.615217+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32916422,"num_objects":8038,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13002,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3652},{"shards":"0,2","objects":1954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12533","reported_seq":12673,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.435955+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:41.435955+0000","last_peered":"2024-09-16T10:28:41.435955+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:41.435955+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32302022,"num_objects":7888,"num_object_clones":0,"num_object_copies":15776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1956},{"shards":"2","objects":3628}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12633","reported_seq":12746,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.428453+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:41.428453+0000","last_peered":"2024-09-16T10:28:41.428453+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:41.428453+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535494,"num_objects":7945,"num_object_clones":0,"num_object_copies":15890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12631,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1850},{"shards":"2","objects":3766}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12270","reported_seq":12458,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.603022+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:41.603022+0000","last_peered":"2024-09-16T10:28:41.603022+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:41.603022+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31801827,"num_objects":7765,"num_object_clones":0,"num_object_copies":15530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12269,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3660},{"shards":"0,2","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12870","reported_seq":13074,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.552342+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:41.552342+0000","last_peered":"2024-09-16T10:28:41.552342+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:41.552342+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32949673,"num_objects":8047,"num_object_clones":0,"num_object_copies":16094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12867,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3745},{"shards":"0,2","objects":1928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12387","reported_seq":12556,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.581598+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:41.581598+0000","last_peered":"2024-09-16T10:28:41.581598+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:41.581598+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31757254,"num_objects":7755,"num_object_clones":0,"num_object_copies":15510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3632},{"shards":"0,2","objects":1852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12915","reported_seq":13102,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.561028+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:41.561028+0000","last_peered":"2024-09-16T10:28:41.561028+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:41.561028+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32692108,"num_objects":7985,"num_object_clones":0,"num_object_copies":15970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12912,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3637},{"shards":"0,2","objects":1934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":519516686,"num_objects":126852,"num_object_clones":0,"num_object_copies":253704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42548,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":126852,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179453,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7994,"num_bytes_recovered":31072739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1166798848,"data_stored":1166707605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160688,"ondisk_log_size":160688,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018954,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679556,"kb_used_data":391164,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93692284,"statfs":{"total":96636764160,"available":95940898816,"internally_reserved":0,"allocated":400551936,"data_stored":400294230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,2,1,0,0,0,0,5],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.476}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":0.76900000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691899,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589344,"kb_used_data":300952,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93782496,"statfs":{"total":96636764160,"available":96033275904,"internally_reserved":0,"allocated":308174848,"data_stored":307844882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":295300778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.921},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.9550000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":2.3969999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.877}]}]},{"osd":0,"up_from":24,"seq":103079215182,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":814152,"kb_used_data":449536,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93557688,"statfs":{"total":96636764160,"available":95803072512,"internally_reserved":0,"allocated":460324864,"data_stored":460074844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":373354152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,0,2,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.431},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.432}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.9119999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.6970000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":459063296,"data_stored":459059683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307773440,"data_stored":307693495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":399962112,"data_stored":399954427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:47.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:47.963 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:47.965 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:47.984 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-16T10:28:46.674206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516256289,"num_objects":126058,"num_object_clones":0,"num_object_copies":252116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42543,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":126058,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180251,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8000,"num_bytes_recovered":31072758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160787,"ondisk_log_size":160787,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2074464,"kb_used_data":1137160,"kb_used_omap":12,"kb_used_meta":937267,"kb_avail":281041056,"statfs":{"total":289910292480,"available":287786041344,"internally_reserved":0,"allocated":1164451840,"data_stored":1163614148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":959761410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,5,10,4,2,0,0,0,0,1,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20860928,"num_objects":-5093,"num_object_clones":0,"num_object_copies":-10186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-48,"num_objects_misplaced":-302,"num_objects_unfound":0,"num_objects_dirty":-5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5093,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003692"},"pg_stats":[{"pgid":"2.f","version":"44'12518","reported_seq":19364,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:41.658163+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:41.658163+0000","last_peered":"2024-09-16T10:28:41.658163+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:41.658163+0000","last_undegraded":"2024-09-16T10:28:41.658163+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32563200,"num_objects":7950,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":7950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3388,"num_bytes_recovered":13877248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7146},{"shards":"0,2","objects":804}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12677","reported_seq":12835,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:46.406756+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:46.406756+0000","last_peered":"2024-09-16T10:28:46.406756+0000","last_clean":"2024-09-16T10:28:46.406756+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:46.406756+0000","last_undegraded":"2024-09-16T10:28:46.406756+0000","last_fullsized":"2024-09-16T10:28:46.406756+0000","mapping_epoch":34,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31887360,"num_objects":7785,"num_object_clones":0,"num_object_copies":15570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12832","reported_seq":13065,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.457637+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:46.457637+0000","last_peered":"2024-09-16T10:28:46.457637+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:46.457637+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32075800,"num_objects":7832,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8889,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1837},{"shards":"2","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12718","reported_seq":12775,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:46.410229+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:46.410229+0000","last_peered":"2024-09-16T10:28:46.410229+0000","last_clean":"2024-09-16T10:28:46.410229+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:46.410229+0000","last_undegraded":"2024-09-16T10:28:46.410229+0000","last_fullsized":"2024-09-16T10:28:46.410229+0000","mapping_epoch":34,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12718,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12576","reported_seq":12733,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.661650+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:41.661650+0000","last_peered":"2024-09-16T10:28:41.661650+0000","last_clean":"2024-09-16T10:28:41.661650+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:41.661650+0000","last_undegraded":"2024-09-16T10:28:41.661650+0000","last_fullsized":"2024-09-16T10:28:41.661650+0000","mapping_epoch":33,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32477184,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9218,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12449","reported_seq":12680,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.667141+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:41.667141+0000","last_peered":"2024-09-16T10:28:41.667141+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:41.667141+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32448512,"num_objects":7922,"num_object_clones":0,"num_object_copies":15844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3740},{"shards":"0,2","objects":1956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12909","reported_seq":13229,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.616181+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:41.616181+0000","last_peered":"2024-09-16T10:28:41.616181+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:41.616181+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32813056,"num_objects":8011,"num_object_clones":0,"num_object_copies":16022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3681},{"shards":"0,2","objects":1957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12467","reported_seq":12643,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.605153+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:41.605153+0000","last_peered":"2024-09-16T10:28:41.605153+0000","last_clean":"2024-09-16T10:28:41.605153+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:41.605153+0000","last_undegraded":"2024-09-16T10:28:41.605153+0000","last_fullsized":"2024-09-16T10:28:41.605153+0000","mapping_epoch":33,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32452608,"num_objects":7923,"num_object_clones":0,"num_object_copies":15846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8270,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12649","reported_seq":21153,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:45.046935+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:45.046935+0000","last_peered":"2024-09-16T10:28:45.046935+0000","last_clean":"2024-09-16T10:28:45.046935+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:45.046935+0000","last_undegraded":"2024-09-16T10:28:45.046935+0000","last_fullsized":"2024-09-16T10:28:45.046935+0000","mapping_epoch":43,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31998918,"num_objects":7814,"num_object_clones":0,"num_object_copies":15628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13004","reported_seq":13176,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.615217+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:41.615217+0000","last_peered":"2024-09-16T10:28:41.615217+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:41.615217+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32916422,"num_objects":8038,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13002,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3652},{"shards":"0,2","objects":1954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12680","reported_seq":12820,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.427803+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:46.427803+0000","last_peered":"2024-09-16T10:28:46.427803+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:46.427803+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31699910,"num_objects":7741,"num_object_clones":0,"num_object_copies":15482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1809},{"shards":"2","objects":3481}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12763","reported_seq":12884,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.427697+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:46.427697+0000","last_peered":"2024-09-16T10:28:46.427697+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:46.427697+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32003014,"num_objects":7815,"num_object_clones":0,"num_object_copies":15630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12761,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1720},{"shards":"2","objects":3640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12270","reported_seq":12458,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.603022+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:41.603022+0000","last_peered":"2024-09-16T10:28:41.603022+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:41.603022+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31801827,"num_objects":7765,"num_object_clones":0,"num_object_copies":15530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12269,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3660},{"shards":"0,2","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12870","reported_seq":13074,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.552342+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:41.552342+0000","last_peered":"2024-09-16T10:28:41.552342+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:41.552342+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32949673,"num_objects":8047,"num_object_clones":0,"num_object_copies":16094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12867,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3745},{"shards":"0,2","objects":1928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12387","reported_seq":12556,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.581598+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:41.581598+0000","last_peered":"2024-09-16T10:28:41.581598+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:41.581598+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31757254,"num_objects":7755,"num_object_clones":0,"num_object_copies":15510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3632},{"shards":"0,2","objects":1852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12915","reported_seq":13102,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.561028+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:41.561028+0000","last_peered":"2024-09-16T10:28:41.561028+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:41.561028+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32692108,"num_objects":7985,"num_object_clones":0,"num_object_copies":15970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12912,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3637},{"shards":"0,2","objects":1934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":516256270,"num_objects":126056,"num_object_clones":0,"num_object_copies":252112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42543,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":126056,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180249,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7999,"num_bytes_recovered":31072739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1162182656,"data_stored":1162091413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018955,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679536,"kb_used_data":387048,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93692304,"statfs":{"total":96636764160,"available":95940919296,"internally_reserved":0,"allocated":396337152,"data_stored":396079446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,2,3,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.476}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.3600000000000003},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":7.9589999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691900,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580776,"kb_used_data":300576,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93791064,"statfs":{"total":96636764160,"available":96042049536,"internally_reserved":0,"allocated":307789824,"data_stored":307459858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":286912170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.442}]}]},{"osd":0,"up_from":24,"seq":103079215182,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":814152,"kb_used_data":449536,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93557688,"statfs":{"total":96636764160,"available":95803072512,"internally_reserved":0,"allocated":460324864,"data_stored":460074844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":373354152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,0,2,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.431},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.432}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.9119999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.6970000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":459063296,"data_stored":459059683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307380224,"data_stored":307300279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":395739136,"data_stored":395731451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:47.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:48.525 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:48.526 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:48.555 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-16T10:28:46.674206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516256289,"num_objects":126058,"num_object_clones":0,"num_object_copies":252116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42543,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":126058,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180251,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8000,"num_bytes_recovered":31072758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160787,"ondisk_log_size":160787,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2074464,"kb_used_data":1137160,"kb_used_omap":12,"kb_used_meta":937267,"kb_avail":281041056,"statfs":{"total":289910292480,"available":287786041344,"internally_reserved":0,"allocated":1164451840,"data_stored":1163614148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":959761410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,5,10,4,2,0,0,0,0,1,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20860928,"num_objects":-5093,"num_object_clones":0,"num_object_copies":-10186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-48,"num_objects_misplaced":-302,"num_objects_unfound":0,"num_objects_dirty":-5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5093,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003692"},"pg_stats":[{"pgid":"2.f","version":"44'12518","reported_seq":19364,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:41.658163+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:41.658163+0000","last_peered":"2024-09-16T10:28:41.658163+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:41.658163+0000","last_undegraded":"2024-09-16T10:28:41.658163+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32563200,"num_objects":7950,"num_object_clones":0,"num_object_copies":15900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":7950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3388,"num_bytes_recovered":13877248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7146},{"shards":"0,2","objects":804}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12677","reported_seq":12835,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:46.406756+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:46.406756+0000","last_peered":"2024-09-16T10:28:46.406756+0000","last_clean":"2024-09-16T10:28:46.406756+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:46.406756+0000","last_undegraded":"2024-09-16T10:28:46.406756+0000","last_fullsized":"2024-09-16T10:28:46.406756+0000","mapping_epoch":34,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31887360,"num_objects":7785,"num_object_clones":0,"num_object_copies":15570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12832","reported_seq":13065,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.457637+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:46.457637+0000","last_peered":"2024-09-16T10:28:46.457637+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:46.457637+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32075800,"num_objects":7832,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8889,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1837},{"shards":"2","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12718","reported_seq":12775,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:46.410229+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:46.410229+0000","last_peered":"2024-09-16T10:28:46.410229+0000","last_clean":"2024-09-16T10:28:46.410229+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:46.410229+0000","last_undegraded":"2024-09-16T10:28:46.410229+0000","last_fullsized":"2024-09-16T10:28:46.410229+0000","mapping_epoch":34,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12718,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12576","reported_seq":12733,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.661650+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:41.661650+0000","last_peered":"2024-09-16T10:28:41.661650+0000","last_clean":"2024-09-16T10:28:41.661650+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:41.661650+0000","last_undegraded":"2024-09-16T10:28:41.661650+0000","last_fullsized":"2024-09-16T10:28:41.661650+0000","mapping_epoch":33,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32477184,"num_objects":7929,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9218,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12449","reported_seq":12680,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.667141+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:41.667141+0000","last_peered":"2024-09-16T10:28:41.667141+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:41.667141+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32448512,"num_objects":7922,"num_object_clones":0,"num_object_copies":15844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3740},{"shards":"0,2","objects":1956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'12909","reported_seq":13229,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.616181+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:41.616181+0000","last_peered":"2024-09-16T10:28:41.616181+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:41.616181+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32813056,"num_objects":8011,"num_object_clones":0,"num_object_copies":16022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9151,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3681},{"shards":"0,2","objects":1957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12467","reported_seq":12643,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:41.605153+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:41.605153+0000","last_peered":"2024-09-16T10:28:41.605153+0000","last_clean":"2024-09-16T10:28:41.605153+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:41.605153+0000","last_undegraded":"2024-09-16T10:28:41.605153+0000","last_fullsized":"2024-09-16T10:28:41.605153+0000","mapping_epoch":33,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32452608,"num_objects":7923,"num_object_clones":0,"num_object_copies":15846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8270,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12649","reported_seq":21153,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:45.046935+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:45.046935+0000","last_peered":"2024-09-16T10:28:45.046935+0000","last_clean":"2024-09-16T10:28:45.046935+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:45.046935+0000","last_undegraded":"2024-09-16T10:28:45.046935+0000","last_fullsized":"2024-09-16T10:28:45.046935+0000","mapping_epoch":43,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31998918,"num_objects":7814,"num_object_clones":0,"num_object_copies":15628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13004","reported_seq":13176,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.615217+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:41.615217+0000","last_peered":"2024-09-16T10:28:41.615217+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:41.615217+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32916422,"num_objects":8038,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13002,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3652},{"shards":"0,2","objects":1954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12680","reported_seq":12820,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.427803+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:46.427803+0000","last_peered":"2024-09-16T10:28:46.427803+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:46.427803+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31699910,"num_objects":7741,"num_object_clones":0,"num_object_copies":15482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1809},{"shards":"2","objects":3481}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12763","reported_seq":12884,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.427697+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:46.427697+0000","last_peered":"2024-09-16T10:28:46.427697+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:46.427697+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32003014,"num_objects":7815,"num_object_clones":0,"num_object_copies":15630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12761,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1720},{"shards":"2","objects":3640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12270","reported_seq":12458,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.603022+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:41.603022+0000","last_peered":"2024-09-16T10:28:41.603022+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:41.603022+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2200","ondisk_log_start":"35'2200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31801827,"num_objects":7765,"num_object_clones":0,"num_object_copies":15530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12269,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3660},{"shards":"0,2","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12870","reported_seq":13074,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.552342+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:41.552342+0000","last_peered":"2024-09-16T10:28:41.552342+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:41.552342+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32949673,"num_objects":8047,"num_object_clones":0,"num_object_copies":16094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12867,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3745},{"shards":"0,2","objects":1928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12387","reported_seq":12556,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.581598+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:41.581598+0000","last_peered":"2024-09-16T10:28:41.581598+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:41.581598+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2300","ondisk_log_start":"35'2300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31757254,"num_objects":7755,"num_object_clones":0,"num_object_copies":15510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3632},{"shards":"0,2","objects":1852}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'12915","reported_seq":13102,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:41.561028+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:41.561028+0000","last_peered":"2024-09-16T10:28:41.561028+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:41.561028+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32692108,"num_objects":7985,"num_object_clones":0,"num_object_copies":15970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12912,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3637},{"shards":"0,2","objects":1934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":516256270,"num_objects":126056,"num_object_clones":0,"num_object_copies":252112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42543,"num_objects_misplaced":804,"num_objects_unfound":0,"num_objects_dirty":126056,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180249,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7999,"num_bytes_recovered":31072739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1162182656,"data_stored":1162091413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018955,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679536,"kb_used_data":387048,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93692304,"statfs":{"total":96636764160,"available":95940919296,"internally_reserved":0,"allocated":396337152,"data_stored":396079446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,2,3,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.476}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.3600000000000003},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":7.9589999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691900,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580776,"kb_used_data":300576,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93791064,"statfs":{"total":96636764160,"available":96042049536,"internally_reserved":0,"allocated":307789824,"data_stored":307459858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":286912170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.442}]}]},{"osd":0,"up_from":24,"seq":103079215182,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":814152,"kb_used_data":449536,"kb_used_omap":4,"kb_used_meta":364603,"kb_avail":93557688,"statfs":{"total":96636764160,"available":95803072512,"internally_reserved":0,"allocated":460324864,"data_stored":460074844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":373354152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,0,2,0,0,0,0,0,3],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.431},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.432}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.9119999999999999},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.6970000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":459063296,"data_stored":459059683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307380224,"data_stored":307300279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":395739136,"data_stored":395731451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:48.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:48.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'12518", 'reported_seq': 19364, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:41.658163+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:41.658163+0000', 'last_peered': '2024-09-16T10:28:41.658163+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:41.658163+0000', 'last_undegraded': '2024-09-16T10:28:41.658163+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'2500", 'ondisk_log_start': "35'2500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32563200, 'num_objects': 7950, 'num_object_clones': 0, 'num_object_copies': 15900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 804, 'num_objects_unfound': 0, 'num_objects_dirty': 7950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12518, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3388, 'num_bytes_recovered': 13877248, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7146}, {'shards': '0,2', 'objects': 804}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:48.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:48.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'12832", 'reported_seq': 13065, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:46.457637+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:46.457637+0000', 'last_peered': '2024-09-16T10:28:46.457637+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:46.457637+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'2800", 'ondisk_log_start': "35'2800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32075800, 'num_objects': 7832, 'num_object_clones': 0, 'num_object_copies': 15664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7832, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8889, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1837}, {'shards': '2', 'objects': 3520}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:48.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:48.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'12449", 'reported_seq': 12680, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.667141+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:41.667141+0000', 'last_peered': '2024-09-16T10:28:41.667141+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:41.667141+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'2400", 'ondisk_log_start': "35'2400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32448512, 'num_objects': 7922, 'num_object_clones': 0, 'num_object_copies': 15844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4184, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9074, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 35, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3740}, {'shards': '0,2', 'objects': 1956}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:48.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:48.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'12909", 'reported_seq': 13229, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.616181+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:41.616181+0000', 'last_peered': '2024-09-16T10:28:41.616181+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:41.616181+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'2900", 'ondisk_log_start': "35'2900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32813056, 'num_objects': 8011, 'num_object_clones': 0, 'num_object_copies': 16022, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4332, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9151, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3681}, {'shards': '0,2', 'objects': 1957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:48.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:48.557 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'13004", 'reported_seq': 13176, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.615217+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:41.615217+0000', 'last_peered': '2024-09-16T10:28:41.615217+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:41.615217+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'3000", 'ondisk_log_start': "35'3000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32916422, 'num_objects': 8038, 'num_object_clones': 0, 'num_object_copies': 16076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4388, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13002, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 50, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3652}, {'shards': '0,2', 'objects': 1954}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:48.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:48.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'12680", 'reported_seq': 12820, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:46.427803+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:46.427803+0000', 'last_peered': '2024-09-16T10:28:46.427803+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:46.427803+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'2600", 'ondisk_log_start': "35'2600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31699910, 'num_objects': 7741, 'num_object_clones': 0, 'num_object_copies': 15482, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4262, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12678, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1809}, {'shards': '2', 'objects': 3481}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:48.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:48.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'12763", 'reported_seq': 12884, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:46.427697+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:46.427697+0000', 'last_peered': '2024-09-16T10:28:46.427697+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:46.427697+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'2700", 'ondisk_log_start': "35'2700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32003014, 'num_objects': 7815, 'num_object_clones': 0, 'num_object_copies': 15630, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4177, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12761, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1720}, {'shards': '2', 'objects': 3640}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:48.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:48.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'12270", 'reported_seq': 12458, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.603022+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:41.603022+0000', 'last_peered': '2024-09-16T10:28:41.603022+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:41.603022+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'2200", 'ondisk_log_start': "35'2200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31801827, 'num_objects': 7765, 'num_object_clones': 0, 'num_object_copies': 15530, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4106, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12269, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3660}, {'shards': '0,2', 'objects': 1907}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:48.558 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:48.559 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'12870", 'reported_seq': 13074, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.552342+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:41.552342+0000', 'last_peered': '2024-09-16T10:28:41.552342+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:41.552342+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'2800", 'ondisk_log_start': "35'2800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32949673, 'num_objects': 8047, 'num_object_clones': 0, 'num_object_copies': 16094, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12867, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 38, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3745}, {'shards': '0,2', 'objects': 1928}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:48.559 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:48.559 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'12387", 'reported_seq': 12556, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.581598+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:41.581598+0000', 'last_peered': '2024-09-16T10:28:41.581598+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:41.581598+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'2300", 'ondisk_log_start': "35'2300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31757254, 'num_objects': 7755, 'num_object_clones': 0, 'num_object_copies': 15510, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4125, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12385, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3632}, {'shards': '0,2', 'objects': 1852}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:48.559 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:48.559 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'12915", 'reported_seq': 13102, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:41.561028+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:41.561028+0000', 'last_peered': '2024-09-16T10:28:41.561028+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:41.561028+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'2900", 'ondisk_log_start': "35'2900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32692108, 'num_objects': 7985, 'num_object_clones': 0, 'num_object_copies': 15970, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4351, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12912, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 52, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3637}, {'shards': '0,2', 'objects': 1934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:48.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:49.222 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:49.222 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42531/249496 objects degraded (17.047%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":418,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":124748,"data_bytes":510890529,"bytes_used":2109501440,"bytes_avail":287800791040,"bytes_total":289910292480,"degraded_objects":42531,"degraded_total":249496,"degraded_ratio":0.17046766280822137,"misplaced_objects":623,"misplaced_total":249496,"misplaced_ratio":0.0024970340205855003,"recovering_objects_per_sec":31,"recovering_bytes_per_sec":117039,"recovering_keys_per_sec":0,"num_objects_recovered":378,"num_bytes_recovered":1404928,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":354},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:49.256 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:49.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:49.782 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:49.784 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:49.811 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-16T10:28:48.674805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":510890529,"num_objects":124748,"num_object_clones":0,"num_object_copies":249496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42531,"num_objects_misplaced":623,"num_objects_unfound":0,"num_objects_dirty":124748,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181561,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8193,"num_bytes_recovered":31814134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160797,"ondisk_log_size":160797,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2060060,"kb_used_data":1130948,"kb_used_omap":12,"kb_used_meta":929075,"kb_avail":281055460,"statfs":{"total":289910292480,"available":287800791040,"internally_reserved":0,"allocated":1158090752,"data_stored":1157253060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":951372802},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,6,9,8,1,0,0,0,0,9],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17428480,"num_objects":-4255,"num_object_clones":0,"num_object_copies":-8510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-35,"num_objects_misplaced":-343,"num_objects_unfound":0,"num_objects_dirty":-4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4255,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":1404928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003839"},"pg_stats":[{"pgid":"2.f","version":"44'12653","reported_seq":19860,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:46.667753+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:46.667753+0000","last_peered":"2024-09-16T10:28:46.667753+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:46.667753+0000","last_undegraded":"2024-09-16T10:28:46.667753+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32010240,"num_objects":7815,"num_object_clones":0,"num_object_copies":15630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":623,"num_objects_unfound":0,"num_objects_dirty":7815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12653,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3569,"num_bytes_recovered":14618624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7192},{"shards":"0,2","objects":623}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12677","reported_seq":12835,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:46.406756+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:46.406756+0000","last_peered":"2024-09-16T10:28:46.406756+0000","last_clean":"2024-09-16T10:28:46.406756+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:46.406756+0000","last_undegraded":"2024-09-16T10:28:46.406756+0000","last_fullsized":"2024-09-16T10:28:46.406756+0000","mapping_epoch":34,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31887360,"num_objects":7785,"num_object_clones":0,"num_object_copies":15570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12832","reported_seq":13065,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.457637+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:46.457637+0000","last_peered":"2024-09-16T10:28:46.457637+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:46.457637+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32075800,"num_objects":7832,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8889,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1837},{"shards":"2","objects":3520}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12718","reported_seq":12775,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:46.410229+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:46.410229+0000","last_peered":"2024-09-16T10:28:46.410229+0000","last_clean":"2024-09-16T10:28:46.410229+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:46.410229+0000","last_undegraded":"2024-09-16T10:28:46.410229+0000","last_fullsized":"2024-09-16T10:28:46.410229+0000","mapping_epoch":34,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":7744,"num_object_clones":0,"num_object_copies":15488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12718,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12702","reported_seq":12859,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:46.660660+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:46.660660+0000","last_peered":"2024-09-16T10:28:46.660660+0000","last_clean":"2024-09-16T10:28:46.660660+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:46.660660+0000","last_undegraded":"2024-09-16T10:28:46.660660+0000","last_fullsized":"2024-09-16T10:28:46.660660+0000","mapping_epoch":33,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31961088,"num_objects":7803,"num_object_clones":0,"num_object_copies":15606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9344,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12600","reported_seq":12840,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.651218+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:46.651218+0000","last_peered":"2024-09-16T10:28:46.651218+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:46.651218+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31830016,"num_objects":7771,"num_object_clones":0,"num_object_copies":15542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3592},{"shards":"0,2","objects":1805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'13058","reported_seq":13382,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.689198+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:46.689198+0000","last_peered":"2024-09-16T10:28:46.689198+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:46.689198+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32202752,"num_objects":7862,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9300,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3534},{"shards":"0,2","objects":1808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12587","reported_seq":12763,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:46.629024+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:46.629024+0000","last_peered":"2024-09-16T10:28:46.629024+0000","last_clean":"2024-09-16T10:28:46.629024+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:46.629024+0000","last_undegraded":"2024-09-16T10:28:46.629024+0000","last_fullsized":"2024-09-16T10:28:46.629024+0000","mapping_epoch":33,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31961088,"num_objects":7803,"num_object_clones":0,"num_object_copies":15606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8390,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12649","reported_seq":21153,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:45.046935+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:45.046935+0000","last_peered":"2024-09-16T10:28:45.046935+0000","last_clean":"2024-09-16T10:28:45.046935+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:45.046935+0000","last_undegraded":"2024-09-16T10:28:45.046935+0000","last_fullsized":"2024-09-16T10:28:45.046935+0000","mapping_epoch":43,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31998918,"num_objects":7814,"num_object_clones":0,"num_object_copies":15628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13119","reported_seq":13298,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.606008+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:46.606008+0000","last_peered":"2024-09-16T10:28:46.606008+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:46.606008+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32445382,"num_objects":7923,"num_object_clones":0,"num_object_copies":15846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4386,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13117,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3539},{"shards":"0,2","objects":1839}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":88,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942549+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:01.942549+0000","last_peered":"2024-09-16T10:26:01.942549+0000","last_clean":"2024-09-16T10:26:01.942549+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:01.942549+0000","last_undegraded":"2024-09-16T10:26:01.942549+0000","last_fullsized":"2024-09-16T10:26:01.942549+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12680","reported_seq":12820,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.427803+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:46.427803+0000","last_peered":"2024-09-16T10:28:46.427803+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:46.427803+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31699910,"num_objects":7741,"num_object_clones":0,"num_object_copies":15482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1809},{"shards":"2","objects":3481}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12763","reported_seq":12884,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.427697+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:46.427697+0000","last_peered":"2024-09-16T10:28:46.427697+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:46.427697+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32003014,"num_objects":7815,"num_object_clones":0,"num_object_copies":15630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12761,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1720},{"shards":"2","objects":3640}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12417","reported_seq":12608,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.650022+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:46.650022+0000","last_peered":"2024-09-16T10:28:46.650022+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:46.650022+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2400","ondisk_log_start":"35'2400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31199715,"num_objects":7618,"num_object_clones":0,"num_object_copies":15236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12416,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3515},{"shards":"0,2","objects":1760}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'12986","reported_seq":13192,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.561842+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:46.561842+0000","last_peered":"2024-09-16T10:28:46.561842+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:46.561842+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32474537,"num_objects":7931,"num_object_clones":0,"num_object_copies":15862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12983,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3630},{"shards":"0,2","objects":1812}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12515","reported_seq":12686,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.465061+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:46.465061+0000","last_peered":"2024-09-16T10:28:46.465061+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:46.465061+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31232966,"num_objects":7627,"num_object_clones":0,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12513,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3505},{"shards":"0,2","objects":1724}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'13038","reported_seq":13227,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:46.679897+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:46.679897+0000","last_peered":"2024-09-16T10:28:46.679897+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:46.679897+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32188300,"num_objects":7862,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13035,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3515},{"shards":"0,2","objects":1811}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":510890510,"num_objects":124746,"num_object_clones":0,"num_object_copies":249492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42531,"num_objects_misplaced":623,"num_objects_unfound":0,"num_objects_dirty":124746,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181559,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8192,"num_bytes_recovered":31814115,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155805184,"data_stored":1155713941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160794,"ondisk_log_size":160794,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018955,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679536,"kb_used_data":387048,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93692304,"statfs":{"total":96636764160,"available":95940919296,"internally_reserved":0,"allocated":396337152,"data_stored":396079446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,2,3,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":2.476}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.3600000000000003},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":7.9589999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691900,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580776,"kb_used_data":300576,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93791064,"statfs":{"total":96636764160,"available":96042049536,"internally_reserved":0,"allocated":307789824,"data_stored":307459858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":286912170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.68899999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.442}]}]},{"osd":0,"up_from":24,"seq":103079215183,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":799748,"kb_used_data":443324,"kb_used_omap":4,"kb_used_meta":356411,"kb_avail":93572092,"statfs":{"total":96636764160,"available":95817822208,"internally_reserved":0,"allocated":453963776,"data_stored":453713756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":364965544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,4,4,1,0,0,0,0,8],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.8440000000000001},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":2.0790000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.8849999999999998},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":1.8720000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":452685824,"data_stored":452682211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307380224,"data_stored":307300279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":395739136,"data_stored":395731451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:52.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:53.329 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:53.331 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:53.354 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-16T10:28:52.675773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502317601,"num_objects":122655,"num_object_clones":0,"num_object_copies":245310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42489,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":122655,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183654,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8412,"num_bytes_recovered":32539126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2045308,"kb_used_data":1120280,"kb_used_omap":12,"kb_used_meta":924979,"kb_avail":281070212,"statfs":{"total":289910292480,"available":287815897088,"internally_reserved":0,"allocated":1147166720,"data_stored":1146329028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":947178498},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,4,6,0,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25403392,"num_objects":-6202,"num_object_clones":0,"num_object_copies":-12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":-520,"num_objects_unfound":0,"num_objects_dirty":-6202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6202,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":2129920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"2.f","version":"44'12775","reported_seq":20337,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:51.698605+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:51.698605+0000","last_peered":"2024-09-16T10:28:51.698605+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:51.698605+0000","last_undegraded":"2024-09-16T10:28:51.698605+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31510528,"num_objects":7693,"num_object_clones":0,"num_object_copies":15386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":7693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3746,"num_bytes_recovered":15343616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7247},{"shards":"0,2","objects":446}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12809","reported_seq":12967,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.390425+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:51.390425+0000","last_peered":"2024-09-16T10:28:51.390425+0000","last_clean":"2024-09-16T10:28:51.390425+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:51.390425+0000","last_undegraded":"2024-09-16T10:28:51.390425+0000","last_fullsized":"2024-09-16T10:28:51.390425+0000","mapping_epoch":34,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31346688,"num_objects":7653,"num_object_clones":0,"num_object_copies":15306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12950","reported_seq":13185,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.445266+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:51.445266+0000","last_peered":"2024-09-16T10:28:51.445266+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:51.445266+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31592472,"num_objects":7714,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7714,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9007,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1719},{"shards":"2","objects":3403}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12846","reported_seq":12903,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.349977+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:51.349977+0000","last_peered":"2024-09-16T10:28:51.349977+0000","last_clean":"2024-09-16T10:28:51.349977+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:51.349977+0000","last_undegraded":"2024-09-16T10:28:51.349977+0000","last_fullsized":"2024-09-16T10:28:51.349977+0000","mapping_epoch":34,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":7616,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12846,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12842","reported_seq":12998,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.701213+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:51.701213+0000","last_peered":"2024-09-16T10:28:51.701213+0000","last_clean":"2024-09-16T10:28:51.701213+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:51.701213+0000","last_undegraded":"2024-09-16T10:28:51.701213+0000","last_fullsized":"2024-09-16T10:28:51.701213+0000","mapping_epoch":33,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31387648,"num_objects":7663,"num_object_clones":0,"num_object_copies":15326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9484,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12740","reported_seq":12992,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.651294+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:51.651294+0000","last_peered":"2024-09-16T10:28:51.651294+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:51.651294+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31256576,"num_objects":7631,"num_object_clones":0,"num_object_copies":15262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9365,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3458},{"shards":"0,2","objects":1665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'13193","reported_seq":13531,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.660745+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:51.660745+0000","last_peered":"2024-09-16T10:28:51.660745+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:51.660745+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31649792,"num_objects":7727,"num_object_clones":0,"num_object_copies":15454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9435,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3406},{"shards":"0,2","objects":1673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12716","reported_seq":12892,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.673555+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:51.673555+0000","last_peered":"2024-09-16T10:28:51.673555+0000","last_clean":"2024-09-16T10:28:51.673555+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:51.673555+0000","last_undegraded":"2024-09-16T10:28:51.673555+0000","last_fullsized":"2024-09-16T10:28:51.673555+0000","mapping_epoch":33,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31432704,"num_objects":7674,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12770","reported_seq":21274,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:50.185585+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:50.185585+0000","last_peered":"2024-09-16T10:28:50.185585+0000","last_clean":"2024-09-16T10:28:50.185585+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:50.185585+0000","last_undegraded":"2024-09-16T10:28:50.185585+0000","last_fullsized":"2024-09-16T10:28:50.185585+0000","mapping_epoch":43,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31503302,"num_objects":7693,"num_object_clones":0,"num_object_copies":15386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13265","reported_seq":13458,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.701063+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:51.701063+0000","last_peered":"2024-09-16T10:28:51.701063+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:51.701063+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31847366,"num_objects":7777,"num_object_clones":0,"num_object_copies":15554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13263,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3402},{"shards":"0,2","objects":1693}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.501852+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:45.501852+0000","last_peered":"2024-09-16T10:26:45.501852+0000","last_clean":"2024-09-16T10:26:45.501852+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:45.501852+0000","last_undegraded":"2024-09-16T10:26:45.501852+0000","last_fullsized":"2024-09-16T10:26:45.501852+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12819","reported_seq":12965,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.469249+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:51.469249+0000","last_peered":"2024-09-16T10:28:51.469249+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:51.469249+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31130566,"num_objects":7602,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12817,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1670},{"shards":"2","objects":3345}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12893","reported_seq":13016,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.472952+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:51.472952+0000","last_peered":"2024-09-16T10:28:51.472952+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:51.472952+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31470534,"num_objects":7685,"num_object_clones":0,"num_object_copies":15370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1590},{"shards":"2","objects":3511}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12538","reported_seq":12734,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.595960+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:51.595960+0000","last_peered":"2024-09-16T10:28:51.595960+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:51.595960+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30704099,"num_objects":7497,"num_object_clones":0,"num_object_copies":14994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12537,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3396},{"shards":"0,2","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'13105","reported_seq":13325,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.581954+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:51.581954+0000","last_peered":"2024-09-16T10:28:51.581954+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:51.581954+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31987113,"num_objects":7812,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13102,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3518},{"shards":"0,2","objects":1693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12659","reported_seq":12836,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.695673+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:51.695673+0000","last_peered":"2024-09-16T10:28:51.695673+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:51.695673+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30643142,"num_objects":7483,"num_object_clones":0,"num_object_copies":14966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12657,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3364},{"shards":"0,2","objects":1580}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'13167","reported_seq":13362,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.580389+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:51.580389+0000","last_peered":"2024-09-16T10:28:51.580389+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:51.580389+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31659916,"num_objects":7733,"num_object_clones":0,"num_object_copies":15466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13164,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3389},{"shards":"0,2","objects":1682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":502317582,"num_objects":122653,"num_object_clones":0,"num_object_copies":245306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42489,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":122653,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183652,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8411,"num_bytes_recovered":32539107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1144713216,"data_stored":1144621973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160787,"ondisk_log_size":160787,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018956,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663168,"kb_used_data":382960,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93708672,"statfs":{"total":96636764160,"available":95957680128,"internally_reserved":0,"allocated":392151040,"data_stored":391893334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":286912176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.6399999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.40999999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.599},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":1.677}]}]},{"osd":1,"up_from":40,"seq":171798691901,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584600,"kb_used_data":300304,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93787240,"statfs":{"total":96636764160,"available":96038133760,"internally_reserved":0,"allocated":307511296,"data_stored":307181330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":291106474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.4580000000000002},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":8.0519999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":10.581},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.2370000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215184,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797540,"kb_used_data":437016,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93574300,"statfs":{"total":96636764160,"available":95820083200,"internally_reserved":0,"allocated":447504384,"data_stored":447254364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.5029999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.5059999999999998},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":446189568,"data_stored":446185955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307093504,"data_stored":307013559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391430144,"data_stored":391422459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:53.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:53.897 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:53.897 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:53.919 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-16T10:28:52.675773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502317601,"num_objects":122655,"num_object_clones":0,"num_object_copies":245310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42489,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":122655,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183654,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8412,"num_bytes_recovered":32539126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2045308,"kb_used_data":1120280,"kb_used_omap":12,"kb_used_meta":924979,"kb_avail":281070212,"statfs":{"total":289910292480,"available":287815897088,"internally_reserved":0,"allocated":1147166720,"data_stored":1146329028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":947178498},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,4,6,0,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25403392,"num_objects":-6202,"num_object_clones":0,"num_object_copies":-12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":-520,"num_objects_unfound":0,"num_objects_dirty":-6202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6202,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":597,"num_bytes_recovered":2129920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"2.f","version":"44'12775","reported_seq":20337,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:51.698605+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:51.698605+0000","last_peered":"2024-09-16T10:28:51.698605+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:51.698605+0000","last_undegraded":"2024-09-16T10:28:51.698605+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31510528,"num_objects":7693,"num_object_clones":0,"num_object_copies":15386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":7693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3746,"num_bytes_recovered":15343616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7247},{"shards":"0,2","objects":446}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12809","reported_seq":12967,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.390425+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:51.390425+0000","last_peered":"2024-09-16T10:28:51.390425+0000","last_clean":"2024-09-16T10:28:51.390425+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:51.390425+0000","last_undegraded":"2024-09-16T10:28:51.390425+0000","last_fullsized":"2024-09-16T10:28:51.390425+0000","mapping_epoch":34,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31346688,"num_objects":7653,"num_object_clones":0,"num_object_copies":15306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12950","reported_seq":13185,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.445266+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:51.445266+0000","last_peered":"2024-09-16T10:28:51.445266+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:51.445266+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31592472,"num_objects":7714,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7714,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9007,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1719},{"shards":"2","objects":3403}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12846","reported_seq":12903,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.349977+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:51.349977+0000","last_peered":"2024-09-16T10:28:51.349977+0000","last_clean":"2024-09-16T10:28:51.349977+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:51.349977+0000","last_undegraded":"2024-09-16T10:28:51.349977+0000","last_fullsized":"2024-09-16T10:28:51.349977+0000","mapping_epoch":34,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":7616,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12846,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12842","reported_seq":12998,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.701213+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:51.701213+0000","last_peered":"2024-09-16T10:28:51.701213+0000","last_clean":"2024-09-16T10:28:51.701213+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:51.701213+0000","last_undegraded":"2024-09-16T10:28:51.701213+0000","last_fullsized":"2024-09-16T10:28:51.701213+0000","mapping_epoch":33,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31387648,"num_objects":7663,"num_object_clones":0,"num_object_copies":15326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9484,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12740","reported_seq":12992,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.651294+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:51.651294+0000","last_peered":"2024-09-16T10:28:51.651294+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:51.651294+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31256576,"num_objects":7631,"num_object_clones":0,"num_object_copies":15262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9365,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3458},{"shards":"0,2","objects":1665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'13193","reported_seq":13531,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.660745+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:51.660745+0000","last_peered":"2024-09-16T10:28:51.660745+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:51.660745+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31649792,"num_objects":7727,"num_object_clones":0,"num_object_copies":15454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9435,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3406},{"shards":"0,2","objects":1673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12716","reported_seq":12892,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.673555+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:51.673555+0000","last_peered":"2024-09-16T10:28:51.673555+0000","last_clean":"2024-09-16T10:28:51.673555+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:51.673555+0000","last_undegraded":"2024-09-16T10:28:51.673555+0000","last_fullsized":"2024-09-16T10:28:51.673555+0000","mapping_epoch":33,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31432704,"num_objects":7674,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12770","reported_seq":21274,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:50.185585+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:50.185585+0000","last_peered":"2024-09-16T10:28:50.185585+0000","last_clean":"2024-09-16T10:28:50.185585+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:50.185585+0000","last_undegraded":"2024-09-16T10:28:50.185585+0000","last_fullsized":"2024-09-16T10:28:50.185585+0000","mapping_epoch":43,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31503302,"num_objects":7693,"num_object_clones":0,"num_object_copies":15386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13265","reported_seq":13458,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.701063+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:51.701063+0000","last_peered":"2024-09-16T10:28:51.701063+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:51.701063+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31847366,"num_objects":7777,"num_object_clones":0,"num_object_copies":15554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13263,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3402},{"shards":"0,2","objects":1693}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.501852+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:45.501852+0000","last_peered":"2024-09-16T10:26:45.501852+0000","last_clean":"2024-09-16T10:26:45.501852+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:45.501852+0000","last_undegraded":"2024-09-16T10:26:45.501852+0000","last_fullsized":"2024-09-16T10:26:45.501852+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12819","reported_seq":12965,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.469249+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:51.469249+0000","last_peered":"2024-09-16T10:28:51.469249+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:51.469249+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31130566,"num_objects":7602,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12817,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1670},{"shards":"2","objects":3345}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12893","reported_seq":13016,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.472952+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:51.472952+0000","last_peered":"2024-09-16T10:28:51.472952+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:51.472952+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31470534,"num_objects":7685,"num_object_clones":0,"num_object_copies":15370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1590},{"shards":"2","objects":3511}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12538","reported_seq":12734,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.595960+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:51.595960+0000","last_peered":"2024-09-16T10:28:51.595960+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:51.595960+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30704099,"num_objects":7497,"num_object_clones":0,"num_object_copies":14994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12537,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3396},{"shards":"0,2","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'13105","reported_seq":13325,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.581954+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:51.581954+0000","last_peered":"2024-09-16T10:28:51.581954+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:51.581954+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31987113,"num_objects":7812,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13102,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3518},{"shards":"0,2","objects":1693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12659","reported_seq":12836,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.695673+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:51.695673+0000","last_peered":"2024-09-16T10:28:51.695673+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:51.695673+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30643142,"num_objects":7483,"num_object_clones":0,"num_object_copies":14966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12657,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3364},{"shards":"0,2","objects":1580}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'13167","reported_seq":13362,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.580389+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:51.580389+0000","last_peered":"2024-09-16T10:28:51.580389+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:51.580389+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31659916,"num_objects":7733,"num_object_clones":0,"num_object_copies":15466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13164,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3389},{"shards":"0,2","objects":1682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":502317582,"num_objects":122653,"num_object_clones":0,"num_object_copies":245306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42489,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":122653,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183652,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8411,"num_bytes_recovered":32539107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1144713216,"data_stored":1144621973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160787,"ondisk_log_size":160787,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018956,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663168,"kb_used_data":382960,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93708672,"statfs":{"total":96636764160,"available":95957680128,"internally_reserved":0,"allocated":392151040,"data_stored":391893334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":286912176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.6399999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.40999999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.599},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":1.677}]}]},{"osd":1,"up_from":40,"seq":171798691901,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584600,"kb_used_data":300304,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93787240,"statfs":{"total":96636764160,"available":96038133760,"internally_reserved":0,"allocated":307511296,"data_stored":307181330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":291106474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.4580000000000002},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":8.0519999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":10.581},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.2370000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215184,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797540,"kb_used_data":437016,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93574300,"statfs":{"total":96636764160,"available":95820083200,"internally_reserved":0,"allocated":447504384,"data_stored":447254364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.5029999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.5059999999999998},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":446189568,"data_stored":446185955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307093504,"data_stored":307013559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391430144,"data_stored":391422459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:53.920 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:53.920 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'12775", 'reported_seq': 20337, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:51.698605+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:51.698605+0000', 'last_peered': '2024-09-16T10:28:51.698605+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:51.698605+0000', 'last_undegraded': '2024-09-16T10:28:51.698605+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'2700", 'ondisk_log_start': "35'2700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31510528, 'num_objects': 7693, 'num_object_clones': 0, 'num_object_copies': 15386, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 446, 'num_objects_unfound': 0, 'num_objects_dirty': 7693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12775, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3746, 'num_bytes_recovered': 15343616, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7247}, {'shards': '0,2', 'objects': 446}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:53.920 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:53.920 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'12950", 'reported_seq': 13185, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:51.445266+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:51.445266+0000', 'last_peered': '2024-09-16T10:28:51.445266+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:51.445266+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'2900", 'ondisk_log_start': "35'2900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31592472, 'num_objects': 7714, 'num_object_clones': 0, 'num_object_copies': 15428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7714, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9007, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1719}, {'shards': '2', 'objects': 3403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:53.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:53.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'12740", 'reported_seq': 12992, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:51.651294+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:51.651294+0000', 'last_peered': '2024-09-16T10:28:51.651294+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:51.651294+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'2700", 'ondisk_log_start': "35'2700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31256576, 'num_objects': 7631, 'num_object_clones': 0, 'num_object_copies': 15262, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4175, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9365, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 44, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3458}, {'shards': '0,2', 'objects': 1665}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:53.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:53.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'13193", 'reported_seq': 13531, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:51.660745+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:51.660745+0000', 'last_peered': '2024-09-16T10:28:51.660745+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:51.660745+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'3100", 'ondisk_log_start': "35'3100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31649792, 'num_objects': 7727, 'num_object_clones': 0, 'num_object_copies': 15454, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4323, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9435, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 82, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3406}, {'shards': '0,2', 'objects': 1673}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:53.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:53.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'13265", 'reported_seq': 13458, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:51.701063+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:51.701063+0000', 'last_peered': '2024-09-16T10:28:51.701063+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:51.701063+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'3200", 'ondisk_log_start': "35'3200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31847366, 'num_objects': 7777, 'num_object_clones': 0, 'num_object_copies': 15554, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4377, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13263, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 61, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3402}, {'shards': '0,2', 'objects': 1693}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:53.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:53.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'12819", 'reported_seq': 12965, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:51.469249+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:51.469249+0000', 'last_peered': '2024-09-16T10:28:51.469249+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:51.469249+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'2800", 'ondisk_log_start': "35'2800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31130566, 'num_objects': 7602, 'num_object_clones': 0, 'num_object_copies': 15204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12817, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1670}, {'shards': '2', 'objects': 3345}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:53.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:53.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'12893", 'reported_seq': 13016, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:51.472952+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:51.472952+0000', 'last_peered': '2024-09-16T10:28:51.472952+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:51.472952+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'2800", 'ondisk_log_start': "35'2800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31470534, 'num_objects': 7685, 'num_object_clones': 0, 'num_object_copies': 15370, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4176, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12891, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1590}, {'shards': '2', 'objects': 3511}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:53.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:53.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'12538", 'reported_seq': 12734, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:51.595960+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:51.595960+0000', 'last_peered': '2024-09-16T10:28:51.595960+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:51.595960+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'2500", 'ondisk_log_start': "35'2500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30704099, 'num_objects': 7497, 'num_object_clones': 0, 'num_object_copies': 14994, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4102, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12537, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 59, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3396}, {'shards': '0,2', 'objects': 1639}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:53.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:53.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'13105", 'reported_seq': 13325, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:51.581954+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:51.581954+0000', 'last_peered': '2024-09-16T10:28:51.581954+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:51.581954+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'3100", 'ondisk_log_start': "35'3100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31987113, 'num_objects': 7812, 'num_object_clones': 0, 'num_object_copies': 15624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4297, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13102, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3518}, {'shards': '0,2', 'objects': 1693}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:53.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:53.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'12659", 'reported_seq': 12836, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:51.695673+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:51.695673+0000', 'last_peered': '2024-09-16T10:28:51.695673+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:51.695673+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'2600", 'ondisk_log_start': "35'2600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30643142, 'num_objects': 7483, 'num_object_clones': 0, 'num_object_copies': 14966, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12657, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 50, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3364}, {'shards': '0,2', 'objects': 1580}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:53.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:53.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'13167", 'reported_seq': 13362, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:51.580389+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:51.580389+0000', 'last_peered': '2024-09-16T10:28:51.580389+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:51.580389+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'3100", 'ondisk_log_start': "35'3100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31659916, 'num_objects': 7733, 'num_object_clones': 0, 'num_object_copies': 15466, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4347, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13164, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3389}, {'shards': '0,2', 'objects': 1682}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:53.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:54.538 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:54.538 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42489/245310 objects degraded (17.321%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":423,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":122655,"data_bytes":502317601,"bytes_used":2094395392,"bytes_avail":287815897088,"bytes_total":289910292480,"degraded_objects":42489,"degraded_total":245310,"degraded_ratio":0.17320533202886143,"misplaced_objects":446,"misplaced_total":245310,"misplaced_ratio":0.0018181077004606415,"recovering_objects_per_sec":49,"recovering_bytes_per_sec":177447,"recovering_keys_per_sec":0,"num_objects_recovered":597,"num_bytes_recovered":2129920,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":516},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:54.560 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:54.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:55.119 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:55.119 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:55.154 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-16T10:28:54.676301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502317601,"num_objects":122655,"num_object_clones":0,"num_object_copies":245310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42489,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":122655,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183654,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8412,"num_bytes_recovered":32539126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160790,"ondisk_log_size":160790,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2045308,"kb_used_data":1120280,"kb_used_omap":12,"kb_used_meta":924979,"kb_avail":281070212,"statfs":{"total":289910292480,"available":287815897088,"internally_reserved":0,"allocated":1147166720,"data_stored":1146329028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":947178498},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,4,6,0,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17199104,"num_objects":-4199,"num_object_clones":0,"num_object_copies":-8398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-59,"num_objects_misplaced":-358,"num_objects_unfound":0,"num_objects_dirty":-4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003105"},"pg_stats":[{"pgid":"2.f","version":"44'12775","reported_seq":20337,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:51.698605+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:51.698605+0000","last_peered":"2024-09-16T10:28:51.698605+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:51.698605+0000","last_undegraded":"2024-09-16T10:28:51.698605+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31510528,"num_objects":7693,"num_object_clones":0,"num_object_copies":15386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":7693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3746,"num_bytes_recovered":15343616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7247},{"shards":"0,2","objects":446}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12809","reported_seq":12967,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.390425+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:51.390425+0000","last_peered":"2024-09-16T10:28:51.390425+0000","last_clean":"2024-09-16T10:28:51.390425+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:51.390425+0000","last_undegraded":"2024-09-16T10:28:51.390425+0000","last_fullsized":"2024-09-16T10:28:51.390425+0000","mapping_epoch":34,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31346688,"num_objects":7653,"num_object_clones":0,"num_object_copies":15306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'12950","reported_seq":13185,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.445266+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:51.445266+0000","last_peered":"2024-09-16T10:28:51.445266+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:51.445266+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31592472,"num_objects":7714,"num_object_clones":0,"num_object_copies":15428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7714,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9007,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1719},{"shards":"2","objects":3403}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12846","reported_seq":12903,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.349977+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:51.349977+0000","last_peered":"2024-09-16T10:28:51.349977+0000","last_clean":"2024-09-16T10:28:51.349977+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:51.349977+0000","last_undegraded":"2024-09-16T10:28:51.349977+0000","last_fullsized":"2024-09-16T10:28:51.349977+0000","mapping_epoch":34,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31195136,"num_objects":7616,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12846,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12842","reported_seq":12998,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.701213+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:51.701213+0000","last_peered":"2024-09-16T10:28:51.701213+0000","last_clean":"2024-09-16T10:28:51.701213+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:51.701213+0000","last_undegraded":"2024-09-16T10:28:51.701213+0000","last_fullsized":"2024-09-16T10:28:51.701213+0000","mapping_epoch":33,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31387648,"num_objects":7663,"num_object_clones":0,"num_object_copies":15326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9484,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12740","reported_seq":12992,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.651294+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:51.651294+0000","last_peered":"2024-09-16T10:28:51.651294+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:51.651294+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31256576,"num_objects":7631,"num_object_clones":0,"num_object_copies":15262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9365,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3458},{"shards":"0,2","objects":1665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'13193","reported_seq":13531,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.660745+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:51.660745+0000","last_peered":"2024-09-16T10:28:51.660745+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:51.660745+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31649792,"num_objects":7727,"num_object_clones":0,"num_object_copies":15454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9435,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3406},{"shards":"0,2","objects":1673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12716","reported_seq":12892,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.673555+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:51.673555+0000","last_peered":"2024-09-16T10:28:51.673555+0000","last_clean":"2024-09-16T10:28:51.673555+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:51.673555+0000","last_undegraded":"2024-09-16T10:28:51.673555+0000","last_fullsized":"2024-09-16T10:28:51.673555+0000","mapping_epoch":33,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31432704,"num_objects":7674,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12770","reported_seq":21274,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:50.185585+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:50.185585+0000","last_peered":"2024-09-16T10:28:50.185585+0000","last_clean":"2024-09-16T10:28:50.185585+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:50.185585+0000","last_undegraded":"2024-09-16T10:28:50.185585+0000","last_fullsized":"2024-09-16T10:28:50.185585+0000","mapping_epoch":43,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31503302,"num_objects":7693,"num_object_clones":0,"num_object_copies":15386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13265","reported_seq":13458,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.701063+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:51.701063+0000","last_peered":"2024-09-16T10:28:51.701063+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:51.701063+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31847366,"num_objects":7777,"num_object_clones":0,"num_object_copies":15554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13263,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3402},{"shards":"0,2","objects":1693}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.501852+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:45.501852+0000","last_peered":"2024-09-16T10:26:45.501852+0000","last_clean":"2024-09-16T10:26:45.501852+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:45.501852+0000","last_undegraded":"2024-09-16T10:26:45.501852+0000","last_fullsized":"2024-09-16T10:26:45.501852+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12819","reported_seq":12965,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.469249+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:51.469249+0000","last_peered":"2024-09-16T10:28:51.469249+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:51.469249+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31130566,"num_objects":7602,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12817,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1670},{"shards":"2","objects":3345}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'12893","reported_seq":13016,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.472952+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:51.472952+0000","last_peered":"2024-09-16T10:28:51.472952+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:51.472952+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31470534,"num_objects":7685,"num_object_clones":0,"num_object_copies":15370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12891,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1590},{"shards":"2","objects":3511}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12538","reported_seq":12734,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.595960+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:51.595960+0000","last_peered":"2024-09-16T10:28:51.595960+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:51.595960+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30704099,"num_objects":7497,"num_object_clones":0,"num_object_copies":14994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12537,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3396},{"shards":"0,2","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'13105","reported_seq":13325,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.581954+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:51.581954+0000","last_peered":"2024-09-16T10:28:51.581954+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:51.581954+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31987113,"num_objects":7812,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13102,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3518},{"shards":"0,2","objects":1693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12659","reported_seq":12836,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.695673+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:51.695673+0000","last_peered":"2024-09-16T10:28:51.695673+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:51.695673+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30643142,"num_objects":7483,"num_object_clones":0,"num_object_copies":14966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12657,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3364},{"shards":"0,2","objects":1580}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'13167","reported_seq":13362,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.580389+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:51.580389+0000","last_peered":"2024-09-16T10:28:51.580389+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:51.580389+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31659916,"num_objects":7733,"num_object_clones":0,"num_object_copies":15466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13164,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3389},{"shards":"0,2","objects":1682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":502317582,"num_objects":122653,"num_object_clones":0,"num_object_copies":245306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42489,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":122653,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":183652,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8411,"num_bytes_recovered":32539107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1144713216,"data_stored":1144621973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160787,"ondisk_log_size":160787,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018956,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663168,"kb_used_data":382960,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93708672,"statfs":{"total":96636764160,"available":95957680128,"internally_reserved":0,"allocated":392151040,"data_stored":391893334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":286912176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.6399999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.40999999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.599},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":1.677}]}]},{"osd":1,"up_from":40,"seq":171798691901,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584600,"kb_used_data":300304,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93787240,"statfs":{"total":96636764160,"available":96038133760,"internally_reserved":0,"allocated":307511296,"data_stored":307181330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":291106474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.4580000000000002},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":8.0519999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":10.581},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.2370000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215184,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797540,"kb_used_data":437016,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93574300,"statfs":{"total":96636764160,"available":95820083200,"internally_reserved":0,"allocated":447504384,"data_stored":447254364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.5029999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.5059999999999998},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":446189568,"data_stored":446185955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":307093504,"data_stored":307013559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391430144,"data_stored":391422459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:58.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:58.677 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:58.678 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:58.707 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-16T10:28:56.676784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":499180065,"num_objects":121889,"num_object_clones":0,"num_object_copies":243778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42483,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":121889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184420,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8418,"num_bytes_recovered":32539126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160756,"ondisk_log_size":160756,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2053324,"kb_used_data":1116016,"kb_used_omap":12,"kb_used_meta":937267,"kb_avail":281062196,"statfs":{"total":289910292480,"available":287807688704,"internally_reserved":0,"allocated":1142800384,"data_stored":1141962692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":959761410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,4,3,8,1,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20336640,"num_objects":-4965,"num_object_clones":0,"num_object_copies":-9930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-65,"num_objects_misplaced":-358,"num_objects_unfound":0,"num_objects_dirty":-4965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4965,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003048"},"pg_stats":[{"pgid":"2.f","version":"44'12775","reported_seq":20337,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:51.698605+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:51.698605+0000","last_peered":"2024-09-16T10:28:51.698605+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:51.698605+0000","last_undegraded":"2024-09-16T10:28:51.698605+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31510528,"num_objects":7693,"num_object_clones":0,"num_object_copies":15386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":7693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3746,"num_bytes_recovered":15343616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7247},{"shards":"0,2","objects":446}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12934","reported_seq":13092,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:56.399700+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:56.399700+0000","last_peered":"2024-09-16T10:28:56.399700+0000","last_clean":"2024-09-16T10:28:56.399700+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:56.399700+0000","last_undegraded":"2024-09-16T10:28:56.399700+0000","last_fullsized":"2024-09-16T10:28:56.399700+0000","mapping_epoch":34,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30834688,"num_objects":7528,"num_object_clones":0,"num_object_copies":15056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9147,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'13073","reported_seq":13312,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.452909+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:56.452909+0000","last_peered":"2024-09-16T10:28:56.452909+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:56.452909+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31088664,"num_objects":7591,"num_object_clones":0,"num_object_copies":15182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7591,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9130,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1596},{"shards":"2","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12964","reported_seq":13021,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:56.482260+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:56.482260+0000","last_peered":"2024-09-16T10:28:56.482260+0000","last_clean":"2024-09-16T10:28:56.482260+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:56.482260+0000","last_undegraded":"2024-09-16T10:28:56.482260+0000","last_fullsized":"2024-09-16T10:28:56.482260+0000","mapping_epoch":34,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711808,"num_objects":7498,"num_object_clones":0,"num_object_copies":14996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12964,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12842","reported_seq":12998,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.701213+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:51.701213+0000","last_peered":"2024-09-16T10:28:51.701213+0000","last_clean":"2024-09-16T10:28:51.701213+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:51.701213+0000","last_undegraded":"2024-09-16T10:28:51.701213+0000","last_fullsized":"2024-09-16T10:28:51.701213+0000","mapping_epoch":33,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31387648,"num_objects":7663,"num_object_clones":0,"num_object_copies":15326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9484,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12740","reported_seq":12992,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.651294+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:51.651294+0000","last_peered":"2024-09-16T10:28:51.651294+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:51.651294+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31256576,"num_objects":7631,"num_object_clones":0,"num_object_copies":15262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9365,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3458},{"shards":"0,2","objects":1665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'13193","reported_seq":13531,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.660745+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:51.660745+0000","last_peered":"2024-09-16T10:28:51.660745+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:51.660745+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31649792,"num_objects":7727,"num_object_clones":0,"num_object_copies":15454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9435,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3406},{"shards":"0,2","objects":1673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12716","reported_seq":12892,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:51.673555+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:51.673555+0000","last_peered":"2024-09-16T10:28:51.673555+0000","last_clean":"2024-09-16T10:28:51.673555+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:51.673555+0000","last_undegraded":"2024-09-16T10:28:51.673555+0000","last_fullsized":"2024-09-16T10:28:51.673555+0000","mapping_epoch":33,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31432704,"num_objects":7674,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8519,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12905","reported_seq":21409,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:55.106174+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:55.106174+0000","last_peered":"2024-09-16T10:28:55.106174+0000","last_clean":"2024-09-16T10:28:55.106174+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:55.106174+0000","last_undegraded":"2024-09-16T10:28:55.106174+0000","last_fullsized":"2024-09-16T10:28:55.106174+0000","mapping_epoch":43,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30950342,"num_objects":7558,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12903,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13265","reported_seq":13458,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.701063+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:51.701063+0000","last_peered":"2024-09-16T10:28:51.701063+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:51.701063+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31847366,"num_objects":7777,"num_object_clones":0,"num_object_copies":15554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13263,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3402},{"shards":"0,2","objects":1693}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.501852+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:45.501852+0000","last_peered":"2024-09-16T10:26:45.501852+0000","last_clean":"2024-09-16T10:26:45.501852+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:45.501852+0000","last_undegraded":"2024-09-16T10:26:45.501852+0000","last_fullsized":"2024-09-16T10:26:45.501852+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12959","reported_seq":13107,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.399217+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:56.399217+0000","last_peered":"2024-09-16T10:28:56.399217+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:56.399217+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30557126,"num_objects":7462,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12957,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1530},{"shards":"2","objects":3206}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'13018","reported_seq":13148,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.417787+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:56.417787+0000","last_peered":"2024-09-16T10:28:56.417787+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:56.417787+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30958534,"num_objects":7560,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13016,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1465},{"shards":"2","objects":3389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12538","reported_seq":12734,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.595960+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:51.595960+0000","last_peered":"2024-09-16T10:28:51.595960+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:51.595960+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2500","ondisk_log_start":"35'2500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30704099,"num_objects":7497,"num_object_clones":0,"num_object_copies":14994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12537,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3396},{"shards":"0,2","objects":1639}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'13105","reported_seq":13325,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.581954+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:51.581954+0000","last_peered":"2024-09-16T10:28:51.581954+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:51.581954+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31987113,"num_objects":7812,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13102,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3518},{"shards":"0,2","objects":1693}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12659","reported_seq":12836,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.695673+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:51.695673+0000","last_peered":"2024-09-16T10:28:51.695673+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:51.695673+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30643142,"num_objects":7483,"num_object_clones":0,"num_object_copies":14966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12657,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3364},{"shards":"0,2","objects":1580}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'13167","reported_seq":13362,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:51.580389+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:51.580389+0000","last_peered":"2024-09-16T10:28:51.580389+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:51.580389+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31659916,"num_objects":7733,"num_object_clones":0,"num_object_copies":15466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13164,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3389},{"shards":"0,2","objects":1682}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":499180046,"num_objects":121887,"num_object_clones":0,"num_object_copies":243774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42483,"num_objects_misplaced":446,"num_objects_unfound":0,"num_objects_dirty":121887,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184418,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8417,"num_bytes_recovered":32539107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1140318208,"data_stored":1140226965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160753,"ondisk_log_size":160753,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018957,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663240,"kb_used_data":378944,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93708600,"statfs":{"total":96636764160,"available":95957606400,"internally_reserved":0,"allocated":388038656,"data_stored":387780950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,2,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20399999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":2.9209999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592544,"kb_used_data":300056,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93779296,"statfs":{"total":96636764160,"available":96029999104,"internally_reserved":0,"allocated":307257344,"data_stored":306927378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":299495082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.93700000000000006}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":5.4210000000000003},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":5.2380000000000004}]}]},{"osd":0,"up_from":24,"seq":103079215184,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797540,"kb_used_data":437016,"kb_used_omap":4,"kb_used_meta":360507,"kb_avail":93574300,"statfs":{"total":96636764160,"available":95820083200,"internally_reserved":0,"allocated":447504384,"data_stored":447254364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":369159848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,3,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":0.46700000000000003},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.5029999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.5059999999999998},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":446189568,"data_stored":446185955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306860032,"data_stored":306780087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387268608,"data_stored":387260923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:58.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:28:59.250 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:59.250 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:28:59.273 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-16T10:28:58.677270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493736481,"num_objects":120560,"num_object_clones":0,"num_object_copies":241120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42477,"num_objects_misplaced":242,"num_objects_unfound":0,"num_objects_dirty":120560,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185749,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8628,"num_bytes_recovered":33374710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2069088,"kb_used_data":1109640,"kb_used_omap":12,"kb_used_meta":959411,"kb_avail":281046432,"statfs":{"total":289910292480,"available":287791546368,"internally_reserved":0,"allocated":1136271360,"data_stored":1135433668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":982436866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,2,4,9,2,0,0,0,0,0,8],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22519808,"num_objects":-5498,"num_object_clones":0,"num_object_copies":-10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-66,"num_objects_misplaced":-562,"num_objects_unfound":0,"num_objects_dirty":-5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5498,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":2301952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003063"},"pg_stats":[{"pgid":"2.f","version":"44'12896","reported_seq":20866,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:56.730093+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:56.730093+0000","last_peered":"2024-09-16T10:28:56.730093+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:56.730093+0000","last_undegraded":"2024-09-16T10:28:56.730093+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31014912,"num_objects":7572,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":242,"num_objects_unfound":0,"num_objects_dirty":7572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12896,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3950,"num_bytes_recovered":16179200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7330},{"shards":"0,2","objects":242}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12934","reported_seq":13092,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:56.399700+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:56.399700+0000","last_peered":"2024-09-16T10:28:56.399700+0000","last_clean":"2024-09-16T10:28:56.399700+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:56.399700+0000","last_undegraded":"2024-09-16T10:28:56.399700+0000","last_fullsized":"2024-09-16T10:28:56.399700+0000","mapping_epoch":34,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30834688,"num_objects":7528,"num_object_clones":0,"num_object_copies":15056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9147,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'13073","reported_seq":13312,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.452909+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:56.452909+0000","last_peered":"2024-09-16T10:28:56.452909+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:56.452909+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31088664,"num_objects":7591,"num_object_clones":0,"num_object_copies":15182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7591,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9130,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1596},{"shards":"2","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12964","reported_seq":13021,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:56.482260+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:56.482260+0000","last_peered":"2024-09-16T10:28:56.482260+0000","last_clean":"2024-09-16T10:28:56.482260+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:56.482260+0000","last_undegraded":"2024-09-16T10:28:56.482260+0000","last_fullsized":"2024-09-16T10:28:56.482260+0000","mapping_epoch":34,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711808,"num_objects":7498,"num_object_clones":0,"num_object_copies":14996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12964,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12966","reported_seq":13123,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:56.681029+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:56.681029+0000","last_peered":"2024-09-16T10:28:56.681029+0000","last_clean":"2024-09-16T10:28:56.681029+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:56.681029+0000","last_undegraded":"2024-09-16T10:28:56.681029+0000","last_fullsized":"2024-09-16T10:28:56.681029+0000","mapping_epoch":33,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30879744,"num_objects":7539,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9608,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12887","reported_seq":13141,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.709196+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:56.709196+0000","last_peered":"2024-09-16T10:28:56.709196+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:56.709196+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30654464,"num_objects":7484,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3312},{"shards":"0,2","objects":1518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'13307","reported_seq":13645,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.630230+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:56.630230+0000","last_peered":"2024-09-16T10:28:56.630230+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:56.630230+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31182848,"num_objects":7613,"num_object_clones":0,"num_object_copies":15226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3292},{"shards":"0,2","objects":1559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12851","reported_seq":13027,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:56.643852+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:56.643852+0000","last_peered":"2024-09-16T10:28:56.643852+0000","last_clean":"2024-09-16T10:28:56.643852+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:56.643852+0000","last_undegraded":"2024-09-16T10:28:56.643852+0000","last_fullsized":"2024-09-16T10:28:56.643852+0000","mapping_epoch":33,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30879744,"num_objects":7539,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12905","reported_seq":21409,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:55.106174+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:55.106174+0000","last_peered":"2024-09-16T10:28:55.106174+0000","last_clean":"2024-09-16T10:28:55.106174+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:55.106174+0000","last_undegraded":"2024-09-16T10:28:55.106174+0000","last_fullsized":"2024-09-16T10:28:55.106174+0000","mapping_epoch":43,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30950342,"num_objects":7558,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12903,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13408","reported_seq":13614,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.683273+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:56.683273+0000","last_peered":"2024-09-16T10:28:56.683273+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:56.683273+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31261638,"num_objects":7634,"num_object_clones":0,"num_object_copies":15268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13406,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3263},{"shards":"0,2","objects":1550}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.501852+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:45.501852+0000","last_peered":"2024-09-16T10:26:45.501852+0000","last_clean":"2024-09-16T10:26:45.501852+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:45.501852+0000","last_undegraded":"2024-09-16T10:26:45.501852+0000","last_fullsized":"2024-09-16T10:26:45.501852+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12959","reported_seq":13107,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.399217+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:56.399217+0000","last_peered":"2024-09-16T10:28:56.399217+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:56.399217+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30557126,"num_objects":7462,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12957,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1530},{"shards":"2","objects":3206}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'13018","reported_seq":13148,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.417787+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:56.417787+0000","last_peered":"2024-09-16T10:28:56.417787+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:56.417787+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30958534,"num_objects":7560,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13016,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1465},{"shards":"2","objects":3389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12672","reported_seq":12868,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.690048+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:56.690048+0000","last_peered":"2024-09-16T10:28:56.690048+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:56.690048+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30155235,"num_objects":7363,"num_object_clones":0,"num_object_copies":14726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12671,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3262},{"shards":"0,2","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'13251","reported_seq":13473,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.706920+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:56.706920+0000","last_peered":"2024-09-16T10:28:56.706920+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:56.706920+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31389097,"num_objects":7666,"num_object_clones":0,"num_object_copies":15332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13248,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3373},{"shards":"0,2","objects":1547}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12788","reported_seq":12965,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.705404+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:56.705404+0000","last_peered":"2024-09-16T10:28:56.705404+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:56.705404+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30114758,"num_objects":7354,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3235},{"shards":"0,2","objects":1451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'13303","reported_seq":13498,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.733742+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:56.733742+0000","last_peered":"2024-09-16T10:28:56.733742+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:56.733742+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31102860,"num_objects":7597,"num_object_clones":0,"num_object_copies":15194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13300,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3253},{"shards":"0,2","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":493736462,"num_objects":120558,"num_object_clones":0,"num_object_copies":241116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42477,"num_objects_misplaced":242,"num_objects_unfound":0,"num_objects_dirty":120558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185747,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8627,"num_bytes_recovered":33374691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1133879296,"data_stored":1133788053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160782,"ondisk_log_size":160782,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018957,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663240,"kb_used_data":378944,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93708600,"statfs":{"total":96636764160,"available":95957606400,"internally_reserved":0,"allocated":388038656,"data_stored":387780950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,2,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20399999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":2.9209999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592544,"kb_used_data":300056,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93779296,"statfs":{"total":96636764160,"available":96029999104,"internally_reserved":0,"allocated":307257344,"data_stored":306927378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":299495082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.93700000000000006}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":5.4210000000000003},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":5.2380000000000004}]}]},{"osd":0,"up_from":24,"seq":103079215185,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":813304,"kb_used_data":430640,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93558536,"statfs":{"total":96636764160,"available":95803940864,"internally_reserved":0,"allocated":440975360,"data_stored":440725340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":391835304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,4,1,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":13.795999999999999},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.7850000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":7.2400000000000002},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":13.752000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":439750656,"data_stored":439747043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306860032,"data_stored":306780087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387268608,"data_stored":387260923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:28:59.274 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:28:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'12896", 'reported_seq': 20866, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:28:56.730093+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:28:56.730093+0000', 'last_peered': '2024-09-16T10:28:56.730093+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:28:56.730093+0000', 'last_undegraded': '2024-09-16T10:28:56.730093+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'2800", 'ondisk_log_start': "35'2800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31014912, 'num_objects': 7572, 'num_object_clones': 0, 'num_object_copies': 15144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 242, 'num_objects_unfound': 0, 'num_objects_dirty': 7572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12896, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3950, 'num_bytes_recovered': 16179200, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7330}, {'shards': '0,2', 'objects': 242}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:28:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'13073", 'reported_seq': 13312, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:56.452909+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:28:56.452909+0000', 'last_peered': '2024-09-16T10:28:56.452909+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:28:56.452909+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'3000", 'ondisk_log_start': "35'3000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31088664, 'num_objects': 7591, 'num_object_clones': 0, 'num_object_copies': 15182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7591, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9130, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1596}, {'shards': '2', 'objects': 3282}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:28:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'12887", 'reported_seq': 13141, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:56.709196+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:28:56.709196+0000', 'last_peered': '2024-09-16T10:28:56.709196+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:28:56.709196+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'2800", 'ondisk_log_start': "35'2800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30654464, 'num_objects': 7484, 'num_object_clones': 0, 'num_object_copies': 14968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4174, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9512, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 45, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3312}, {'shards': '0,2', 'objects': 1518}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:28:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'13307", 'reported_seq': 13645, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:56.630230+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:28:56.630230+0000', 'last_peered': '2024-09-16T10:28:56.630230+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:28:56.630230+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'3300", 'ondisk_log_start': "35'3300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31182848, 'num_objects': 7613, 'num_object_clones': 0, 'num_object_copies': 15226, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4323, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9549, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 82, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3292}, {'shards': '0,2', 'objects': 1559}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:28:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'13408", 'reported_seq': 13614, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:56.683273+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:28:56.683273+0000', 'last_peered': '2024-09-16T10:28:56.683273+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:28:56.683273+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'3400", 'ondisk_log_start': "35'3400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31261638, 'num_objects': 7634, 'num_object_clones': 0, 'num_object_copies': 15268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4373, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13406, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 65, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3263}, {'shards': '0,2', 'objects': 1550}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:28:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'12959", 'reported_seq': 13107, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:56.399217+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:28:56.399217+0000', 'last_peered': '2024-09-16T10:28:56.399217+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:28:56.399217+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'2900", 'ondisk_log_start': "35'2900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30557126, 'num_objects': 7462, 'num_object_clones': 0, 'num_object_copies': 14924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4258, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7462, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12957, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1530}, {'shards': '2', 'objects': 3206}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:28:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'13018", 'reported_seq': 13148, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:56.417787+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:28:56.417787+0000', 'last_peered': '2024-09-16T10:28:56.417787+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:28:56.417787+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'3000", 'ondisk_log_start': "35'3000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30958534, 'num_objects': 7560, 'num_object_clones': 0, 'num_object_copies': 15120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4173, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13016, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 23, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1465}, {'shards': '2', 'objects': 3389}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:28:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:28:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'12672", 'reported_seq': 12868, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:56.690048+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:28:56.690048+0000', 'last_peered': '2024-09-16T10:28:56.690048+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:28:56.690048+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'2600", 'ondisk_log_start': "35'2600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30155235, 'num_objects': 7363, 'num_object_clones': 0, 'num_object_copies': 14726, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4102, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7363, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12671, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 59, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3262}, {'shards': '0,2', 'objects': 1505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:28:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'13251", 'reported_seq': 13473, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:56.706920+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:28:56.706920+0000', 'last_peered': '2024-09-16T10:28:56.706920+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:28:56.706920+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'3200", 'ondisk_log_start': "35'3200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31389097, 'num_objects': 7666, 'num_object_clones': 0, 'num_object_copies': 15332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4296, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13248, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 47, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3373}, {'shards': '0,2', 'objects': 1547}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:28:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'12788", 'reported_seq': 12965, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:56.705404+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:28:56.705404+0000', 'last_peered': '2024-09-16T10:28:56.705404+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:28:56.705404+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'2700", 'ondisk_log_start': "35'2700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30114758, 'num_objects': 7354, 'num_object_clones': 0, 'num_object_copies': 14708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7354, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12786, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 50, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3235}, {'shards': '0,2', 'objects': 1451}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:28:59.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'13303", 'reported_seq': 13498, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:28:56.733742+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:28:56.733742+0000', 'last_peered': '2024-09-16T10:28:56.733742+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:28:56.733742+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'3300", 'ondisk_log_start': "35'3300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31102860, 'num_objects': 7597, 'num_object_clones': 0, 'num_object_copies': 15194, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4347, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13300, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3253}, {'shards': '0,2', 'objects': 1546}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:28:59.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:28:59.931 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:28:59.931 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42477/241120 objects degraded (17.617%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":428,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":44,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":120560,"data_bytes":493736481,"bytes_used":2118746112,"bytes_avail":287791546368,"bytes_total":289910292480,"degraded_objects":42477,"degraded_total":241120,"degraded_ratio":0.17616539482415394,"misplaced_objects":242,"misplaced_total":241120,"misplaced_ratio":0.0010036496350364964,"recovering_objects_per_sec":52,"recovering_bytes_per_sec":191780,"recovering_keys_per_sec":0,"num_objects_recovered":628,"num_bytes_recovered":2301952,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":458},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:28:59.960 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:28:59.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:00.513 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:00.515 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:00.538 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-16T10:28:58.677270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":493736481,"num_objects":120560,"num_object_clones":0,"num_object_copies":241120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42477,"num_objects_misplaced":242,"num_objects_unfound":0,"num_objects_dirty":120560,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185749,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8628,"num_bytes_recovered":33374710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2069088,"kb_used_data":1109640,"kb_used_omap":12,"kb_used_meta":959411,"kb_avail":281046432,"statfs":{"total":289910292480,"available":287791546368,"internally_reserved":0,"allocated":1136271360,"data_stored":1135433668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":982436866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,2,4,9,2,0,0,0,0,0,8],"upper_bound":8192},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22519808,"num_objects":-5498,"num_object_clones":0,"num_object_copies":-10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-66,"num_objects_misplaced":-562,"num_objects_unfound":0,"num_objects_dirty":-5498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5498,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":628,"num_bytes_recovered":2301952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003063"},"pg_stats":[{"pgid":"2.f","version":"44'12896","reported_seq":20866,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:28:56.730093+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:28:56.730093+0000","last_peered":"2024-09-16T10:28:56.730093+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:28:56.730093+0000","last_undegraded":"2024-09-16T10:28:56.730093+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31014912,"num_objects":7572,"num_object_clones":0,"num_object_copies":15144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":242,"num_objects_unfound":0,"num_objects_dirty":7572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12896,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3950,"num_bytes_recovered":16179200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7330},{"shards":"0,2","objects":242}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'12934","reported_seq":13092,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:56.399700+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:28:56.399700+0000","last_peered":"2024-09-16T10:28:56.399700+0000","last_clean":"2024-09-16T10:28:56.399700+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:28:56.399700+0000","last_undegraded":"2024-09-16T10:28:56.399700+0000","last_fullsized":"2024-09-16T10:28:56.399700+0000","mapping_epoch":34,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30834688,"num_objects":7528,"num_object_clones":0,"num_object_copies":15056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9147,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'13073","reported_seq":13312,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.452909+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:28:56.452909+0000","last_peered":"2024-09-16T10:28:56.452909+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:28:56.452909+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31088664,"num_objects":7591,"num_object_clones":0,"num_object_copies":15182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7591,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9130,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1596},{"shards":"2","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'12964","reported_seq":13021,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:56.482260+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:28:56.482260+0000","last_peered":"2024-09-16T10:28:56.482260+0000","last_clean":"2024-09-16T10:28:56.482260+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:28:56.482260+0000","last_undegraded":"2024-09-16T10:28:56.482260+0000","last_fullsized":"2024-09-16T10:28:56.482260+0000","mapping_epoch":34,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711808,"num_objects":7498,"num_object_clones":0,"num_object_copies":14996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12964,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'12966","reported_seq":13123,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:56.681029+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:28:56.681029+0000","last_peered":"2024-09-16T10:28:56.681029+0000","last_clean":"2024-09-16T10:28:56.681029+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:28:56.681029+0000","last_undegraded":"2024-09-16T10:28:56.681029+0000","last_fullsized":"2024-09-16T10:28:56.681029+0000","mapping_epoch":33,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30879744,"num_objects":7539,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9608,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'12887","reported_seq":13141,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.709196+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:28:56.709196+0000","last_peered":"2024-09-16T10:28:56.709196+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:28:56.709196+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30654464,"num_objects":7484,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9512,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3312},{"shards":"0,2","objects":1518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'13307","reported_seq":13645,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.630230+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:28:56.630230+0000","last_peered":"2024-09-16T10:28:56.630230+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:28:56.630230+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31182848,"num_objects":7613,"num_object_clones":0,"num_object_copies":15226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9549,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3292},{"shards":"0,2","objects":1559}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12851","reported_seq":13027,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:56.643852+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:28:56.643852+0000","last_peered":"2024-09-16T10:28:56.643852+0000","last_clean":"2024-09-16T10:28:56.643852+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:28:56.643852+0000","last_undegraded":"2024-09-16T10:28:56.643852+0000","last_fullsized":"2024-09-16T10:28:56.643852+0000","mapping_epoch":33,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30879744,"num_objects":7539,"num_object_clones":0,"num_object_copies":15078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'12905","reported_seq":21409,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:28:55.106174+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:28:55.106174+0000","last_peered":"2024-09-16T10:28:55.106174+0000","last_clean":"2024-09-16T10:28:55.106174+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:28:55.106174+0000","last_undegraded":"2024-09-16T10:28:55.106174+0000","last_fullsized":"2024-09-16T10:28:55.106174+0000","mapping_epoch":43,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30950342,"num_objects":7558,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12903,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13408","reported_seq":13614,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.683273+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:28:56.683273+0000","last_peered":"2024-09-16T10:28:56.683273+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:28:56.683273+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31261638,"num_objects":7634,"num_object_clones":0,"num_object_copies":15268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13406,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3263},{"shards":"0,2","objects":1550}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.501852+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:45.501852+0000","last_peered":"2024-09-16T10:26:45.501852+0000","last_clean":"2024-09-16T10:26:45.501852+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:45.501852+0000","last_undegraded":"2024-09-16T10:26:45.501852+0000","last_fullsized":"2024-09-16T10:26:45.501852+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'12959","reported_seq":13107,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.399217+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:28:56.399217+0000","last_peered":"2024-09-16T10:28:56.399217+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:28:56.399217+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30557126,"num_objects":7462,"num_object_clones":0,"num_object_copies":14924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12957,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1530},{"shards":"2","objects":3206}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'13018","reported_seq":13148,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.417787+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:28:56.417787+0000","last_peered":"2024-09-16T10:28:56.417787+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:28:56.417787+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30958534,"num_objects":7560,"num_object_clones":0,"num_object_copies":15120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13016,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1465},{"shards":"2","objects":3389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12672","reported_seq":12868,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.690048+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:28:56.690048+0000","last_peered":"2024-09-16T10:28:56.690048+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:28:56.690048+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2600","ondisk_log_start":"35'2600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30155235,"num_objects":7363,"num_object_clones":0,"num_object_copies":14726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12671,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3262},{"shards":"0,2","objects":1505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'13251","reported_seq":13473,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.706920+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:28:56.706920+0000","last_peered":"2024-09-16T10:28:56.706920+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:28:56.706920+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31389097,"num_objects":7666,"num_object_clones":0,"num_object_copies":15332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13248,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3373},{"shards":"0,2","objects":1547}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12788","reported_seq":12965,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.705404+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:28:56.705404+0000","last_peered":"2024-09-16T10:28:56.705404+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:28:56.705404+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2700","ondisk_log_start":"35'2700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30114758,"num_objects":7354,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12786,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3235},{"shards":"0,2","objects":1451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'13303","reported_seq":13498,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:28:56.733742+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:28:56.733742+0000","last_peered":"2024-09-16T10:28:56.733742+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:28:56.733742+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31102860,"num_objects":7597,"num_object_clones":0,"num_object_copies":15194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13300,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3253},{"shards":"0,2","objects":1546}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":493736462,"num_objects":120558,"num_object_clones":0,"num_object_copies":241116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42477,"num_objects_misplaced":242,"num_objects_unfound":0,"num_objects_dirty":120558,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185747,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8627,"num_bytes_recovered":33374691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1133879296,"data_stored":1133788053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160782,"ondisk_log_size":160782,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018957,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663240,"kb_used_data":378944,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93708600,"statfs":{"total":96636764160,"available":95957606400,"internally_reserved":0,"allocated":388038656,"data_stored":387780950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,2,0,0,0,0,0,0,2],"upper_bound":8192},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.464,"15min":1.0620000000000001},"min":{"1min":0.221,"5min":0.184,"15min":0.184},"max":{"1min":4.6079999999999997,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.335,"15min":1.3779999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.8319999999999999,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Sep 16 10:27:59 2024","interfaces":[{"interface":"back","average":{"1min":3.915,"5min":2.867,"15min":2.887},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":15.157999999999999,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20399999999999999},{"interface":"front","average":{"1min":3.2810000000000001,"5min":2.556,"15min":2.3460000000000001},"min":{"1min":0.30299999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":16.741,"5min":16.741,"15min":16.741},"last":2.9209999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":592544,"kb_used_data":300056,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93779296,"statfs":{"total":96636764160,"available":96029999104,"internally_reserved":0,"allocated":307257344,"data_stored":306927378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":299495082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.93700000000000006}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":5.4210000000000003},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":5.2380000000000004}]}]},{"osd":0,"up_from":24,"seq":103079215185,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":813304,"kb_used_data":430640,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93558536,"statfs":{"total":96636764160,"available":95803940864,"internally_reserved":0,"allocated":440975360,"data_stored":440725340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":391835304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,4,1,0,0,0,0,0,6],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":13.795999999999999},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.7850000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":7.2400000000000002},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":13.752000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":439750656,"data_stored":439747043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306860032,"data_stored":306780087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387268608,"data_stored":387260923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:03.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:03.836 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:03.837 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:03.849 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-09-16T10:29:02.966250+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484979233,"num_objects":118422,"num_object_clones":0,"num_object_copies":236844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42452,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":118422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187887,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8866,"num_bytes_recovered":34247158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2037708,"kb_used_data":1098740,"kb_used_omap":12,"kb_used_meta":938931,"kb_avail":281077812,"statfs":{"total":289910292480,"available":287823679488,"internally_reserved":0,"allocated":1125109760,"data_stored":1124272068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":961465346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,0,5,2,8,1,0,6,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17338368,"num_objects":-4233,"num_object_clones":0,"num_object_copies":-8466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-37,"num_objects_misplaced":-417,"num_objects_unfound":0,"num_objects_dirty":-4233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4233,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":454,"num_bytes_recovered":1708032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.290477"},"pg_stats":[{"pgid":"2.f","version":"44'13019","reported_seq":21415,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:29:01.752042+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:29:01.752042+0000","last_peered":"2024-09-16T10:29:01.752042+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:29:01.752042+0000","last_undegraded":"2024-09-16T10:29:01.752042+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30511104,"num_objects":7449,"num_object_clones":0,"num_object_copies":14898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":7449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13019,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4163,"num_bytes_recovered":17051648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7420},{"shards":"0,2","objects":29}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'13061","reported_seq":13219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.523266+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:01.523266+0000","last_peered":"2024-09-16T10:29:01.523266+0000","last_clean":"2024-09-16T10:29:01.523266+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:01.523266+0000","last_undegraded":"2024-09-16T10:29:01.523266+0000","last_fullsized":"2024-09-16T10:29:01.523266+0000","mapping_epoch":34,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30314496,"num_objects":7401,"num_object_clones":0,"num_object_copies":14802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'13191","reported_seq":13438,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.534444+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:01.534444+0000","last_peered":"2024-09-16T10:29:01.534444+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:01.534444+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30605336,"num_objects":7473,"num_object_clones":0,"num_object_copies":14946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9248,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1478},{"shards":"2","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'13106","reported_seq":13161,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.530190+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:01.530190+0000","last_peered":"2024-09-16T10:29:01.530190+0000","last_clean":"2024-09-16T10:29:01.530190+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:01.530190+0000","last_undegraded":"2024-09-16T10:29:01.530190+0000","last_fullsized":"2024-09-16T10:29:01.530190+0000","mapping_epoch":34,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30130176,"num_objects":7356,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13106,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'13102","reported_seq":13257,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.723676+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:01.723676+0000","last_peered":"2024-09-16T10:29:01.723676+0000","last_clean":"2024-09-16T10:29:01.723676+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:01.723676+0000","last_undegraded":"2024-09-16T10:29:01.723676+0000","last_fullsized":"2024-09-16T10:29:01.723676+0000","mapping_epoch":33,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322688,"num_objects":7403,"num_object_clones":0,"num_object_copies":14806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'13029","reported_seq":13291,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.742515+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:01.742515+0000","last_peered":"2024-09-16T10:29:01.742515+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:01.742515+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30072832,"num_objects":7342,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3174},{"shards":"0,2","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'13436","reported_seq":13774,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.738143+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:01.738143+0000","last_peered":"2024-09-16T10:29:01.738143+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:01.738143+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30654464,"num_objects":7484,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3163},{"shards":"0,2","objects":1430}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12979","reported_seq":13155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.673493+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:01.673493+0000","last_peered":"2024-09-16T10:29:01.673493+0000","last_clean":"2024-09-16T10:29:01.673493+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:01.673493+0000","last_undegraded":"2024-09-16T10:29:01.673493+0000","last_fullsized":"2024-09-16T10:29:01.673493+0000","mapping_epoch":33,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30355456,"num_objects":7411,"num_object_clones":0,"num_object_copies":14822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'13059","reported_seq":21563,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:00.252112+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:00.252112+0000","last_peered":"2024-09-16T10:29:00.252112+0000","last_clean":"2024-09-16T10:29:00.252112+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:00.252112+0000","last_undegraded":"2024-09-16T10:29:00.252112+0000","last_fullsized":"2024-09-16T10:29:00.252112+0000","mapping_epoch":43,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30319558,"num_objects":7404,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13057,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13534","reported_seq":13746,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.731345+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:01.731345+0000","last_peered":"2024-09-16T10:29:01.731345+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:01.731345+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30745542,"num_objects":7508,"num_object_clones":0,"num_object_copies":15016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13532,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3140},{"shards":"0,2","objects":1424}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.501852+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:45.501852+0000","last_peered":"2024-09-16T10:26:45.501852+0000","last_clean":"2024-09-16T10:26:45.501852+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:45.501852+0000","last_undegraded":"2024-09-16T10:26:45.501852+0000","last_fullsized":"2024-09-16T10:26:45.501852+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'13093","reported_seq":13247,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.443105+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:01.443105+0000","last_peered":"2024-09-16T10:29:01.443105+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:01.443105+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30008262,"num_objects":7328,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13091,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1396},{"shards":"2","objects":3075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'13147","reported_seq":13283,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.423237+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:29:01.423237+0000","last_peered":"2024-09-16T10:29:01.423237+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:01.423237+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30430150,"num_objects":7431,"num_object_clones":0,"num_object_copies":14862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13145,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1336},{"shards":"2","objects":3263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12808","reported_seq":13012,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.553886+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:01.553886+0000","last_peered":"2024-09-16T10:29:01.553886+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:01.553886+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29598179,"num_objects":7227,"num_object_clones":0,"num_object_copies":14454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12807,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3130},{"shards":"0,2","objects":1369}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'13397","reported_seq":13625,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.753216+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:01.753216+0000","last_peered":"2024-09-16T10:29:01.753216+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:01.753216+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30791081,"num_objects":7520,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13394,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3230},{"shards":"0,2","objects":1401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12928","reported_seq":13107,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.702451+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:01.702451+0000","last_peered":"2024-09-16T10:29:01.702451+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:01.702451+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29541318,"num_objects":7214,"num_object_clones":0,"num_object_copies":14428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12926,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3096},{"shards":"0,2","objects":1311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'13431","reported_seq":13626,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.677051+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:01.677051+0000","last_peered":"2024-09-16T10:29:01.677051+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:01.677051+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30578572,"num_objects":7469,"num_object_clones":0,"num_object_copies":14938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13428,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3125},{"shards":"0,2","objects":1418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":484979214,"num_objects":118420,"num_object_clones":0,"num_object_copies":236840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42452,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":118420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187885,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8865,"num_bytes_recovered":34247139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1122664448,"data_stored":1122573205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160720,"ondisk_log_size":160720,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018958,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663080,"kb_used_data":374688,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93708760,"statfs":{"total":96636764160,"available":95957770240,"internally_reserved":0,"allocated":383680512,"data_stored":383422806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,2,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.19},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1520000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":17.914000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579972,"kb_used_data":299772,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93791868,"statfs":{"total":96636764160,"available":96042872832,"internally_reserved":0,"allocated":306966528,"data_stored":306636562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":286912170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":4.4729999999999999},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.84399999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.32500000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215186,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":794656,"kb_used_data":424280,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93577184,"statfs":{"total":96636764160,"available":95823036416,"internally_reserved":0,"allocated":434462720,"data_stored":434212700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,5,0,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":6.7430000000000003},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.5950000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":3.641},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":3.6640000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":433188864,"data_stored":433185251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306565120,"data_stored":306485175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382910464,"data_stored":382902779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:03.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:04.289 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:04.290 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:04.316 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-16T10:29:03.967160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484979233,"num_objects":118422,"num_object_clones":0,"num_object_copies":236844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42452,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":118422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187887,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8866,"num_bytes_recovered":34247158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2037708,"kb_used_data":1098740,"kb_used_omap":12,"kb_used_meta":938931,"kb_avail":281077812,"statfs":{"total":289910292480,"available":287823679488,"internally_reserved":0,"allocated":1125109760,"data_stored":1124272068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":961465346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,0,5,2,8,1,0,6,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17338368,"num_objects":-4233,"num_object_clones":0,"num_object_copies":-8466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-37,"num_objects_misplaced":-417,"num_objects_unfound":0,"num_objects_dirty":-4233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4233,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":454,"num_bytes_recovered":1708032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.290859"},"pg_stats":[{"pgid":"2.f","version":"44'13019","reported_seq":21415,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:29:01.752042+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:29:01.752042+0000","last_peered":"2024-09-16T10:29:01.752042+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:29:01.752042+0000","last_undegraded":"2024-09-16T10:29:01.752042+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30511104,"num_objects":7449,"num_object_clones":0,"num_object_copies":14898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":7449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13019,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4163,"num_bytes_recovered":17051648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7420},{"shards":"0,2","objects":29}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'13061","reported_seq":13219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.523266+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:01.523266+0000","last_peered":"2024-09-16T10:29:01.523266+0000","last_clean":"2024-09-16T10:29:01.523266+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:01.523266+0000","last_undegraded":"2024-09-16T10:29:01.523266+0000","last_fullsized":"2024-09-16T10:29:01.523266+0000","mapping_epoch":34,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30314496,"num_objects":7401,"num_object_clones":0,"num_object_copies":14802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'13191","reported_seq":13438,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.534444+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:01.534444+0000","last_peered":"2024-09-16T10:29:01.534444+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:01.534444+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30605336,"num_objects":7473,"num_object_clones":0,"num_object_copies":14946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9248,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1478},{"shards":"2","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'13106","reported_seq":13161,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.530190+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:01.530190+0000","last_peered":"2024-09-16T10:29:01.530190+0000","last_clean":"2024-09-16T10:29:01.530190+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:01.530190+0000","last_undegraded":"2024-09-16T10:29:01.530190+0000","last_fullsized":"2024-09-16T10:29:01.530190+0000","mapping_epoch":34,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30130176,"num_objects":7356,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13106,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'13102","reported_seq":13257,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.723676+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:01.723676+0000","last_peered":"2024-09-16T10:29:01.723676+0000","last_clean":"2024-09-16T10:29:01.723676+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:01.723676+0000","last_undegraded":"2024-09-16T10:29:01.723676+0000","last_fullsized":"2024-09-16T10:29:01.723676+0000","mapping_epoch":33,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322688,"num_objects":7403,"num_object_clones":0,"num_object_copies":14806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'13029","reported_seq":13291,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.742515+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:01.742515+0000","last_peered":"2024-09-16T10:29:01.742515+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:01.742515+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30072832,"num_objects":7342,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3174},{"shards":"0,2","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'13436","reported_seq":13774,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.738143+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:01.738143+0000","last_peered":"2024-09-16T10:29:01.738143+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:01.738143+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30654464,"num_objects":7484,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3163},{"shards":"0,2","objects":1430}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12979","reported_seq":13155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.673493+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:01.673493+0000","last_peered":"2024-09-16T10:29:01.673493+0000","last_clean":"2024-09-16T10:29:01.673493+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:01.673493+0000","last_undegraded":"2024-09-16T10:29:01.673493+0000","last_fullsized":"2024-09-16T10:29:01.673493+0000","mapping_epoch":33,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30355456,"num_objects":7411,"num_object_clones":0,"num_object_copies":14822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'13059","reported_seq":21563,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:00.252112+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:00.252112+0000","last_peered":"2024-09-16T10:29:00.252112+0000","last_clean":"2024-09-16T10:29:00.252112+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:00.252112+0000","last_undegraded":"2024-09-16T10:29:00.252112+0000","last_fullsized":"2024-09-16T10:29:00.252112+0000","mapping_epoch":43,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30319558,"num_objects":7404,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13057,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13534","reported_seq":13746,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.731345+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:01.731345+0000","last_peered":"2024-09-16T10:29:01.731345+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:01.731345+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30745542,"num_objects":7508,"num_object_clones":0,"num_object_copies":15016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13532,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3140},{"shards":"0,2","objects":1424}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.501852+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:45.501852+0000","last_peered":"2024-09-16T10:26:45.501852+0000","last_clean":"2024-09-16T10:26:45.501852+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:45.501852+0000","last_undegraded":"2024-09-16T10:26:45.501852+0000","last_fullsized":"2024-09-16T10:26:45.501852+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'13093","reported_seq":13247,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.443105+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:01.443105+0000","last_peered":"2024-09-16T10:29:01.443105+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:01.443105+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30008262,"num_objects":7328,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13091,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1396},{"shards":"2","objects":3075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'13147","reported_seq":13283,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.423237+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:29:01.423237+0000","last_peered":"2024-09-16T10:29:01.423237+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:01.423237+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30430150,"num_objects":7431,"num_object_clones":0,"num_object_copies":14862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13145,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1336},{"shards":"2","objects":3263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12808","reported_seq":13012,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.553886+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:01.553886+0000","last_peered":"2024-09-16T10:29:01.553886+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:01.553886+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29598179,"num_objects":7227,"num_object_clones":0,"num_object_copies":14454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12807,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3130},{"shards":"0,2","objects":1369}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'13397","reported_seq":13625,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.753216+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:01.753216+0000","last_peered":"2024-09-16T10:29:01.753216+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:01.753216+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30791081,"num_objects":7520,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13394,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3230},{"shards":"0,2","objects":1401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12928","reported_seq":13107,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.702451+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:01.702451+0000","last_peered":"2024-09-16T10:29:01.702451+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:01.702451+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29541318,"num_objects":7214,"num_object_clones":0,"num_object_copies":14428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12926,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3096},{"shards":"0,2","objects":1311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'13431","reported_seq":13626,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.677051+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:01.677051+0000","last_peered":"2024-09-16T10:29:01.677051+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:01.677051+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30578572,"num_objects":7469,"num_object_clones":0,"num_object_copies":14938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13428,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3125},{"shards":"0,2","objects":1418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":484979214,"num_objects":118420,"num_object_clones":0,"num_object_copies":236840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42452,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":118420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187885,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8865,"num_bytes_recovered":34247139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1122664448,"data_stored":1122573205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160720,"ondisk_log_size":160720,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018958,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663080,"kb_used_data":374688,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93708760,"statfs":{"total":96636764160,"available":95957770240,"internally_reserved":0,"allocated":383680512,"data_stored":383422806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,2,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.19},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1520000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":17.914000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579972,"kb_used_data":299772,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93791868,"statfs":{"total":96636764160,"available":96042872832,"internally_reserved":0,"allocated":306966528,"data_stored":306636562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":286912170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":4.4729999999999999},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.84399999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.32500000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215186,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":794656,"kb_used_data":424280,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93577184,"statfs":{"total":96636764160,"available":95823036416,"internally_reserved":0,"allocated":434462720,"data_stored":434212700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,5,0,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":6.7430000000000003},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.5950000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":3.641},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":3.6640000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":433188864,"data_stored":433185251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306565120,"data_stored":306485175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382910464,"data_stored":382902779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:04.317 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.f is not active+clean 2024-09-16T10:29:04.317 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.f', 'version': "44'13019", 'reported_seq': 21415, 'reported_epoch': 44, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-16T10:29:01.752042+0000', 'last_change': '2024-09-16T10:26:00.070363+0000', 'last_active': '2024-09-16T10:29:01.752042+0000', 'last_peered': '2024-09-16T10:29:01.752042+0000', 'last_clean': '2024-09-16T10:23:54.356173+0000', 'last_became_active': '2024-09-16T10:23:56.894572+0000', 'last_became_peered': '2024-09-16T10:23:56.894572+0000', 'last_unstale': '2024-09-16T10:29:01.752042+0000', 'last_undegraded': '2024-09-16T10:29:01.752042+0000', 'last_fullsized': '2024-09-16T10:23:56.367880+0000', 'mapping_epoch': 41, 'log_start': "35'3000", 'ondisk_log_start': "35'3000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30511104, 'num_objects': 7449, 'num_object_clones': 0, 'num_object_copies': 14898, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 29, 'num_objects_unfound': 0, 'num_objects_dirty': 7449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13019, 'num_write_kb': 40936, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4163, 'num_bytes_recovered': 17051648, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 7420}, {'shards': '0,2', 'objects': 29}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:04.317 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:04.318 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "44'13191", 'reported_seq': 13438, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:01.534444+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:01.534444+0000', 'last_peered': '2024-09-16T10:29:01.534444+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:01.534444+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'3100", 'ondisk_log_start': "35'3100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30605336, 'num_objects': 7473, 'num_object_clones': 0, 'num_object_copies': 14946, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7473, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9248, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1478}, {'shards': '2', 'objects': 3168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:04.318 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:04.318 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "44'13029", 'reported_seq': 13291, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:01.742515+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:01.742515+0000', 'last_peered': '2024-09-16T10:29:01.742515+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:01.742515+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'3000", 'ondisk_log_start': "35'3000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30072832, 'num_objects': 7342, 'num_object_clones': 0, 'num_object_copies': 14684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4170, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9654, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 49, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3174}, {'shards': '0,2', 'objects': 1376}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:04.318 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:04.318 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "44'13436", 'reported_seq': 13774, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:01.738143+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:01.738143+0000', 'last_peered': '2024-09-16T10:29:01.738143+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:01.738143+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'3400", 'ondisk_log_start': "35'3400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30654464, 'num_objects': 7484, 'num_object_clones': 0, 'num_object_copies': 14968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4323, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9678, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 82, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3163}, {'shards': '0,2', 'objects': 1430}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:04.318 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:04.318 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "44'13534", 'reported_seq': 13746, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:01.731345+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:01.731345+0000', 'last_peered': '2024-09-16T10:29:01.731345+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:01.731345+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'3500", 'ondisk_log_start': "35'3500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30745542, 'num_objects': 7508, 'num_object_clones': 0, 'num_object_copies': 15016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4370, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13532, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3140}, {'shards': '0,2', 'objects': 1424}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:04.318 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:04.318 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "44'13093", 'reported_seq': 13247, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:01.443105+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:01.443105+0000', 'last_peered': '2024-09-16T10:29:01.443105+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:01.443105+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'3000", 'ondisk_log_start': "35'3000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30008262, 'num_objects': 7328, 'num_object_clones': 0, 'num_object_copies': 14656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4255, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13091, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1396}, {'shards': '2', 'objects': 3075}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:04.318 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:04.319 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "44'13147", 'reported_seq': 13283, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:01.423237+0000', 'last_change': '2024-09-16T10:23:56.814173+0000', 'last_active': '2024-09-16T10:29:01.423237+0000', 'last_peered': '2024-09-16T10:29:01.423237+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:01.423237+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'3100", 'ondisk_log_start': "35'3100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30430150, 'num_objects': 7431, 'num_object_clones': 0, 'num_object_copies': 14862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4170, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13145, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 26, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1336}, {'shards': '2', 'objects': 3263}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:04.319 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:04.319 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "44'12808", 'reported_seq': 13012, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:01.553886+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:01.553886+0000', 'last_peered': '2024-09-16T10:29:01.553886+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:01.553886+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'2800", 'ondisk_log_start': "35'2800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29598179, 'num_objects': 7227, 'num_object_clones': 0, 'num_object_copies': 14454, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4098, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12807, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3130}, {'shards': '0,2', 'objects': 1369}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:04.319 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:04.319 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "44'13397", 'reported_seq': 13625, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:01.753216+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:01.753216+0000', 'last_peered': '2024-09-16T10:29:01.753216+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:01.753216+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'3300", 'ondisk_log_start': "35'3300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30791081, 'num_objects': 7520, 'num_object_clones': 0, 'num_object_copies': 15040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4293, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7520, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13394, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 50, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3230}, {'shards': '0,2', 'objects': 1401}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:04.319 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:04.319 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "44'12928", 'reported_seq': 13107, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:01.702451+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:01.702451+0000', 'last_peered': '2024-09-16T10:29:01.702451+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:01.702451+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'2900", 'ondisk_log_start': "35'2900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29541318, 'num_objects': 7214, 'num_object_clones': 0, 'num_object_copies': 14428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4120, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12926, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3096}, {'shards': '0,2', 'objects': 1311}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:04.319 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:04.319 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "44'13431", 'reported_seq': 13626, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:01.677051+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:01.677051+0000', 'last_peered': '2024-09-16T10:29:01.677051+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:01.677051+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'3400", 'ondisk_log_start': "35'3400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30578572, 'num_objects': 7469, 'num_object_clones': 0, 'num_object_copies': 14938, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4347, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13428, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3125}, {'shards': '0,2', 'objects': 1418}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:04.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:04.983 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:04.983 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 42452/236844 objects degraded (17.924%), 10 pgs degraded, 11 pgs undersized","count":21},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":433,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":118422,"data_bytes":484979233,"bytes_used":2086612992,"bytes_avail":287823679488,"bytes_total":289910292480,"degraded_objects":42452,"degraded_total":236844,"degraded_ratio":0.17924034385502693,"misplaced_objects":29,"misplaced_total":236844,"misplaced_ratio":0.00012244346489672528,"recovering_objects_per_sec":55,"recovering_bytes_per_sec":213449,"recovering_keys_per_sec":0,"num_objects_recovered":448,"num_bytes_recovered":1708032,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":433},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:29:05.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:05.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:05.570 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:05.572 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:05.597 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2024-09-16T10:29:04.678813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":484979233,"num_objects":118422,"num_object_clones":0,"num_object_copies":236844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42452,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":118422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187887,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8866,"num_bytes_recovered":34247158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":48,"acting":37,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2037708,"kb_used_data":1098740,"kb_used_omap":12,"kb_used_meta":938931,"kb_avail":281077812,"statfs":{"total":289910292480,"available":287823679488,"internally_reserved":0,"allocated":1125109760,"data_stored":1124272068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":961465346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,0,5,2,8,1,0,6,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14200832,"num_objects":-3467,"num_object_clones":0,"num_object_copies":-6934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-31,"num_objects_misplaced":-417,"num_objects_unfound":0,"num_objects_dirty":-3467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3467,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":448,"num_bytes_recovered":1708032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002029"},"pg_stats":[{"pgid":"2.f","version":"44'13019","reported_seq":21415,"reported_epoch":44,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-16T10:29:01.752042+0000","last_change":"2024-09-16T10:26:00.070363+0000","last_active":"2024-09-16T10:29:01.752042+0000","last_peered":"2024-09-16T10:29:01.752042+0000","last_clean":"2024-09-16T10:23:54.356173+0000","last_became_active":"2024-09-16T10:23:56.894572+0000","last_became_peered":"2024-09-16T10:23:56.894572+0000","last_unstale":"2024-09-16T10:29:01.752042+0000","last_undegraded":"2024-09-16T10:29:01.752042+0000","last_fullsized":"2024-09-16T10:23:56.367880+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:52.392724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30511104,"num_objects":7449,"num_object_clones":0,"num_object_copies":14898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":7449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13019,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4163,"num_bytes_recovered":17051648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":7420},{"shards":"0,2","objects":29}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"44'13061","reported_seq":13219,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.523266+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:01.523266+0000","last_peered":"2024-09-16T10:29:01.523266+0000","last_clean":"2024-09-16T10:29:01.523266+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:01.523266+0000","last_undegraded":"2024-09-16T10:29:01.523266+0000","last_fullsized":"2024-09-16T10:29:01.523266+0000","mapping_epoch":34,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30314496,"num_objects":7401,"num_object_clones":0,"num_object_copies":14802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9274,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"44'13191","reported_seq":13438,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.534444+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:01.534444+0000","last_peered":"2024-09-16T10:29:01.534444+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:01.534444+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30605336,"num_objects":7473,"num_object_clones":0,"num_object_copies":14946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7473,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9248,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1478},{"shards":"2","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"44'13106","reported_seq":13161,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.530190+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:01.530190+0000","last_peered":"2024-09-16T10:29:01.530190+0000","last_clean":"2024-09-16T10:29:01.530190+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:01.530190+0000","last_undegraded":"2024-09-16T10:29:01.530190+0000","last_fullsized":"2024-09-16T10:29:01.530190+0000","mapping_epoch":34,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30130176,"num_objects":7356,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13106,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"44'13102","reported_seq":13257,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.723676+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:01.723676+0000","last_peered":"2024-09-16T10:29:01.723676+0000","last_clean":"2024-09-16T10:29:01.723676+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:01.723676+0000","last_undegraded":"2024-09-16T10:29:01.723676+0000","last_fullsized":"2024-09-16T10:29:01.723676+0000","mapping_epoch":33,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30322688,"num_objects":7403,"num_object_clones":0,"num_object_copies":14806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9744,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"44'13029","reported_seq":13291,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.742515+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:01.742515+0000","last_peered":"2024-09-16T10:29:01.742515+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:01.742515+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30072832,"num_objects":7342,"num_object_clones":0,"num_object_copies":14684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9654,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3174},{"shards":"0,2","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"44'13436","reported_seq":13774,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.738143+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:01.738143+0000","last_peered":"2024-09-16T10:29:01.738143+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:01.738143+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30654464,"num_objects":7484,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3163},{"shards":"0,2","objects":1430}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"44'12979","reported_seq":13155,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:01.673493+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:01.673493+0000","last_peered":"2024-09-16T10:29:01.673493+0000","last_clean":"2024-09-16T10:29:01.673493+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:01.673493+0000","last_undegraded":"2024-09-16T10:29:01.673493+0000","last_fullsized":"2024-09-16T10:29:01.673493+0000","mapping_epoch":33,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30355456,"num_objects":7411,"num_object_clones":0,"num_object_copies":14822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8782,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943017+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:26:01.943017+0000","last_peered":"2024-09-16T10:26:01.943017+0000","last_clean":"2024-09-16T10:26:01.943017+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:26:01.943017+0000","last_undegraded":"2024-09-16T10:26:01.943017+0000","last_fullsized":"2024-09-16T10:26:01.943017+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"44'13059","reported_seq":21563,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:29:00.252112+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:00.252112+0000","last_peered":"2024-09-16T10:29:00.252112+0000","last_clean":"2024-09-16T10:29:00.252112+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:00.252112+0000","last_undegraded":"2024-09-16T10:29:00.252112+0000","last_fullsized":"2024-09-16T10:29:00.252112+0000","mapping_epoch":43,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30319558,"num_objects":7404,"num_object_clones":0,"num_object_copies":14808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13057,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.947691+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:26:01.947691+0000","last_peered":"2024-09-16T10:26:01.947691+0000","last_clean":"2024-09-16T10:26:01.947691+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:26:01.947691+0000","last_undegraded":"2024-09-16T10:26:01.947691+0000","last_fullsized":"2024-09-16T10:26:01.947691+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"44'13534","reported_seq":13746,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.731345+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:01.731345+0000","last_peered":"2024-09-16T10:29:01.731345+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:01.731345+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30745542,"num_objects":7508,"num_object_clones":0,"num_object_copies":15016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13532,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3140},{"shards":"0,2","objects":1424}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":89,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:45.501852+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:26:45.501852+0000","last_peered":"2024-09-16T10:26:45.501852+0000","last_clean":"2024-09-16T10:26:45.501852+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:26:45.501852+0000","last_undegraded":"2024-09-16T10:26:45.501852+0000","last_fullsized":"2024-09-16T10:26:45.501852+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"44'13093","reported_seq":13247,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.443105+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:01.443105+0000","last_peered":"2024-09-16T10:29:01.443105+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:01.443105+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30008262,"num_objects":7328,"num_object_clones":0,"num_object_copies":14656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13091,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1396},{"shards":"2","objects":3075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":69,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.942894+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:26:01.942894+0000","last_peered":"2024-09-16T10:26:01.942894+0000","last_clean":"2024-09-16T10:26:01.942894+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:26:01.942894+0000","last_undegraded":"2024-09-16T10:26:01.942894+0000","last_fullsized":"2024-09-16T10:26:01.942894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"44'13147","reported_seq":13283,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.423237+0000","last_change":"2024-09-16T10:23:56.814173+0000","last_active":"2024-09-16T10:29:01.423237+0000","last_peered":"2024-09-16T10:29:01.423237+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:01.423237+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30430150,"num_objects":7431,"num_object_clones":0,"num_object_copies":14862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13145,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1336},{"shards":"2","objects":3263}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"44'12808","reported_seq":13012,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.553886+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:01.553886+0000","last_peered":"2024-09-16T10:29:01.553886+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:01.553886+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2800","ondisk_log_start":"35'2800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29598179,"num_objects":7227,"num_object_clones":0,"num_object_copies":14454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12807,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3130},{"shards":"0,2","objects":1369}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943514+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:26:01.943514+0000","last_peered":"2024-09-16T10:26:01.943514+0000","last_clean":"2024-09-16T10:26:01.943514+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:26:01.943514+0000","last_undegraded":"2024-09-16T10:26:01.943514+0000","last_fullsized":"2024-09-16T10:26:01.943514+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"44'13397","reported_seq":13625,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.753216+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:01.753216+0000","last_peered":"2024-09-16T10:29:01.753216+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:01.753216+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30791081,"num_objects":7520,"num_object_clones":0,"num_object_copies":15040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4293,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13394,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3230},{"shards":"0,2","objects":1401}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":57,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.949019+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:26:01.949019+0000","last_peered":"2024-09-16T10:26:01.949019+0000","last_clean":"2024-09-16T10:26:01.949019+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:26:01.949019+0000","last_undegraded":"2024-09-16T10:26:01.949019+0000","last_fullsized":"2024-09-16T10:26:01.949019+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"44'12928","reported_seq":13107,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.702451+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:01.702451+0000","last_peered":"2024-09-16T10:29:01.702451+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:01.702451+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29541318,"num_objects":7214,"num_object_clones":0,"num_object_copies":14428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12926,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3096},{"shards":"0,2","objects":1311}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":55,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:01.943590+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:26:01.943590+0000","last_peered":"2024-09-16T10:26:01.943590+0000","last_clean":"2024-09-16T10:26:01.943590+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:26:01.943590+0000","last_undegraded":"2024-09-16T10:26:01.943590+0000","last_fullsized":"2024-09-16T10:26:01.943590+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":64,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T10:26:03.446313+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:26:03.446313+0000","last_peered":"2024-09-16T10:26:03.446313+0000","last_clean":"2024-09-16T10:26:03.446313+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:26:03.446313+0000","last_undegraded":"2024-09-16T10:26:03.446313+0000","last_fullsized":"2024-09-16T10:26:03.446313+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"44'13431","reported_seq":13626,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:01.677051+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:01.677051+0000","last_peered":"2024-09-16T10:29:01.677051+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:01.677051+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30578572,"num_objects":7469,"num_object_clones":0,"num_object_copies":14938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13428,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3125},{"shards":"0,2","objects":1418}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":484979214,"num_objects":118420,"num_object_clones":0,"num_object_copies":236840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":42452,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":118420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187885,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8865,"num_bytes_recovered":34247139,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1122664448,"data_stored":1122573205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160720,"ondisk_log_size":160720,"up":32,"acting":21,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018958,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663080,"kb_used_data":374688,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93708760,"statfs":{"total":96636764160,"available":95957770240,"internally_reserved":0,"allocated":383680512,"data_stored":383422806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,2,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.19},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1520000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":17.914000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579972,"kb_used_data":299772,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93791868,"statfs":{"total":96636764160,"available":96042872832,"internally_reserved":0,"allocated":306966528,"data_stored":306636562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":286912170},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":4.4729999999999999},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.84399999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.32500000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215186,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":794656,"kb_used_data":424280,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93577184,"statfs":{"total":96636764160,"available":95823036416,"internally_reserved":0,"allocated":434462720,"data_stored":434212700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,5,0,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":6.7430000000000003},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.5950000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":3.641},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":3.6640000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":433188864,"data_stored":433185251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306565120,"data_stored":306485175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382910464,"data_stored":382902779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:08.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:09.148 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:09.149 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:09.174 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-16T10:29:08.679784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":477655585,"num_objects":116634,"num_object_clones":0,"num_object_copies":233268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41127,"num_objects_misplaced":1142,"num_objects_unfound":0,"num_objects_dirty":116634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189675,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9078,"num_bytes_recovered":35017206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160711,"ondisk_log_size":160711,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2049156,"kb_used_data":1089696,"kb_used_omap":12,"kb_used_meta":959411,"kb_avail":281066364,"statfs":{"total":289910292480,"available":287811956736,"internally_reserved":0,"allocated":1115848704,"data_stored":1114986903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":982436866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,1,5,6,2,0,0,0,5,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15450112,"num_objects":-3772,"num_object_clones":0,"num_object_copies":-7544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1350,"num_objects_misplaced":900,"num_objects_unfound":0,"num_objects_dirty":-3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3772,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":450,"num_bytes_recovered":1642496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001997"},"pg_stats":[{"pgid":"2.f","version":"46'13141","reported_seq":21602,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.730759+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:06.730759+0000","last_peered":"2024-09-16T10:29:06.730759+0000","last_clean":"2024-09-16T10:29:06.730759+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:06.730759+0000","last_undegraded":"2024-09-16T10:29:06.730759+0000","last_fullsized":"2024-09-16T10:29:06.730759+0000","mapping_epoch":45,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30011392,"num_objects":7327,"num_object_clones":0,"num_object_copies":14654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13141,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13178","reported_seq":13338,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.515423+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:06.515423+0000","last_peered":"2024-09-16T10:29:06.515423+0000","last_clean":"2024-09-16T10:29:06.515423+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:06.515423+0000","last_undegraded":"2024-09-16T10:29:06.515423+0000","last_fullsized":"2024-09-16T10:29:06.515423+0000","mapping_epoch":34,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29835264,"num_objects":7284,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9391,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13306","reported_seq":13555,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.502562+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:06.502562+0000","last_peered":"2024-09-16T10:29:06.502562+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:06.502562+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30134296,"num_objects":7358,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9363,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1363},{"shards":"2","objects":3053}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13231","reported_seq":13290,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.459888+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:06.459888+0000","last_peered":"2024-09-16T10:29:06.459888+0000","last_clean":"2024-09-16T10:29:06.459888+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:06.459888+0000","last_undegraded":"2024-09-16T10:29:06.459888+0000","last_fullsized":"2024-09-16T10:29:06.459888+0000","mapping_epoch":34,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29618176,"num_objects":7231,"num_object_clones":0,"num_object_copies":14462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13231,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13213","reported_seq":13372,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.649523+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:06.649523+0000","last_peered":"2024-09-16T10:29:06.649523+0000","last_clean":"2024-09-16T10:29:06.649523+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:06.649523+0000","last_undegraded":"2024-09-16T10:29:06.649523+0000","last_fullsized":"2024-09-16T10:29:06.649523+0000","mapping_epoch":33,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29868032,"num_objects":7292,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13124","reported_seq":13390,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.741238+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:06.741238+0000","last_peered":"2024-09-16T10:29:06.741238+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:06.741238+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29683712,"num_objects":7247,"num_object_clones":0,"num_object_copies":14494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9749,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3080},{"shards":"0,2","objects":1281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13545","reported_seq":13894,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.754607+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:06.754607+0000","last_peered":"2024-09-16T10:29:06.754607+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:06.754607+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30208000,"num_objects":7375,"num_object_clones":0,"num_object_copies":14750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9787,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3059},{"shards":"0,2","objects":1321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13089","reported_seq":13267,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.695805+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:06.695805+0000","last_peered":"2024-09-16T10:29:06.695805+0000","last_clean":"2024-09-16T10:29:06.695805+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:06.695805+0000","last_undegraded":"2024-09-16T10:29:06.695805+0000","last_fullsized":"2024-09-16T10:29:06.695805+0000","mapping_epoch":33,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29904896,"num_objects":7301,"num_object_clones":0,"num_object_copies":14602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8892,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13164","reported_seq":21670,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:05.078815+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:05.078815+0000","last_peered":"2024-09-16T10:29:05.078815+0000","last_clean":"2024-09-16T10:29:05.078815+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:05.078815+0000","last_undegraded":"2024-09-16T10:29:05.078815+0000","last_fullsized":"2024-09-16T10:29:05.078815+0000","mapping_epoch":43,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29889478,"num_objects":7299,"num_object_clones":0,"num_object_copies":14598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13162,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'13647","reported_seq":13869,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.765051+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:06.765051+0000","last_peered":"2024-09-16T10:29:06.765051+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:06.765051+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30282694,"num_objects":7395,"num_object_clones":0,"num_object_copies":14790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13645,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3031},{"shards":"0,2","objects":1311}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13197","reported_seq":13356,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.482265+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:06.482265+0000","last_peered":"2024-09-16T10:29:06.482265+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:06.482265+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29582278,"num_objects":7224,"num_object_clones":0,"num_object_copies":14448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13195,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1292},{"shards":"2","objects":2972}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13260","reported_seq":13722,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.518520+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:06.518520+0000","last_peered":"2024-09-16T10:29:06.518520+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:06.518520+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29967302,"num_objects":7318,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2866,"num_objects_misplaced":1142,"num_objects_unfound":0,"num_objects_dirty":7318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13258,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":651264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1142},{"shards":"2","objects":3310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'12932","reported_seq":13146,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.712043+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:06.712043+0000","last_peered":"2024-09-16T10:29:06.712043+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:06.712043+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29090275,"num_objects":7103,"num_object_clones":0,"num_object_copies":14206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12931,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3010},{"shards":"0,2","objects":1245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13520","reported_seq":13756,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.773464+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:06.773464+0000","last_peered":"2024-09-16T10:29:06.773464+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:06.773464+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30287273,"num_objects":7397,"num_object_clones":0,"num_object_copies":14794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13517,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3110},{"shards":"0,2","objects":1278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.979813+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:02.979813+0000","last_peered":"2024-09-16T10:29:02.979813+0000","last_clean":"2024-09-16T10:29:02.979813+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:02.979813+0000","last_undegraded":"2024-09-16T10:29:02.979813+0000","last_fullsized":"2024-09-16T10:29:02.979813+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13031","reported_seq":13217,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.598236+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:06.598236+0000","last_peered":"2024-09-16T10:29:06.598236+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:06.598236+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119430,"num_objects":7111,"num_object_clones":0,"num_object_copies":14222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2995},{"shards":"0,2","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13530","reported_seq":13730,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.742326+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:06.742326+0000","last_peered":"2024-09-16T10:29:06.742326+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:06.742326+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30173068,"num_objects":7370,"num_object_clones":0,"num_object_copies":14740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13527,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3027},{"shards":"0,2","objects":1319}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":477655566,"num_objects":116632,"num_object_clones":0,"num_object_copies":233264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41127,"num_objects_misplaced":1142,"num_objects_unfound":0,"num_objects_dirty":116632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189673,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9077,"num_bytes_recovered":35017187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1113272320,"data_stored":1113188303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160708,"ondisk_log_size":160708,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018959,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663612,"kb_used_data":371120,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93708228,"statfs":{"total":96636764160,"available":95957225472,"internally_reserved":0,"allocated":380026880,"data_stored":379758729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,2,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.19},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1520000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":17.914000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691904,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588116,"kb_used_data":299716,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93783724,"statfs":{"total":96636764160,"available":96034533376,"internally_reserved":0,"allocated":306909184,"data_stored":306575999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":295300778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.8759999999999999},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.452}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":18.638999999999999},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.1749999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215187,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797428,"kb_used_data":418860,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93574412,"statfs":{"total":96636764160,"available":95820197888,"internally_reserved":0,"allocated":428912640,"data_stored":428652175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":387641000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,4,3,1,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.891},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.357}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.3079999999999998},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.47599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":427622400,"data_stored":427618787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306429952,"data_stored":306357233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379219968,"data_stored":379212283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:09.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:09.717 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:09.718 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:09.746 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-16T10:29:08.679784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":477655585,"num_objects":116634,"num_object_clones":0,"num_object_copies":233268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41127,"num_objects_misplaced":1142,"num_objects_unfound":0,"num_objects_dirty":116634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189675,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9078,"num_bytes_recovered":35017206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160711,"ondisk_log_size":160711,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2049156,"kb_used_data":1089696,"kb_used_omap":12,"kb_used_meta":959411,"kb_avail":281066364,"statfs":{"total":289910292480,"available":287811956736,"internally_reserved":0,"allocated":1115848704,"data_stored":1114986903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":982436866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,1,5,6,2,0,0,0,5,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15450112,"num_objects":-3772,"num_object_clones":0,"num_object_copies":-7544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1350,"num_objects_misplaced":900,"num_objects_unfound":0,"num_objects_dirty":-3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3772,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":450,"num_bytes_recovered":1642496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001997"},"pg_stats":[{"pgid":"2.f","version":"46'13141","reported_seq":21602,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.730759+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:06.730759+0000","last_peered":"2024-09-16T10:29:06.730759+0000","last_clean":"2024-09-16T10:29:06.730759+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:06.730759+0000","last_undegraded":"2024-09-16T10:29:06.730759+0000","last_fullsized":"2024-09-16T10:29:06.730759+0000","mapping_epoch":45,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30011392,"num_objects":7327,"num_object_clones":0,"num_object_copies":14654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13141,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13178","reported_seq":13338,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.515423+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:06.515423+0000","last_peered":"2024-09-16T10:29:06.515423+0000","last_clean":"2024-09-16T10:29:06.515423+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:06.515423+0000","last_undegraded":"2024-09-16T10:29:06.515423+0000","last_fullsized":"2024-09-16T10:29:06.515423+0000","mapping_epoch":34,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29835264,"num_objects":7284,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9391,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13306","reported_seq":13555,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.502562+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:06.502562+0000","last_peered":"2024-09-16T10:29:06.502562+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:06.502562+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30134296,"num_objects":7358,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9363,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1363},{"shards":"2","objects":3053}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13231","reported_seq":13290,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.459888+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:06.459888+0000","last_peered":"2024-09-16T10:29:06.459888+0000","last_clean":"2024-09-16T10:29:06.459888+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:06.459888+0000","last_undegraded":"2024-09-16T10:29:06.459888+0000","last_fullsized":"2024-09-16T10:29:06.459888+0000","mapping_epoch":34,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29618176,"num_objects":7231,"num_object_clones":0,"num_object_copies":14462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13231,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13213","reported_seq":13372,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.649523+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:06.649523+0000","last_peered":"2024-09-16T10:29:06.649523+0000","last_clean":"2024-09-16T10:29:06.649523+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:06.649523+0000","last_undegraded":"2024-09-16T10:29:06.649523+0000","last_fullsized":"2024-09-16T10:29:06.649523+0000","mapping_epoch":33,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29868032,"num_objects":7292,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13124","reported_seq":13390,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.741238+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:06.741238+0000","last_peered":"2024-09-16T10:29:06.741238+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:06.741238+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29683712,"num_objects":7247,"num_object_clones":0,"num_object_copies":14494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9749,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3080},{"shards":"0,2","objects":1281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13545","reported_seq":13894,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.754607+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:06.754607+0000","last_peered":"2024-09-16T10:29:06.754607+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:06.754607+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30208000,"num_objects":7375,"num_object_clones":0,"num_object_copies":14750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9787,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3059},{"shards":"0,2","objects":1321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13089","reported_seq":13267,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.695805+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:06.695805+0000","last_peered":"2024-09-16T10:29:06.695805+0000","last_clean":"2024-09-16T10:29:06.695805+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:06.695805+0000","last_undegraded":"2024-09-16T10:29:06.695805+0000","last_fullsized":"2024-09-16T10:29:06.695805+0000","mapping_epoch":33,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29904896,"num_objects":7301,"num_object_clones":0,"num_object_copies":14602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8892,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13164","reported_seq":21670,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:05.078815+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:05.078815+0000","last_peered":"2024-09-16T10:29:05.078815+0000","last_clean":"2024-09-16T10:29:05.078815+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:05.078815+0000","last_undegraded":"2024-09-16T10:29:05.078815+0000","last_fullsized":"2024-09-16T10:29:05.078815+0000","mapping_epoch":43,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29889478,"num_objects":7299,"num_object_clones":0,"num_object_copies":14598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13162,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'13647","reported_seq":13869,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.765051+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:06.765051+0000","last_peered":"2024-09-16T10:29:06.765051+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:06.765051+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30282694,"num_objects":7395,"num_object_clones":0,"num_object_copies":14790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13645,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3031},{"shards":"0,2","objects":1311}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13197","reported_seq":13356,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.482265+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:06.482265+0000","last_peered":"2024-09-16T10:29:06.482265+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:06.482265+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29582278,"num_objects":7224,"num_object_clones":0,"num_object_copies":14448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13195,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1292},{"shards":"2","objects":2972}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13260","reported_seq":13722,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.518520+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:06.518520+0000","last_peered":"2024-09-16T10:29:06.518520+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:06.518520+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29967302,"num_objects":7318,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2866,"num_objects_misplaced":1142,"num_objects_unfound":0,"num_objects_dirty":7318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13258,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":651264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1142},{"shards":"2","objects":3310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'12932","reported_seq":13146,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.712043+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:06.712043+0000","last_peered":"2024-09-16T10:29:06.712043+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:06.712043+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29090275,"num_objects":7103,"num_object_clones":0,"num_object_copies":14206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12931,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3010},{"shards":"0,2","objects":1245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13520","reported_seq":13756,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.773464+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:06.773464+0000","last_peered":"2024-09-16T10:29:06.773464+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:06.773464+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30287273,"num_objects":7397,"num_object_clones":0,"num_object_copies":14794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13517,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3110},{"shards":"0,2","objects":1278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.979813+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:02.979813+0000","last_peered":"2024-09-16T10:29:02.979813+0000","last_clean":"2024-09-16T10:29:02.979813+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:02.979813+0000","last_undegraded":"2024-09-16T10:29:02.979813+0000","last_fullsized":"2024-09-16T10:29:02.979813+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13031","reported_seq":13217,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.598236+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:06.598236+0000","last_peered":"2024-09-16T10:29:06.598236+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:06.598236+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119430,"num_objects":7111,"num_object_clones":0,"num_object_copies":14222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2995},{"shards":"0,2","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13530","reported_seq":13730,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.742326+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:06.742326+0000","last_peered":"2024-09-16T10:29:06.742326+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:06.742326+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30173068,"num_objects":7370,"num_object_clones":0,"num_object_copies":14740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13527,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3027},{"shards":"0,2","objects":1319}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":477655566,"num_objects":116632,"num_object_clones":0,"num_object_copies":233264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41127,"num_objects_misplaced":1142,"num_objects_unfound":0,"num_objects_dirty":116632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189673,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9077,"num_bytes_recovered":35017187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1113272320,"data_stored":1113188303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160708,"ondisk_log_size":160708,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018959,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663612,"kb_used_data":371120,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93708228,"statfs":{"total":96636764160,"available":95957225472,"internally_reserved":0,"allocated":380026880,"data_stored":379758729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,2,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.19},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1520000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":17.914000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691904,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588116,"kb_used_data":299716,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93783724,"statfs":{"total":96636764160,"available":96034533376,"internally_reserved":0,"allocated":306909184,"data_stored":306575999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":295300778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":1.6599999999999999,"5min":3.548,"15min":3.5819999999999999},"min":{"1min":0.184,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.236000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.8759999999999999},{"interface":"front","average":{"1min":3.3380000000000001,"5min":3.2029999999999998,"15min":2.6709999999999998},"min":{"1min":0.29299999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":12.195,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.452}]},{"osd":2,"last update":"Mon Sep 16 10:28:07 2024","interfaces":[{"interface":"back","average":{"1min":2.9660000000000002,"5min":2.6000000000000001,"15min":2.4119999999999999},"min":{"1min":0.34000000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":13.579000000000001,"5min":13.579000000000001,"15min":13.579000000000001},"last":18.638999999999999},{"interface":"front","average":{"1min":3.3540000000000001,"5min":2.9620000000000002,"15min":2.9159999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":10.664999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.1749999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215187,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797428,"kb_used_data":418860,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93574412,"statfs":{"total":96636764160,"available":95820197888,"internally_reserved":0,"allocated":428912640,"data_stored":428652175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":387641000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,4,3,1,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.891},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.357}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.3079999999999998},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.47599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":427622400,"data_stored":427618787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306429952,"data_stored":306357233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379219968,"data_stored":379212283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:09.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:09.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'13306", 'reported_seq': 13555, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:06.502562+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:06.502562+0000', 'last_peered': '2024-09-16T10:29:06.502562+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:06.502562+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'3300", 'ondisk_log_start': "35'3300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30134296, 'num_objects': 7358, 'num_object_clones': 0, 'num_object_copies': 14716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7358, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9363, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1363}, {'shards': '2', 'objects': 3053}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:09.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:09.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'13124", 'reported_seq': 13390, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:06.741238+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:06.741238+0000', 'last_peered': '2024-09-16T10:29:06.741238+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:06.741238+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'3100", 'ondisk_log_start': "35'3100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29683712, 'num_objects': 7247, 'num_object_clones': 0, 'num_object_copies': 14494, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4169, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7247, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9749, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 50, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3080}, {'shards': '0,2', 'objects': 1281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:09.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:09.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'13545", 'reported_seq': 13894, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:06.754607+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:06.754607+0000', 'last_peered': '2024-09-16T10:29:06.754607+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:06.754607+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'3500", 'ondisk_log_start': "35'3500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30208000, 'num_objects': 7375, 'num_object_clones': 0, 'num_object_copies': 14750, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4318, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7375, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9787, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 87, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3059}, {'shards': '0,2', 'objects': 1321}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:09.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:09.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'13647", 'reported_seq': 13869, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:06.765051+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:06.765051+0000', 'last_peered': '2024-09-16T10:29:06.765051+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:06.765051+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'3600", 'ondisk_log_start': "35'3600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30282694, 'num_objects': 7395, 'num_object_clones': 0, 'num_object_copies': 14790, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4366, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13645, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 72, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3031}, {'shards': '0,2', 'objects': 1311}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'13197", 'reported_seq': 13356, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:06.482265+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:06.482265+0000', 'last_peered': '2024-09-16T10:29:06.482265+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:06.482265+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'3100", 'ondisk_log_start': "35'3100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29582278, 'num_objects': 7224, 'num_object_clones': 0, 'num_object_copies': 14448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4254, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7224, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13195, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1292}, {'shards': '2', 'objects': 2972}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'13260", 'reported_seq': 13722, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:06.518520+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:29:06.518520+0000', 'last_peered': '2024-09-16T10:29:06.518520+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:06.518520+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'3200", 'ondisk_log_start': "35'3200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29967302, 'num_objects': 7318, 'num_object_clones': 0, 'num_object_copies': 14636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2866, 'num_objects_misplaced': 1142, 'num_objects_unfound': 0, 'num_objects_dirty': 7318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13258, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 188, 'num_bytes_recovered': 651264, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1142}, {'shards': '2', 'objects': 3310}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'12932", 'reported_seq': 13146, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:06.712043+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:06.712043+0000', 'last_peered': '2024-09-16T10:29:06.712043+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:06.712043+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'2900", 'ondisk_log_start': "35'2900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29090275, 'num_objects': 7103, 'num_object_clones': 0, 'num_object_copies': 14206, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4094, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12931, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3010}, {'shards': '0,2', 'objects': 1245}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'13520", 'reported_seq': 13756, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:06.773464+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:06.773464+0000', 'last_peered': '2024-09-16T10:29:06.773464+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:06.773464+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'3500", 'ondisk_log_start': "35'3500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30287273, 'num_objects': 7397, 'num_object_clones': 0, 'num_object_copies': 14794, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4290, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13517, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 53, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3110}, {'shards': '0,2', 'objects': 1278}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'13031", 'reported_seq': 13217, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:06.598236+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:06.598236+0000', 'last_peered': '2024-09-16T10:29:06.598236+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:06.598236+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'3000", 'ondisk_log_start': "35'3000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29119430, 'num_objects': 7111, 'num_object_clones': 0, 'num_object_copies': 14222, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4118, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13029, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 53, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2995}, {'shards': '0,2', 'objects': 1208}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:09.749 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'13530", 'reported_seq': 13730, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:06.742326+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:06.742326+0000', 'last_peered': '2024-09-16T10:29:06.742326+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:06.742326+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'3500", 'ondisk_log_start': "35'3500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30173068, 'num_objects': 7370, 'num_object_clones': 0, 'num_object_copies': 14740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4346, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13527, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 57, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 3027}, {'shards': '0,2', 'objects': 1319}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:09.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:10.419 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:10.419 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41127/233268 objects degraded (17.631%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":439,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":116634,"data_bytes":477655585,"bytes_used":2098335744,"bytes_avail":287811956736,"bytes_total":289910292480,"degraded_objects":41127,"degraded_total":233268,"degraded_ratio":0.17630793765111374,"misplaced_objects":1142,"misplaced_total":233268,"misplaced_ratio":0.0048956564981051838,"recovering_objects_per_sec":56,"recovering_bytes_per_sec":205260,"recovering_keys_per_sec":0,"num_objects_recovered":450,"num_bytes_recovered":1642496,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":471},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:29:10.445 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:10.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:10.989 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:10.991 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:11.024 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-16T10:29:10.680552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":477102625,"num_objects":116499,"num_object_clones":0,"num_object_copies":232998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41127,"num_objects_misplaced":1142,"num_objects_unfound":0,"num_objects_dirty":116499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189810,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9078,"num_bytes_recovered":35017206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160746,"ondisk_log_size":160746,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2052196,"kb_used_data":1088648,"kb_used_omap":12,"kb_used_meta":963507,"kb_avail":281063324,"statfs":{"total":289910292480,"available":287808843776,"internally_reserved":0,"allocated":1114775552,"data_stored":1113913751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":986631170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,1,6,7,2,0,0,0,5,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7876608,"num_objects":-1923,"num_object_clones":0,"num_object_copies":-3846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1325,"num_objects_misplaced":1113,"num_objects_unfound":0,"num_objects_dirty":-1923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1923,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":770048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002264"},"pg_stats":[{"pgid":"2.f","version":"46'13141","reported_seq":21602,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.730759+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:06.730759+0000","last_peered":"2024-09-16T10:29:06.730759+0000","last_clean":"2024-09-16T10:29:06.730759+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:06.730759+0000","last_undegraded":"2024-09-16T10:29:06.730759+0000","last_fullsized":"2024-09-16T10:29:06.730759+0000","mapping_epoch":45,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30011392,"num_objects":7327,"num_object_clones":0,"num_object_copies":14654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13141,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13178","reported_seq":13338,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.515423+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:06.515423+0000","last_peered":"2024-09-16T10:29:06.515423+0000","last_clean":"2024-09-16T10:29:06.515423+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:06.515423+0000","last_undegraded":"2024-09-16T10:29:06.515423+0000","last_fullsized":"2024-09-16T10:29:06.515423+0000","mapping_epoch":34,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29835264,"num_objects":7284,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9391,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13306","reported_seq":13555,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.502562+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:06.502562+0000","last_peered":"2024-09-16T10:29:06.502562+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:06.502562+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30134296,"num_objects":7358,"num_object_clones":0,"num_object_copies":14716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7358,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9363,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1363},{"shards":"2","objects":3053}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13231","reported_seq":13290,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.459888+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:06.459888+0000","last_peered":"2024-09-16T10:29:06.459888+0000","last_clean":"2024-09-16T10:29:06.459888+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:06.459888+0000","last_undegraded":"2024-09-16T10:29:06.459888+0000","last_fullsized":"2024-09-16T10:29:06.459888+0000","mapping_epoch":34,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29618176,"num_objects":7231,"num_object_clones":0,"num_object_copies":14462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13231,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13213","reported_seq":13372,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.649523+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:06.649523+0000","last_peered":"2024-09-16T10:29:06.649523+0000","last_clean":"2024-09-16T10:29:06.649523+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:06.649523+0000","last_undegraded":"2024-09-16T10:29:06.649523+0000","last_fullsized":"2024-09-16T10:29:06.649523+0000","mapping_epoch":33,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29868032,"num_objects":7292,"num_object_clones":0,"num_object_copies":14584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9855,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13124","reported_seq":13390,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.741238+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:06.741238+0000","last_peered":"2024-09-16T10:29:06.741238+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:06.741238+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29683712,"num_objects":7247,"num_object_clones":0,"num_object_copies":14494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9749,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3080},{"shards":"0,2","objects":1281}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13545","reported_seq":13894,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.754607+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:06.754607+0000","last_peered":"2024-09-16T10:29:06.754607+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:06.754607+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30208000,"num_objects":7375,"num_object_clones":0,"num_object_copies":14750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9787,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3059},{"shards":"0,2","objects":1321}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13089","reported_seq":13267,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.695805+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:06.695805+0000","last_peered":"2024-09-16T10:29:06.695805+0000","last_clean":"2024-09-16T10:29:06.695805+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:06.695805+0000","last_undegraded":"2024-09-16T10:29:06.695805+0000","last_fullsized":"2024-09-16T10:29:06.695805+0000","mapping_epoch":33,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29904896,"num_objects":7301,"num_object_clones":0,"num_object_copies":14602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8892,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13299","reported_seq":21805,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:10.122336+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:10.122336+0000","last_peered":"2024-09-16T10:29:10.122336+0000","last_clean":"2024-09-16T10:29:10.122336+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:10.122336+0000","last_undegraded":"2024-09-16T10:29:10.122336+0000","last_fullsized":"2024-09-16T10:29:10.122336+0000","mapping_epoch":43,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29336518,"num_objects":7164,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13297,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'13647","reported_seq":13869,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.765051+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:06.765051+0000","last_peered":"2024-09-16T10:29:06.765051+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:06.765051+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30282694,"num_objects":7395,"num_object_clones":0,"num_object_copies":14790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13645,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3031},{"shards":"0,2","objects":1311}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13197","reported_seq":13356,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.482265+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:06.482265+0000","last_peered":"2024-09-16T10:29:06.482265+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:06.482265+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29582278,"num_objects":7224,"num_object_clones":0,"num_object_copies":14448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13195,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1292},{"shards":"2","objects":2972}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13260","reported_seq":13722,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.518520+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:06.518520+0000","last_peered":"2024-09-16T10:29:06.518520+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:06.518520+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29967302,"num_objects":7318,"num_object_clones":0,"num_object_copies":14636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2866,"num_objects_misplaced":1142,"num_objects_unfound":0,"num_objects_dirty":7318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13258,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":651264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1142},{"shards":"2","objects":3310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'12932","reported_seq":13146,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.712043+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:06.712043+0000","last_peered":"2024-09-16T10:29:06.712043+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:06.712043+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'2900","ondisk_log_start":"35'2900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29090275,"num_objects":7103,"num_object_clones":0,"num_object_copies":14206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12931,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3010},{"shards":"0,2","objects":1245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13520","reported_seq":13756,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.773464+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:06.773464+0000","last_peered":"2024-09-16T10:29:06.773464+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:06.773464+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30287273,"num_objects":7397,"num_object_clones":0,"num_object_copies":14794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4290,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13517,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3110},{"shards":"0,2","objects":1278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":59,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.979813+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:02.979813+0000","last_peered":"2024-09-16T10:29:02.979813+0000","last_clean":"2024-09-16T10:29:02.979813+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:02.979813+0000","last_undegraded":"2024-09-16T10:29:02.979813+0000","last_fullsized":"2024-09-16T10:29:02.979813+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13031","reported_seq":13217,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.598236+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:06.598236+0000","last_peered":"2024-09-16T10:29:06.598236+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:06.598236+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29119430,"num_objects":7111,"num_object_clones":0,"num_object_copies":14222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13029,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2995},{"shards":"0,2","objects":1208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13530","reported_seq":13730,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:06.742326+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:06.742326+0000","last_peered":"2024-09-16T10:29:06.742326+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:06.742326+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30173068,"num_objects":7370,"num_object_clones":0,"num_object_copies":14740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13527,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":3027},{"shards":"0,2","objects":1319}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":477102606,"num_objects":116497,"num_object_clones":0,"num_object_copies":232994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41127,"num_objects_misplaced":1142,"num_objects_unfound":0,"num_objects_dirty":116497,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189808,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9077,"num_bytes_recovered":35017187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1112203264,"data_stored":1112119247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160743,"ondisk_log_size":160743,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018959,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":663612,"kb_used_data":371120,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93708228,"statfs":{"total":96636764160,"available":95957225472,"internally_reserved":0,"allocated":380026880,"data_stored":379758729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,2,1,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.19},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1520000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":17.914000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591156,"kb_used_data":298668,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93780684,"statfs":{"total":96636764160,"available":96031420416,"internally_reserved":0,"allocated":305836032,"data_stored":305502847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":299495082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.5449999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.57699999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215187,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":797428,"kb_used_data":418860,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93574412,"statfs":{"total":96636764160,"available":95820197888,"internally_reserved":0,"allocated":428912640,"data_stored":428652175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":387641000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,4,3,1,0,0,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":3.0880000000000001,"5min":2.738,"15min":2.786},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":10.762,"5min":10.762,"15min":10.762},"last":1.891},{"interface":"front","average":{"1min":3.6829999999999998,"5min":2.3479999999999999,"15min":2.1309999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":12.999000000000001,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.357}]},{"osd":2,"last update":"Mon Sep 16 10:28:02 2024","interfaces":[{"interface":"back","average":{"1min":2.3530000000000002,"5min":2.2890000000000001,"15min":1.165},"min":{"1min":0.28399999999999997,"5min":0.182,"15min":0.182},"max":{"1min":10.382999999999999,"5min":18.093,"15min":18.093},"last":2.3079999999999998},{"interface":"front","average":{"1min":1.913,"5min":2.214,"15min":1.5129999999999999},"min":{"1min":0.24199999999999999,"5min":0.192,"15min":0.192},"max":{"1min":5.5190000000000001,"5min":9.3290000000000006,"15min":9.3290000000000006},"last":0.47599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":427622400,"data_stored":427618787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305360896,"data_stored":305288177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":379219968,"data_stored":379212283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:14.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:14.597 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:14.597 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:14.624 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-16T10:29:12.681436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468972065,"num_objects":114514,"num_object_clones":0,"num_object_copies":229028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41108,"num_objects_misplaced":1009,"num_objects_unfound":0,"num_objects_dirty":114514,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191795,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9230,"num_bytes_recovered":35561974,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160931,"ondisk_log_size":160931,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2033428,"kb_used_data":1078076,"kb_used_omap":12,"kb_used_meta":955315,"kb_avail":281082092,"statfs":{"total":289910292480,"available":287828062208,"internally_reserved":0,"allocated":1103949824,"data_stored":1103088023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":978242562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,7,5,6,1,0,2,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16007168,"num_objects":-3908,"num_object_clones":0,"num_object_copies":-7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1344,"num_objects_misplaced":980,"num_objects_unfound":0,"num_objects_dirty":-3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3908,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":364,"num_bytes_recovered":1314816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.715186"},"pg_stats":[{"pgid":"2.f","version":"46'13279","reported_seq":21743,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.802568+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:11.802568+0000","last_peered":"2024-09-16T10:29:11.802568+0000","last_clean":"2024-09-16T10:29:11.802568+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:11.802568+0000","last_undegraded":"2024-09-16T10:29:11.802568+0000","last_fullsized":"2024-09-16T10:29:11.802568+0000","mapping_epoch":45,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29446144,"num_objects":7189,"num_object_clones":0,"num_object_copies":14378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13279,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13305","reported_seq":13465,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.502062+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:11.502062+0000","last_peered":"2024-09-16T10:29:11.502062+0000","last_clean":"2024-09-16T10:29:11.502062+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:11.502062+0000","last_undegraded":"2024-09-16T10:29:11.502062+0000","last_fullsized":"2024-09-16T10:29:11.502062+0000","mapping_epoch":34,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29315072,"num_objects":7157,"num_object_clones":0,"num_object_copies":14314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9518,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13458","reported_seq":13707,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.509386+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:11.509386+0000","last_peered":"2024-09-16T10:29:11.509386+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:11.509386+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29511704,"num_objects":7206,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9515,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1211},{"shards":"2","objects":2901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13359","reported_seq":13418,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.486625+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:11.486625+0000","last_peered":"2024-09-16T10:29:11.486625+0000","last_clean":"2024-09-16T10:29:11.486625+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:11.486625+0000","last_undegraded":"2024-09-16T10:29:11.486625+0000","last_fullsized":"2024-09-16T10:29:11.486625+0000","mapping_epoch":34,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29093888,"num_objects":7103,"num_object_clones":0,"num_object_copies":14206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13359,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13351","reported_seq":13511,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.789897+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:11.789897+0000","last_peered":"2024-09-16T10:29:11.789897+0000","last_clean":"2024-09-16T10:29:11.789897+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:11.789897+0000","last_undegraded":"2024-09-16T10:29:11.789897+0000","last_fullsized":"2024-09-16T10:29:11.789897+0000","mapping_epoch":33,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29302784,"num_objects":7154,"num_object_clones":0,"num_object_copies":14308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13241","reported_seq":13517,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.743949+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:11.743949+0000","last_peered":"2024-09-16T10:29:11.743949+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:11.743949+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29204480,"num_objects":7130,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9866,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2968},{"shards":"0,2","objects":1164}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13688","reported_seq":14046,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.791029+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:11.791029+0000","last_peered":"2024-09-16T10:29:11.791029+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:11.791029+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622272,"num_objects":7232,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9930,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2920},{"shards":"0,2","objects":1178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13247","reported_seq":13425,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.728425+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:11.728425+0000","last_peered":"2024-09-16T10:29:11.728425+0000","last_clean":"2024-09-16T10:29:11.728425+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:11.728425+0000","last_undegraded":"2024-09-16T10:29:11.728425+0000","last_fullsized":"2024-09-16T10:29:11.728425+0000","mapping_epoch":33,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29257728,"num_objects":7143,"num_object_clones":0,"num_object_copies":14286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9050,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13299","reported_seq":21805,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:10.122336+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:10.122336+0000","last_peered":"2024-09-16T10:29:10.122336+0000","last_clean":"2024-09-16T10:29:10.122336+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:10.122336+0000","last_undegraded":"2024-09-16T10:29:10.122336+0000","last_fullsized":"2024-09-16T10:29:10.122336+0000","mapping_epoch":43,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29336518,"num_objects":7164,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13297,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'13783","reported_seq":14003,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.769009+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:11.769009+0000","last_peered":"2024-09-16T10:29:11.769009+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:11.769009+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29725638,"num_objects":7259,"num_object_clones":0,"num_object_copies":14518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13781,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":1175}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13331","reported_seq":13490,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.515016+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:11.515016+0000","last_peered":"2024-09-16T10:29:11.515016+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:11.515016+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29033414,"num_objects":7090,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13329,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1158},{"shards":"2","objects":2838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13376","reported_seq":14104,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.549197+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:11.549197+0000","last_peered":"2024-09-16T10:29:11.549197+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:11.549197+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29492166,"num_objects":7202,"num_object_clones":0,"num_object_copies":14404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2866,"num_objects_misplaced":1009,"num_objects_unfound":0,"num_objects_dirty":7202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13374,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1009},{"shards":"2","objects":3327}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13049","reported_seq":13267,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.758540+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:11.758540+0000","last_peered":"2024-09-16T10:29:11.758540+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:11.758540+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28611043,"num_objects":6986,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13048,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":1128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13643","reported_seq":13880,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.836495+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:11.836495+0000","last_peered":"2024-09-16T10:29:11.836495+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:11.836495+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29783465,"num_objects":7274,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13640,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2988},{"shards":"0,2","objects":1155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13152","reported_seq":13344,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.753326+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:11.753326+0000","last_peered":"2024-09-16T10:29:11.753326+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:11.753326+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28623814,"num_objects":6990,"num_object_clones":0,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13150,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2877},{"shards":"0,2","objects":1087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13667","reported_seq":13875,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.822573+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:11.822573+0000","last_peered":"2024-09-16T10:29:11.822573+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:11.822573+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29611916,"num_objects":7233,"num_object_clones":0,"num_object_copies":14466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13664,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2894},{"shards":"0,2","objects":1182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":468972046,"num_objects":114512,"num_object_clones":0,"num_object_copies":229024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41108,"num_objects_misplaced":1009,"num_objects_unfound":0,"num_objects_dirty":114512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191793,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9229,"num_bytes_recovered":35561955,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1101275136,"data_stored":1101191119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018960,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":647028,"kb_used_data":366828,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93724812,"statfs":{"total":96636764160,"available":95974207488,"internally_reserved":0,"allocated":375631872,"data_stored":375363721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":286912176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1,0,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1890000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.6579999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591156,"kb_used_data":298668,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93780684,"statfs":{"total":96636764160,"available":96031420416,"internally_reserved":0,"allocated":305836032,"data_stored":305502847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":299495082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.5449999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.57699999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215188,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":795244,"kb_used_data":412580,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93576596,"statfs":{"total":96636764160,"available":95822434304,"internally_reserved":0,"allocated":422481920,"data_stored":422221455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":391835304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,6,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.214},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":421126144,"data_stored":421122531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305360896,"data_stored":305288177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374788096,"data_stored":374780411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:14.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:15.179 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:15.180 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:15.206 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-16T10:29:14.682367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468972065,"num_objects":114514,"num_object_clones":0,"num_object_copies":229028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41108,"num_objects_misplaced":1009,"num_objects_unfound":0,"num_objects_dirty":114514,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191795,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9230,"num_bytes_recovered":35561974,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160931,"ondisk_log_size":160931,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2033428,"kb_used_data":1078076,"kb_used_omap":12,"kb_used_meta":955315,"kb_avail":281082092,"statfs":{"total":289910292480,"available":287828062208,"internally_reserved":0,"allocated":1103949824,"data_stored":1103088023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":978242562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,7,5,6,1,0,2,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16007168,"num_objects":-3908,"num_object_clones":0,"num_object_copies":-7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1344,"num_objects_misplaced":980,"num_objects_unfound":0,"num_objects_dirty":-3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3908,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":364,"num_bytes_recovered":1314816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.715207"},"pg_stats":[{"pgid":"2.f","version":"46'13279","reported_seq":21743,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.802568+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:11.802568+0000","last_peered":"2024-09-16T10:29:11.802568+0000","last_clean":"2024-09-16T10:29:11.802568+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:11.802568+0000","last_undegraded":"2024-09-16T10:29:11.802568+0000","last_fullsized":"2024-09-16T10:29:11.802568+0000","mapping_epoch":45,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29446144,"num_objects":7189,"num_object_clones":0,"num_object_copies":14378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13279,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13305","reported_seq":13465,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.502062+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:11.502062+0000","last_peered":"2024-09-16T10:29:11.502062+0000","last_clean":"2024-09-16T10:29:11.502062+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:11.502062+0000","last_undegraded":"2024-09-16T10:29:11.502062+0000","last_fullsized":"2024-09-16T10:29:11.502062+0000","mapping_epoch":34,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29315072,"num_objects":7157,"num_object_clones":0,"num_object_copies":14314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9518,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13458","reported_seq":13707,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.509386+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:11.509386+0000","last_peered":"2024-09-16T10:29:11.509386+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:11.509386+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29511704,"num_objects":7206,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9515,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1211},{"shards":"2","objects":2901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13359","reported_seq":13418,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.486625+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:11.486625+0000","last_peered":"2024-09-16T10:29:11.486625+0000","last_clean":"2024-09-16T10:29:11.486625+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:11.486625+0000","last_undegraded":"2024-09-16T10:29:11.486625+0000","last_fullsized":"2024-09-16T10:29:11.486625+0000","mapping_epoch":34,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29093888,"num_objects":7103,"num_object_clones":0,"num_object_copies":14206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13359,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13351","reported_seq":13511,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.789897+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:11.789897+0000","last_peered":"2024-09-16T10:29:11.789897+0000","last_clean":"2024-09-16T10:29:11.789897+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:11.789897+0000","last_undegraded":"2024-09-16T10:29:11.789897+0000","last_fullsized":"2024-09-16T10:29:11.789897+0000","mapping_epoch":33,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29302784,"num_objects":7154,"num_object_clones":0,"num_object_copies":14308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13241","reported_seq":13517,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.743949+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:11.743949+0000","last_peered":"2024-09-16T10:29:11.743949+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:11.743949+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29204480,"num_objects":7130,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9866,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2968},{"shards":"0,2","objects":1164}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13688","reported_seq":14046,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.791029+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:11.791029+0000","last_peered":"2024-09-16T10:29:11.791029+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:11.791029+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622272,"num_objects":7232,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9930,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2920},{"shards":"0,2","objects":1178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13247","reported_seq":13425,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.728425+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:11.728425+0000","last_peered":"2024-09-16T10:29:11.728425+0000","last_clean":"2024-09-16T10:29:11.728425+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:11.728425+0000","last_undegraded":"2024-09-16T10:29:11.728425+0000","last_fullsized":"2024-09-16T10:29:11.728425+0000","mapping_epoch":33,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29257728,"num_objects":7143,"num_object_clones":0,"num_object_copies":14286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9050,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13299","reported_seq":21805,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:10.122336+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:10.122336+0000","last_peered":"2024-09-16T10:29:10.122336+0000","last_clean":"2024-09-16T10:29:10.122336+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:10.122336+0000","last_undegraded":"2024-09-16T10:29:10.122336+0000","last_fullsized":"2024-09-16T10:29:10.122336+0000","mapping_epoch":43,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29336518,"num_objects":7164,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13297,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'13783","reported_seq":14003,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.769009+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:11.769009+0000","last_peered":"2024-09-16T10:29:11.769009+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:11.769009+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29725638,"num_objects":7259,"num_object_clones":0,"num_object_copies":14518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13781,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":1175}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13331","reported_seq":13490,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.515016+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:11.515016+0000","last_peered":"2024-09-16T10:29:11.515016+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:11.515016+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29033414,"num_objects":7090,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13329,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1158},{"shards":"2","objects":2838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13376","reported_seq":14104,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.549197+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:11.549197+0000","last_peered":"2024-09-16T10:29:11.549197+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:11.549197+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29492166,"num_objects":7202,"num_object_clones":0,"num_object_copies":14404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2866,"num_objects_misplaced":1009,"num_objects_unfound":0,"num_objects_dirty":7202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13374,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1009},{"shards":"2","objects":3327}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13049","reported_seq":13267,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.758540+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:11.758540+0000","last_peered":"2024-09-16T10:29:11.758540+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:11.758540+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28611043,"num_objects":6986,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13048,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":1128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13643","reported_seq":13880,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.836495+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:11.836495+0000","last_peered":"2024-09-16T10:29:11.836495+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:11.836495+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29783465,"num_objects":7274,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13640,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2988},{"shards":"0,2","objects":1155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13152","reported_seq":13344,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.753326+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:11.753326+0000","last_peered":"2024-09-16T10:29:11.753326+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:11.753326+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28623814,"num_objects":6990,"num_object_clones":0,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13150,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2877},{"shards":"0,2","objects":1087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13667","reported_seq":13875,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.822573+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:11.822573+0000","last_peered":"2024-09-16T10:29:11.822573+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:11.822573+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29611916,"num_objects":7233,"num_object_clones":0,"num_object_copies":14466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13664,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2894},{"shards":"0,2","objects":1182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":468972046,"num_objects":114512,"num_object_clones":0,"num_object_copies":229024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41108,"num_objects_misplaced":1009,"num_objects_unfound":0,"num_objects_dirty":114512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191793,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9229,"num_bytes_recovered":35561955,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1101275136,"data_stored":1101191119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018960,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":647028,"kb_used_data":366828,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93724812,"statfs":{"total":96636764160,"available":95974207488,"internally_reserved":0,"allocated":375631872,"data_stored":375363721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":286912176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1,0,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1890000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.6579999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591156,"kb_used_data":298668,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93780684,"statfs":{"total":96636764160,"available":96031420416,"internally_reserved":0,"allocated":305836032,"data_stored":305502847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":299495082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.5449999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.57699999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215188,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":795244,"kb_used_data":412580,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93576596,"statfs":{"total":96636764160,"available":95822434304,"internally_reserved":0,"allocated":422481920,"data_stored":422221455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":391835304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,6,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.214},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":421126144,"data_stored":421122531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305360896,"data_stored":305288177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374788096,"data_stored":374780411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:15.208 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:15.208 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'13458", 'reported_seq': 13707, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:11.509386+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:11.509386+0000', 'last_peered': '2024-09-16T10:29:11.509386+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:11.509386+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'3400", 'ondisk_log_start': "35'3400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29511704, 'num_objects': 7206, 'num_object_clones': 0, 'num_object_copies': 14412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7206, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9515, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1211}, {'shards': '2', 'objects': 2901}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:15.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:15.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'13241", 'reported_seq': 13517, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:11.743949+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:11.743949+0000', 'last_peered': '2024-09-16T10:29:11.743949+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:11.743949+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'3200", 'ondisk_log_start': "35'3200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29204480, 'num_objects': 7130, 'num_object_clones': 0, 'num_object_copies': 14260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9866, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2968}, {'shards': '0,2', 'objects': 1164}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:15.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:15.209 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'13688", 'reported_seq': 14046, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:11.791029+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:11.791029+0000', 'last_peered': '2024-09-16T10:29:11.791029+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:11.791029+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'3600", 'ondisk_log_start': "35'3600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29622272, 'num_objects': 7232, 'num_object_clones': 0, 'num_object_copies': 14464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4314, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9930, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2920}, {'shards': '0,2', 'objects': 1178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:15.210 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:15.210 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'13783", 'reported_seq': 14003, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:11.769009+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:11.769009+0000', 'last_peered': '2024-09-16T10:29:11.769009+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:11.769009+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'3700", 'ondisk_log_start': "35'3700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29725638, 'num_objects': 7259, 'num_object_clones': 0, 'num_object_copies': 14518, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4366, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13781, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 72, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2895}, {'shards': '0,2', 'objects': 1175}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:15.210 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:15.210 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'13331", 'reported_seq': 13490, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:11.515016+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:11.515016+0000', 'last_peered': '2024-09-16T10:29:11.515016+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:11.515016+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'3300", 'ondisk_log_start': "35'3300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29033414, 'num_objects': 7090, 'num_object_clones': 0, 'num_object_copies': 14180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4254, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7090, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13329, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1158}, {'shards': '2', 'objects': 2838}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:15.211 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:15.211 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'13376", 'reported_seq': 14104, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:11.549197+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:29:11.549197+0000', 'last_peered': '2024-09-16T10:29:11.549197+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:11.549197+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'3300", 'ondisk_log_start': "35'3300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29492166, 'num_objects': 7202, 'num_object_clones': 0, 'num_object_copies': 14404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2866, 'num_objects_misplaced': 1009, 'num_objects_unfound': 0, 'num_objects_dirty': 7202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13374, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 321, 'num_bytes_recovered': 1196032, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1009}, {'shards': '2', 'objects': 3327}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:15.211 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:15.211 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'13049", 'reported_seq': 13267, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:11.758540+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:11.758540+0000', 'last_peered': '2024-09-16T10:29:11.758540+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:11.758540+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'3000", 'ondisk_log_start': "35'3000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28611043, 'num_objects': 6986, 'num_object_clones': 0, 'num_object_copies': 13972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4092, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13048, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2895}, {'shards': '0,2', 'objects': 1128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:15.211 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:15.212 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'13643", 'reported_seq': 13880, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:11.836495+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:11.836495+0000', 'last_peered': '2024-09-16T10:29:11.836495+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:11.836495+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'3600", 'ondisk_log_start': "35'3600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29783465, 'num_objects': 7274, 'num_object_clones': 0, 'num_object_copies': 14548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13640, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 54, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2988}, {'shards': '0,2', 'objects': 1155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:15.212 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:15.212 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'13152", 'reported_seq': 13344, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:11.753326+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:11.753326+0000', 'last_peered': '2024-09-16T10:29:11.753326+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:11.753326+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'3100", 'ondisk_log_start': "35'3100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28623814, 'num_objects': 6990, 'num_object_clones': 0, 'num_object_copies': 13980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4115, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6990, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13150, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 56, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2877}, {'shards': '0,2', 'objects': 1087}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:15.212 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:15.212 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'13667", 'reported_seq': 13875, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:11.822573+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:11.822573+0000', 'last_peered': '2024-09-16T10:29:11.822573+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:11.822573+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'3600", 'ondisk_log_start': "35'3600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29611916, 'num_objects': 7233, 'num_object_clones': 0, 'num_object_copies': 14466, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4342, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7233, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13664, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 61, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2894}, {'shards': '0,2', 'objects': 1182}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:15.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:15.894 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:15.894 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41108/229028 objects degraded (17.949%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":444,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":114514,"data_bytes":468972065,"bytes_used":2082230272,"bytes_avail":287828062208,"bytes_total":289910292480,"degraded_objects":41108,"degraded_total":229028,"degraded_ratio":0.17948897078086523,"misplaced_objects":1009,"misplaced_total":229028,"misplaced_ratio":0.0044055748642087434,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":122705,"recovering_keys_per_sec":0,"num_objects_recovered":364,"num_bytes_recovered":1314816,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":364},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:29:15.917 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:15.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:16.450 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:16.451 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:16.477 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-16T10:29:14.682367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468972065,"num_objects":114514,"num_object_clones":0,"num_object_copies":229028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41108,"num_objects_misplaced":1009,"num_objects_unfound":0,"num_objects_dirty":114514,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191795,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9230,"num_bytes_recovered":35561974,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160931,"ondisk_log_size":160931,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2033428,"kb_used_data":1078076,"kb_used_omap":12,"kb_used_meta":955315,"kb_avail":281082092,"statfs":{"total":289910292480,"available":287828062208,"internally_reserved":0,"allocated":1103949824,"data_stored":1103088023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":978242562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,7,5,6,1,0,2,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16007168,"num_objects":-3908,"num_object_clones":0,"num_object_copies":-7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1344,"num_objects_misplaced":980,"num_objects_unfound":0,"num_objects_dirty":-3908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3908,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":364,"num_bytes_recovered":1314816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.715207"},"pg_stats":[{"pgid":"2.f","version":"46'13279","reported_seq":21743,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.802568+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:11.802568+0000","last_peered":"2024-09-16T10:29:11.802568+0000","last_clean":"2024-09-16T10:29:11.802568+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:11.802568+0000","last_undegraded":"2024-09-16T10:29:11.802568+0000","last_fullsized":"2024-09-16T10:29:11.802568+0000","mapping_epoch":45,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29446144,"num_objects":7189,"num_object_clones":0,"num_object_copies":14378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13279,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13305","reported_seq":13465,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.502062+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:11.502062+0000","last_peered":"2024-09-16T10:29:11.502062+0000","last_clean":"2024-09-16T10:29:11.502062+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:11.502062+0000","last_undegraded":"2024-09-16T10:29:11.502062+0000","last_fullsized":"2024-09-16T10:29:11.502062+0000","mapping_epoch":34,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29315072,"num_objects":7157,"num_object_clones":0,"num_object_copies":14314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9518,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13458","reported_seq":13707,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.509386+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:11.509386+0000","last_peered":"2024-09-16T10:29:11.509386+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:11.509386+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29511704,"num_objects":7206,"num_object_clones":0,"num_object_copies":14412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9515,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1211},{"shards":"2","objects":2901}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13359","reported_seq":13418,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.486625+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:11.486625+0000","last_peered":"2024-09-16T10:29:11.486625+0000","last_clean":"2024-09-16T10:29:11.486625+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:11.486625+0000","last_undegraded":"2024-09-16T10:29:11.486625+0000","last_fullsized":"2024-09-16T10:29:11.486625+0000","mapping_epoch":34,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29093888,"num_objects":7103,"num_object_clones":0,"num_object_copies":14206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13359,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13351","reported_seq":13511,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.789897+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:11.789897+0000","last_peered":"2024-09-16T10:29:11.789897+0000","last_clean":"2024-09-16T10:29:11.789897+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:11.789897+0000","last_undegraded":"2024-09-16T10:29:11.789897+0000","last_fullsized":"2024-09-16T10:29:11.789897+0000","mapping_epoch":33,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29302784,"num_objects":7154,"num_object_clones":0,"num_object_copies":14308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13241","reported_seq":13517,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.743949+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:11.743949+0000","last_peered":"2024-09-16T10:29:11.743949+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:11.743949+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29204480,"num_objects":7130,"num_object_clones":0,"num_object_copies":14260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9866,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2968},{"shards":"0,2","objects":1164}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13688","reported_seq":14046,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.791029+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:11.791029+0000","last_peered":"2024-09-16T10:29:11.791029+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:11.791029+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29622272,"num_objects":7232,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9930,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2920},{"shards":"0,2","objects":1178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13247","reported_seq":13425,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.728425+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:11.728425+0000","last_peered":"2024-09-16T10:29:11.728425+0000","last_clean":"2024-09-16T10:29:11.728425+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:11.728425+0000","last_undegraded":"2024-09-16T10:29:11.728425+0000","last_fullsized":"2024-09-16T10:29:11.728425+0000","mapping_epoch":33,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29257728,"num_objects":7143,"num_object_clones":0,"num_object_copies":14286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9050,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13299","reported_seq":21805,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:10.122336+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:10.122336+0000","last_peered":"2024-09-16T10:29:10.122336+0000","last_clean":"2024-09-16T10:29:10.122336+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:10.122336+0000","last_undegraded":"2024-09-16T10:29:10.122336+0000","last_fullsized":"2024-09-16T10:29:10.122336+0000","mapping_epoch":43,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29336518,"num_objects":7164,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13297,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'13783","reported_seq":14003,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.769009+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:11.769009+0000","last_peered":"2024-09-16T10:29:11.769009+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:11.769009+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29725638,"num_objects":7259,"num_object_clones":0,"num_object_copies":14518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13781,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":1175}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13331","reported_seq":13490,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.515016+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:11.515016+0000","last_peered":"2024-09-16T10:29:11.515016+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:11.515016+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29033414,"num_objects":7090,"num_object_clones":0,"num_object_copies":14180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13329,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1158},{"shards":"2","objects":2838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13376","reported_seq":14104,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.549197+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:11.549197+0000","last_peered":"2024-09-16T10:29:11.549197+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:11.549197+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29492166,"num_objects":7202,"num_object_clones":0,"num_object_copies":14404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2866,"num_objects_misplaced":1009,"num_objects_unfound":0,"num_objects_dirty":7202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13374,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":321,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1009},{"shards":"2","objects":3327}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13049","reported_seq":13267,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.758540+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:11.758540+0000","last_peered":"2024-09-16T10:29:11.758540+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:11.758540+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3000","ondisk_log_start":"35'3000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28611043,"num_objects":6986,"num_object_clones":0,"num_object_copies":13972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4092,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13048,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2895},{"shards":"0,2","objects":1128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13643","reported_seq":13880,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.836495+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:11.836495+0000","last_peered":"2024-09-16T10:29:11.836495+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:11.836495+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29783465,"num_objects":7274,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13640,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2988},{"shards":"0,2","objects":1155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13152","reported_seq":13344,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.753326+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:11.753326+0000","last_peered":"2024-09-16T10:29:11.753326+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:11.753326+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28623814,"num_objects":6990,"num_object_clones":0,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13150,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2877},{"shards":"0,2","objects":1087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13667","reported_seq":13875,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:11.822573+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:11.822573+0000","last_peered":"2024-09-16T10:29:11.822573+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:11.822573+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29611916,"num_objects":7233,"num_object_clones":0,"num_object_copies":14466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13664,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2894},{"shards":"0,2","objects":1182}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":468972046,"num_objects":114512,"num_object_clones":0,"num_object_copies":229024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41108,"num_objects_misplaced":1009,"num_objects_unfound":0,"num_objects_dirty":114512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191793,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9229,"num_bytes_recovered":35561955,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1101275136,"data_stored":1101191119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018960,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":647028,"kb_used_data":366828,"kb_used_omap":4,"kb_used_meta":280187,"kb_avail":93724812,"statfs":{"total":96636764160,"available":95974207488,"internally_reserved":0,"allocated":375631872,"data_stored":375363721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":286912176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1,0,0,0,0,0,0,0,3],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.29599999999999999},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.1890000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.6579999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691905,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591156,"kb_used_data":298668,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93780684,"statfs":{"total":96636764160,"available":96031420416,"internally_reserved":0,"allocated":305836032,"data_stored":305502847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":299495082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.5449999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.57699999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215188,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":795244,"kb_used_data":412580,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93576596,"statfs":{"total":96636764160,"available":95822434304,"internally_reserved":0,"allocated":422481920,"data_stored":422221455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":391835304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,6,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.214},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.56799999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":421126144,"data_stored":421122531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":305360896,"data_stored":305288177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374788096,"data_stored":374780411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:19.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:20.034 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:20.037 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:20.060 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-16T10:29:18.684150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":460206625,"num_objects":112374,"num_object_clones":0,"num_object_copies":224748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41092,"num_objects_misplaced":892,"num_objects_unfound":0,"num_objects_dirty":112374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193935,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9363,"num_bytes_recovered":36041206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2005340,"kb_used_data":1066356,"kb_used_omap":12,"kb_used_meta":938931,"kb_avail":281110180,"statfs":{"total":289910292480,"available":287856824320,"internally_reserved":0,"allocated":1091948544,"data_stored":1091086743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":961465346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,4,13,2,0,4,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21991424,"num_objects":-5369,"num_object_clones":0,"num_object_copies":-10738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-55,"num_objects_misplaced":-279,"num_objects_unfound":0,"num_objects_dirty":-5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5369,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":1142784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004861"},"pg_stats":[{"pgid":"2.f","version":"46'13439","reported_seq":21903,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.789188+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:16.789188+0000","last_peered":"2024-09-16T10:29:16.789188+0000","last_clean":"2024-09-16T10:29:16.789188+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:16.789188+0000","last_undegraded":"2024-09-16T10:29:16.789188+0000","last_fullsized":"2024-09-16T10:29:16.789188+0000","mapping_epoch":45,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28790784,"num_objects":7029,"num_object_clones":0,"num_object_copies":14058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13439,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13429","reported_seq":13588,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.535130+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:16.535130+0000","last_peered":"2024-09-16T10:29:16.535130+0000","last_clean":"2024-09-16T10:29:16.535130+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:16.535130+0000","last_undegraded":"2024-09-16T10:29:16.535130+0000","last_fullsized":"2024-09-16T10:29:16.535130+0000","mapping_epoch":34,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28807168,"num_objects":7033,"num_object_clones":0,"num_object_copies":14066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13605","reported_seq":13856,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.541703+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:16.541703+0000","last_peered":"2024-09-16T10:29:16.541703+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:16.541703+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28909592,"num_objects":7059,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9662,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1064},{"shards":"2","objects":2755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13489","reported_seq":13548,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.542787+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:16.542787+0000","last_peered":"2024-09-16T10:29:16.542787+0000","last_clean":"2024-09-16T10:29:16.542787+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:16.542787+0000","last_undegraded":"2024-09-16T10:29:16.542787+0000","last_fullsized":"2024-09-16T10:29:16.542787+0000","mapping_epoch":34,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28561408,"num_objects":6973,"num_object_clones":0,"num_object_copies":13946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13489,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13476","reported_seq":13636,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.685702+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:16.685702+0000","last_peered":"2024-09-16T10:29:16.685702+0000","last_clean":"2024-09-16T10:29:16.685702+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:16.685702+0000","last_undegraded":"2024-09-16T10:29:16.685702+0000","last_fullsized":"2024-09-16T10:29:16.685702+0000","mapping_epoch":33,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28790784,"num_objects":7029,"num_object_clones":0,"num_object_copies":14058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13354","reported_seq":13634,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.829272+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:16.829272+0000","last_peered":"2024-09-16T10:29:16.829272+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:16.829272+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28741632,"num_objects":7017,"num_object_clones":0,"num_object_copies":14034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2857},{"shards":"0,2","objects":1051}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13836","reported_seq":14198,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.710140+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:16.710140+0000","last_peered":"2024-09-16T10:29:16.710140+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:16.710140+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29016064,"num_objects":7084,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10078,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2774},{"shards":"0,2","objects":1030}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13387","reported_seq":13565,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.802981+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:16.802981+0000","last_peered":"2024-09-16T10:29:16.802981+0000","last_clean":"2024-09-16T10:29:16.802981+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:16.802981+0000","last_undegraded":"2024-09-16T10:29:16.802981+0000","last_fullsized":"2024-09-16T10:29:16.802981+0000","mapping_epoch":33,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28684288,"num_objects":7003,"num_object_clones":0,"num_object_copies":14006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9190,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13429","reported_seq":21935,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:15.406903+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:15.406903+0000","last_peered":"2024-09-16T10:29:15.406903+0000","last_clean":"2024-09-16T10:29:15.406903+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:15.406903+0000","last_undegraded":"2024-09-16T10:29:15.406903+0000","last_fullsized":"2024-09-16T10:29:15.406903+0000","mapping_epoch":43,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28804038,"num_objects":7034,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13427,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'13914","reported_seq":14138,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.858878+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:16.858878+0000","last_peered":"2024-09-16T10:29:16.858878+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:16.858878+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29189062,"num_objects":7128,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13912,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2765},{"shards":"0,2","objects":1044}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13451","reported_seq":13610,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.492411+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:16.492411+0000","last_peered":"2024-09-16T10:29:16.492411+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:16.492411+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28541894,"num_objects":6970,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13449,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1038},{"shards":"2","objects":2718}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13500","reported_seq":14467,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.552268+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:16.552268+0000","last_peered":"2024-09-16T10:29:16.552268+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:16.552268+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28984262,"num_objects":7078,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":892,"num_objects_unfound":0,"num_objects_dirty":7078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13498,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":440,"num_bytes_recovered":1675264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":892},{"shards":"2","objects":3322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13172","reported_seq":13392,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.623066+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:16.623066+0000","last_peered":"2024-09-16T10:29:16.623066+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:16.623066+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28107235,"num_objects":6863,"num_object_clones":0,"num_object_copies":13726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13171,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2773},{"shards":"0,2","objects":1005}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13790","reported_seq":14036,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.817900+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:16.817900+0000","last_peered":"2024-09-16T10:29:16.817900+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:16.817900+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29181353,"num_objects":7127,"num_object_clones":0,"num_object_copies":14254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13787,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2845},{"shards":"0,2","objects":1008}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13281","reported_seq":13476,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.870309+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:16.870309+0000","last_peered":"2024-09-16T10:29:16.870309+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:16.870309+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28095430,"num_objects":6861,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13279,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2750},{"shards":"0,2","objects":958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13816","reported_seq":14026,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.861000+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:16.861000+0000","last_peered":"2024-09-16T10:29:16.861000+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:16.861000+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29001612,"num_objects":7084,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13813,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2746},{"shards":"0,2","objects":1033}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":460206606,"num_objects":112372,"num_object_clones":0,"num_object_copies":224744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41092,"num_objects_misplaced":892,"num_objects_unfound":0,"num_objects_dirty":112372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193933,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9362,"num_bytes_recovered":36041187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089208320,"data_stored":1089124303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160868,"ondisk_log_size":160868,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018961,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":646972,"kb_used_data":362672,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93724868,"statfs":{"total":96636764160,"available":95974264832,"internally_reserved":0,"allocated":371376128,"data_stored":371107977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,0,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.956},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.58299999999999996}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.372}]}]},{"osd":1,"up_from":40,"seq":171798691906,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581880,"kb_used_data":297584,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93789960,"statfs":{"total":96636764160,"available":96040919040,"internally_reserved":0,"allocated":304726016,"data_stored":304392831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":291106474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.3290000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.2879999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.46400000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215189,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776488,"kb_used_data":406100,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93595352,"statfs":{"total":96636764160,"available":95841640448,"internally_reserved":0,"allocated":415846400,"data_stored":415585935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,8,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":2.9159999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.16800000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.093}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":414507008,"data_stored":414503395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":304173056,"data_stored":304100337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370528256,"data_stored":370520571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:20.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:20.620 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:20.620 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:20.640 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-16T10:29:18.684150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":460206625,"num_objects":112374,"num_object_clones":0,"num_object_copies":224748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41092,"num_objects_misplaced":892,"num_objects_unfound":0,"num_objects_dirty":112374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193935,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9363,"num_bytes_recovered":36041206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160871,"ondisk_log_size":160871,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2005340,"kb_used_data":1066356,"kb_used_omap":12,"kb_used_meta":938931,"kb_avail":281110180,"statfs":{"total":289910292480,"available":287856824320,"internally_reserved":0,"allocated":1091948544,"data_stored":1091086743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":961465346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,4,13,2,0,4,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21991424,"num_objects":-5369,"num_object_clones":0,"num_object_copies":-10738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-55,"num_objects_misplaced":-279,"num_objects_unfound":0,"num_objects_dirty":-5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5369,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":1142784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004861"},"pg_stats":[{"pgid":"2.f","version":"46'13439","reported_seq":21903,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.789188+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:16.789188+0000","last_peered":"2024-09-16T10:29:16.789188+0000","last_clean":"2024-09-16T10:29:16.789188+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:16.789188+0000","last_undegraded":"2024-09-16T10:29:16.789188+0000","last_fullsized":"2024-09-16T10:29:16.789188+0000","mapping_epoch":45,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28790784,"num_objects":7029,"num_object_clones":0,"num_object_copies":14058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13439,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13429","reported_seq":13588,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.535130+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:16.535130+0000","last_peered":"2024-09-16T10:29:16.535130+0000","last_clean":"2024-09-16T10:29:16.535130+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:16.535130+0000","last_undegraded":"2024-09-16T10:29:16.535130+0000","last_fullsized":"2024-09-16T10:29:16.535130+0000","mapping_epoch":34,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28807168,"num_objects":7033,"num_object_clones":0,"num_object_copies":14066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13605","reported_seq":13856,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.541703+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:16.541703+0000","last_peered":"2024-09-16T10:29:16.541703+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:16.541703+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28909592,"num_objects":7059,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9662,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1064},{"shards":"2","objects":2755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13489","reported_seq":13548,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.542787+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:16.542787+0000","last_peered":"2024-09-16T10:29:16.542787+0000","last_clean":"2024-09-16T10:29:16.542787+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:16.542787+0000","last_undegraded":"2024-09-16T10:29:16.542787+0000","last_fullsized":"2024-09-16T10:29:16.542787+0000","mapping_epoch":34,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28561408,"num_objects":6973,"num_object_clones":0,"num_object_copies":13946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13489,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13476","reported_seq":13636,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.685702+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:16.685702+0000","last_peered":"2024-09-16T10:29:16.685702+0000","last_clean":"2024-09-16T10:29:16.685702+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:16.685702+0000","last_undegraded":"2024-09-16T10:29:16.685702+0000","last_fullsized":"2024-09-16T10:29:16.685702+0000","mapping_epoch":33,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28790784,"num_objects":7029,"num_object_clones":0,"num_object_copies":14058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13354","reported_seq":13634,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.829272+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:16.829272+0000","last_peered":"2024-09-16T10:29:16.829272+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:16.829272+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28741632,"num_objects":7017,"num_object_clones":0,"num_object_copies":14034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2857},{"shards":"0,2","objects":1051}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13836","reported_seq":14198,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.710140+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:16.710140+0000","last_peered":"2024-09-16T10:29:16.710140+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:16.710140+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29016064,"num_objects":7084,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10078,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2774},{"shards":"0,2","objects":1030}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13387","reported_seq":13565,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.802981+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:16.802981+0000","last_peered":"2024-09-16T10:29:16.802981+0000","last_clean":"2024-09-16T10:29:16.802981+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:16.802981+0000","last_undegraded":"2024-09-16T10:29:16.802981+0000","last_fullsized":"2024-09-16T10:29:16.802981+0000","mapping_epoch":33,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28684288,"num_objects":7003,"num_object_clones":0,"num_object_copies":14006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9190,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13429","reported_seq":21935,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:15.406903+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:15.406903+0000","last_peered":"2024-09-16T10:29:15.406903+0000","last_clean":"2024-09-16T10:29:15.406903+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:15.406903+0000","last_undegraded":"2024-09-16T10:29:15.406903+0000","last_fullsized":"2024-09-16T10:29:15.406903+0000","mapping_epoch":43,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28804038,"num_objects":7034,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13427,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'13914","reported_seq":14138,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.858878+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:16.858878+0000","last_peered":"2024-09-16T10:29:16.858878+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:16.858878+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29189062,"num_objects":7128,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13912,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2765},{"shards":"0,2","objects":1044}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13451","reported_seq":13610,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.492411+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:16.492411+0000","last_peered":"2024-09-16T10:29:16.492411+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:16.492411+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28541894,"num_objects":6970,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13449,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1038},{"shards":"2","objects":2718}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13500","reported_seq":14467,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.552268+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:16.552268+0000","last_peered":"2024-09-16T10:29:16.552268+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:16.552268+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28984262,"num_objects":7078,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":892,"num_objects_unfound":0,"num_objects_dirty":7078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13498,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":440,"num_bytes_recovered":1675264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":892},{"shards":"2","objects":3322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13172","reported_seq":13392,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.623066+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:16.623066+0000","last_peered":"2024-09-16T10:29:16.623066+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:16.623066+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28107235,"num_objects":6863,"num_object_clones":0,"num_object_copies":13726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13171,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2773},{"shards":"0,2","objects":1005}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13790","reported_seq":14036,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.817900+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:16.817900+0000","last_peered":"2024-09-16T10:29:16.817900+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:16.817900+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29181353,"num_objects":7127,"num_object_clones":0,"num_object_copies":14254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13787,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2845},{"shards":"0,2","objects":1008}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13281","reported_seq":13476,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.870309+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:16.870309+0000","last_peered":"2024-09-16T10:29:16.870309+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:16.870309+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28095430,"num_objects":6861,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13279,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2750},{"shards":"0,2","objects":958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13816","reported_seq":14026,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.861000+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:16.861000+0000","last_peered":"2024-09-16T10:29:16.861000+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:16.861000+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29001612,"num_objects":7084,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13813,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2746},{"shards":"0,2","objects":1033}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":460206606,"num_objects":112372,"num_object_clones":0,"num_object_copies":224744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41092,"num_objects_misplaced":892,"num_objects_unfound":0,"num_objects_dirty":112372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193933,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9362,"num_bytes_recovered":36041187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089208320,"data_stored":1089124303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160868,"ondisk_log_size":160868,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018961,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":646972,"kb_used_data":362672,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93724868,"statfs":{"total":96636764160,"available":95974264832,"internally_reserved":0,"allocated":371376128,"data_stored":371107977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,0,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.956},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.58299999999999996}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.372}]}]},{"osd":1,"up_from":40,"seq":171798691906,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581880,"kb_used_data":297584,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93789960,"statfs":{"total":96636764160,"available":96040919040,"internally_reserved":0,"allocated":304726016,"data_stored":304392831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":291106474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.3290000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.2879999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.46400000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215189,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776488,"kb_used_data":406100,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93595352,"statfs":{"total":96636764160,"available":95841640448,"internally_reserved":0,"allocated":415846400,"data_stored":415585935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,8,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":2.9159999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.16800000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.093}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":414507008,"data_stored":414503395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":304173056,"data_stored":304100337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370528256,"data_stored":370520571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:20.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:20.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'13605", 'reported_seq': 13856, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:16.541703+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:16.541703+0000', 'last_peered': '2024-09-16T10:29:16.541703+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:16.541703+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'3600", 'ondisk_log_start': "35'3600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28909592, 'num_objects': 7059, 'num_object_clones': 0, 'num_object_copies': 14118, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7059, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9662, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1064}, {'shards': '2', 'objects': 2755}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:20.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:20.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'13354", 'reported_seq': 13634, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:16.829272+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:16.829272+0000', 'last_peered': '2024-09-16T10:29:16.829272+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:16.829272+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'3300", 'ondisk_log_start': "35'3300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28741632, 'num_objects': 7017, 'num_object_clones': 0, 'num_object_copies': 14034, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4162, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7017, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9979, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 57, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2857}, {'shards': '0,2', 'objects': 1051}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:20.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:20.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'13836", 'reported_seq': 14198, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:16.710140+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:16.710140+0000', 'last_peered': '2024-09-16T10:29:16.710140+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:16.710140+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'3800", 'ondisk_log_start': "35'3800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29016064, 'num_objects': 7084, 'num_object_clones': 0, 'num_object_copies': 14168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4312, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10078, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 93, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2774}, {'shards': '0,2', 'objects': 1030}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:20.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:20.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'13914", 'reported_seq': 14138, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:16.858878+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:16.858878+0000', 'last_peered': '2024-09-16T10:29:16.858878+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:16.858878+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29189062, 'num_objects': 7128, 'num_object_clones': 0, 'num_object_copies': 14256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4365, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13912, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2765}, {'shards': '0,2', 'objects': 1044}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:20.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:20.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'13451", 'reported_seq': 13610, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:16.492411+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:16.492411+0000', 'last_peered': '2024-09-16T10:29:16.492411+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:16.492411+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'3400", 'ondisk_log_start': "35'3400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28541894, 'num_objects': 6970, 'num_object_clones': 0, 'num_object_copies': 13940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4254, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13449, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 27, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 1038}, {'shards': '2', 'objects': 2718}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:20.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:20.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'13500", 'reported_seq': 14467, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:16.552268+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:29:16.552268+0000', 'last_peered': '2024-09-16T10:29:16.552268+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:16.552268+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'3400", 'ondisk_log_start': "35'3400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28984262, 'num_objects': 7078, 'num_object_clones': 0, 'num_object_copies': 14156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2864, 'num_objects_misplaced': 892, 'num_objects_unfound': 0, 'num_objects_dirty': 7078, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13498, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 440, 'num_bytes_recovered': 1675264, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 892}, {'shards': '2', 'objects': 3322}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:20.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:20.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'13172", 'reported_seq': 13392, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:16.623066+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:16.623066+0000', 'last_peered': '2024-09-16T10:29:16.623066+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:16.623066+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'3100", 'ondisk_log_start': "35'3100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28107235, 'num_objects': 6863, 'num_object_clones': 0, 'num_object_copies': 13726, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4091, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13171, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 70, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2773}, {'shards': '0,2', 'objects': 1005}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:20.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:20.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'13790", 'reported_seq': 14036, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:16.817900+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:16.817900+0000', 'last_peered': '2024-09-16T10:29:16.817900+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:16.817900+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'3700", 'ondisk_log_start': "35'3700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29181353, 'num_objects': 7127, 'num_object_clones': 0, 'num_object_copies': 14254, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13787, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 58, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2845}, {'shards': '0,2', 'objects': 1008}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:20.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:20.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'13281", 'reported_seq': 13476, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:16.870309+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:16.870309+0000', 'last_peered': '2024-09-16T10:29:16.870309+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:16.870309+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'3200", 'ondisk_log_start': "35'3200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28095430, 'num_objects': 6861, 'num_object_clones': 0, 'num_object_copies': 13722, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4113, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6861, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13279, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 58, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2750}, {'shards': '0,2', 'objects': 958}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:20.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:20.644 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'13816", 'reported_seq': 14026, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:16.861000+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:16.861000+0000', 'last_peered': '2024-09-16T10:29:16.861000+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:16.861000+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'3800", 'ondisk_log_start': "35'3800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29001612, 'num_objects': 7084, 'num_object_clones': 0, 'num_object_copies': 14168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4341, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13813, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2746}, {'shards': '0,2', 'objects': 1033}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:20.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:21.314 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:21.315 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41092/224502 objects degraded (18.304%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":450,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":112251,"data_bytes":459702817,"bytes_used":2075054080,"bytes_avail":287835238400,"bytes_total":289910292480,"degraded_objects":41092,"degraded_total":224502,"degraded_ratio":0.1830362313030619,"misplaced_objects":892,"misplaced_total":224502,"misplaced_ratio":0.0039732385457590582,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":85296,"recovering_keys_per_sec":0,"num_objects_recovered":285,"num_bytes_recovered":1024000,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":365},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:29:21.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:21.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:21.880 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:21.881 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:21.904 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-16T10:29:20.684980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459702817,"num_objects":112251,"num_object_clones":0,"num_object_copies":224502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41092,"num_objects_misplaced":892,"num_objects_unfound":0,"num_objects_dirty":112251,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194058,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9363,"num_bytes_recovered":36041206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160894,"ondisk_log_size":160894,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2026420,"kb_used_data":1065292,"kb_used_omap":12,"kb_used_meta":961075,"kb_avail":281089100,"statfs":{"total":289910292480,"available":287835238400,"internally_reserved":0,"allocated":1090859008,"data_stored":1089997207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":984140802},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,7,13,3,0,4,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17952768,"num_objects":-4383,"num_object_clones":0,"num_object_copies":-8766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-35,"num_objects_misplaced":-250,"num_objects_unfound":0,"num_objects_dirty":-4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4383,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":285,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005195"},"pg_stats":[{"pgid":"2.f","version":"46'13439","reported_seq":21903,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.789188+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:16.789188+0000","last_peered":"2024-09-16T10:29:16.789188+0000","last_clean":"2024-09-16T10:29:16.789188+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:16.789188+0000","last_undegraded":"2024-09-16T10:29:16.789188+0000","last_fullsized":"2024-09-16T10:29:16.789188+0000","mapping_epoch":45,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28790784,"num_objects":7029,"num_object_clones":0,"num_object_copies":14058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13439,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13429","reported_seq":13588,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.535130+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:16.535130+0000","last_peered":"2024-09-16T10:29:16.535130+0000","last_clean":"2024-09-16T10:29:16.535130+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:16.535130+0000","last_undegraded":"2024-09-16T10:29:16.535130+0000","last_fullsized":"2024-09-16T10:29:16.535130+0000","mapping_epoch":34,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28807168,"num_objects":7033,"num_object_clones":0,"num_object_copies":14066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13605","reported_seq":13856,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.541703+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:16.541703+0000","last_peered":"2024-09-16T10:29:16.541703+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:16.541703+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28909592,"num_objects":7059,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9662,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1064},{"shards":"2","objects":2755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13489","reported_seq":13548,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.542787+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:16.542787+0000","last_peered":"2024-09-16T10:29:16.542787+0000","last_clean":"2024-09-16T10:29:16.542787+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:16.542787+0000","last_undegraded":"2024-09-16T10:29:16.542787+0000","last_fullsized":"2024-09-16T10:29:16.542787+0000","mapping_epoch":34,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28561408,"num_objects":6973,"num_object_clones":0,"num_object_copies":13946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13489,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13476","reported_seq":13636,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.685702+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:16.685702+0000","last_peered":"2024-09-16T10:29:16.685702+0000","last_clean":"2024-09-16T10:29:16.685702+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:16.685702+0000","last_undegraded":"2024-09-16T10:29:16.685702+0000","last_fullsized":"2024-09-16T10:29:16.685702+0000","mapping_epoch":33,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28790784,"num_objects":7029,"num_object_clones":0,"num_object_copies":14058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10118,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13354","reported_seq":13634,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.829272+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:16.829272+0000","last_peered":"2024-09-16T10:29:16.829272+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:16.829272+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28741632,"num_objects":7017,"num_object_clones":0,"num_object_copies":14034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2857},{"shards":"0,2","objects":1051}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13836","reported_seq":14198,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.710140+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:16.710140+0000","last_peered":"2024-09-16T10:29:16.710140+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:16.710140+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29016064,"num_objects":7084,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10078,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2774},{"shards":"0,2","objects":1030}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13387","reported_seq":13565,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:16.802981+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:16.802981+0000","last_peered":"2024-09-16T10:29:16.802981+0000","last_clean":"2024-09-16T10:29:16.802981+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:16.802981+0000","last_undegraded":"2024-09-16T10:29:16.802981+0000","last_fullsized":"2024-09-16T10:29:16.802981+0000","mapping_epoch":33,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28684288,"num_objects":7003,"num_object_clones":0,"num_object_copies":14006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9190,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13552","reported_seq":22058,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:20.415412+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:20.415412+0000","last_peered":"2024-09-16T10:29:20.415412+0000","last_clean":"2024-09-16T10:29:20.415412+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:20.415412+0000","last_undegraded":"2024-09-16T10:29:20.415412+0000","last_fullsized":"2024-09-16T10:29:20.415412+0000","mapping_epoch":43,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28300230,"num_objects":6911,"num_object_clones":0,"num_object_copies":13822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13550,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'13914","reported_seq":14138,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.858878+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:16.858878+0000","last_peered":"2024-09-16T10:29:16.858878+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:16.858878+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29189062,"num_objects":7128,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13912,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2765},{"shards":"0,2","objects":1044}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13451","reported_seq":13610,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.492411+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:16.492411+0000","last_peered":"2024-09-16T10:29:16.492411+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:16.492411+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28541894,"num_objects":6970,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4254,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13449,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":1038},{"shards":"2","objects":2718}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13500","reported_seq":14467,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.552268+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:16.552268+0000","last_peered":"2024-09-16T10:29:16.552268+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:16.552268+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28984262,"num_objects":7078,"num_object_clones":0,"num_object_copies":14156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":892,"num_objects_unfound":0,"num_objects_dirty":7078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13498,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":440,"num_bytes_recovered":1675264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":892},{"shards":"2","objects":3322}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13172","reported_seq":13392,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.623066+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:16.623066+0000","last_peered":"2024-09-16T10:29:16.623066+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:16.623066+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3100","ondisk_log_start":"35'3100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28107235,"num_objects":6863,"num_object_clones":0,"num_object_copies":13726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4091,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13171,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2773},{"shards":"0,2","objects":1005}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13790","reported_seq":14036,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.817900+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:16.817900+0000","last_peered":"2024-09-16T10:29:16.817900+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:16.817900+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29181353,"num_objects":7127,"num_object_clones":0,"num_object_copies":14254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13787,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2845},{"shards":"0,2","objects":1008}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13281","reported_seq":13476,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.870309+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:16.870309+0000","last_peered":"2024-09-16T10:29:16.870309+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:16.870309+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3200","ondisk_log_start":"35'3200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28095430,"num_objects":6861,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13279,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2750},{"shards":"0,2","objects":958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13816","reported_seq":14026,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:16.861000+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:16.861000+0000","last_peered":"2024-09-16T10:29:16.861000+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:16.861000+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29001612,"num_objects":7084,"num_object_clones":0,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13813,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2746},{"shards":"0,2","objects":1033}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":459702798,"num_objects":112249,"num_object_clones":0,"num_object_copies":224498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41092,"num_objects_misplaced":892,"num_objects_unfound":0,"num_objects_dirty":112249,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":194056,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9362,"num_bytes_recovered":36041187,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1088155648,"data_stored":1088071631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160891,"ondisk_log_size":160891,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018961,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":646972,"kb_used_data":362672,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93724868,"statfs":{"total":96636764160,"available":95974264832,"internally_reserved":0,"allocated":371376128,"data_stored":371107977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":291106480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,0,0,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.956},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.58299999999999996}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.372}]}]},{"osd":1,"up_from":40,"seq":171798691907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602960,"kb_used_data":296520,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93768880,"statfs":{"total":96636764160,"available":96019333120,"internally_reserved":0,"allocated":303636480,"data_stored":303303295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":313781930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.9860000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.2839999999999998},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.2679999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215189,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776488,"kb_used_data":406100,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93595352,"statfs":{"total":96636764160,"available":95841640448,"internally_reserved":0,"allocated":415846400,"data_stored":415585935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,4,8,0,0,0,0,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":2.9159999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.16800000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.093}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":414507008,"data_stored":414503395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":303120384,"data_stored":303047665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":370528256,"data_stored":370520571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:24.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:25.420 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:25.420 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:25.445 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-16T10:29:24.686314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451551777,"num_objects":110261,"num_object_clones":0,"num_object_copies":220522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41071,"num_objects_misplaced":756,"num_objects_unfound":0,"num_objects_dirty":110261,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196048,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9520,"num_bytes_recovered":36598262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160684,"ondisk_log_size":160684,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2028184,"kb_used_data":1054776,"kb_used_omap":12,"kb_used_meta":973363,"kb_avail":281087336,"statfs":{"total":289910292480,"available":287833432064,"internally_reserved":0,"allocated":1080090624,"data_stored":1079228823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":996723714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,8,9,8,0,0,0,0,0,4,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17420288,"num_objects":-4253,"num_object_clones":0,"num_object_copies":-8506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-37,"num_objects_misplaced":-253,"num_objects_unfound":0,"num_objects_dirty":-4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4253,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1036288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004877"},"pg_stats":[{"pgid":"2.f","version":"46'13578","reported_seq":22043,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.774026+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:21.774026+0000","last_peered":"2024-09-16T10:29:21.774026+0000","last_clean":"2024-09-16T10:29:21.774026+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:21.774026+0000","last_undegraded":"2024-09-16T10:29:21.774026+0000","last_fullsized":"2024-09-16T10:29:21.774026+0000","mapping_epoch":45,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28221440,"num_objects":6890,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13578,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13575","reported_seq":13735,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.444229+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:21.444229+0000","last_peered":"2024-09-16T10:29:21.444229+0000","last_clean":"2024-09-16T10:29:21.444229+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:21.444229+0000","last_undegraded":"2024-09-16T10:29:21.444229+0000","last_fullsized":"2024-09-16T10:29:21.444229+0000","mapping_epoch":34,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28209152,"num_objects":6887,"num_object_clones":0,"num_object_copies":13774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9788,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13758","reported_seq":14009,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.543937+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:21.543937+0000","last_peered":"2024-09-16T10:29:21.543937+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:21.543937+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28282904,"num_objects":6906,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9815,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":911},{"shards":"2","objects":2602}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13605","reported_seq":13663,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.567644+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:21.567644+0000","last_peered":"2024-09-16T10:29:21.567644+0000","last_clean":"2024-09-16T10:29:21.567644+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:21.567644+0000","last_undegraded":"2024-09-16T10:29:21.567644+0000","last_fullsized":"2024-09-16T10:29:21.567644+0000","mapping_epoch":34,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28086272,"num_objects":6857,"num_object_clones":0,"num_object_copies":13714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13605,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13617","reported_seq":13777,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.874599+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:21.874599+0000","last_peered":"2024-09-16T10:29:21.874599+0000","last_clean":"2024-09-16T10:29:21.874599+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:21.874599+0000","last_undegraded":"2024-09-16T10:29:21.874599+0000","last_fullsized":"2024-09-16T10:29:21.874599+0000","mapping_epoch":33,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28213248,"num_objects":6888,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10259,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13494","reported_seq":13781,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.853354+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:21.853354+0000","last_peered":"2024-09-16T10:29:21.853354+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:21.853354+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28168192,"num_objects":6877,"num_object_clones":0,"num_object_copies":13754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10119,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2720},{"shards":"0,2","objects":911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13971","reported_seq":14336,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.888141+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:21.888141+0000","last_peered":"2024-09-16T10:29:21.888141+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:21.888141+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28463104,"num_objects":6949,"num_object_clones":0,"num_object_copies":13898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2640},{"shards":"0,2","objects":895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13522","reported_seq":13700,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.835011+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:21.835011+0000","last_peered":"2024-09-16T10:29:21.835011+0000","last_clean":"2024-09-16T10:29:21.835011+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:21.835011+0000","last_undegraded":"2024-09-16T10:29:21.835011+0000","last_fullsized":"2024-09-16T10:29:21.835011+0000","mapping_epoch":33,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28131328,"num_objects":6868,"num_object_clones":0,"num_object_copies":13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13552","reported_seq":22058,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:20.415412+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:20.415412+0000","last_peered":"2024-09-16T10:29:20.415412+0000","last_clean":"2024-09-16T10:29:20.415412+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:20.415412+0000","last_undegraded":"2024-09-16T10:29:20.415412+0000","last_fullsized":"2024-09-16T10:29:20.415412+0000","mapping_epoch":43,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28300230,"num_objects":6911,"num_object_clones":0,"num_object_copies":13822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13550,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14029","reported_seq":14259,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.784383+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:21.784383+0000","last_peered":"2024-09-16T10:29:21.784383+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:21.784383+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28718022,"num_objects":7013,"num_object_clones":0,"num_object_copies":14026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4362,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14027,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2653},{"shards":"0,2","objects":929}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13576","reported_seq":13742,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.554403+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:21.554403+0000","last_peered":"2024-09-16T10:29:21.554403+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:21.554403+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28029894,"num_objects":6845,"num_object_clones":0,"num_object_copies":13690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13574,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":913},{"shards":"2","objects":2597}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13603","reported_seq":14841,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.568177+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:21.568177+0000","last_peered":"2024-09-16T10:29:21.568177+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:21.568177+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28562374,"num_objects":6975,"num_object_clones":0,"num_object_copies":13950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":756,"num_objects_unfound":0,"num_objects_dirty":6975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13601,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":2232320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":756},{"shards":"2","objects":3355}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13301","reported_seq":13527,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.877515+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:21.877515+0000","last_peered":"2024-09-16T10:29:21.877515+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:21.877515+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27578851,"num_objects":6734,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13300,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2647},{"shards":"0,2","objects":876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13919","reported_seq":14169,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.799109+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:21.799109+0000","last_peered":"2024-09-16T10:29:21.799109+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:21.799109+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28652969,"num_objects":6998,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13916,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2718},{"shards":"0,2","objects":879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13425","reported_seq":13628,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.821844+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:21.821844+0000","last_peered":"2024-09-16T10:29:21.821844+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:21.821844+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27505606,"num_objects":6717,"num_object_clones":0,"num_object_copies":13434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13423,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2611},{"shards":"0,2","objects":814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13956","reported_seq":14166,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.825301+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:21.825301+0000","last_peered":"2024-09-16T10:29:21.825301+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:21.825301+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28428172,"num_objects":6944,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13953,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2606},{"shards":"0,2","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":451551758,"num_objects":110259,"num_object_clones":0,"num_object_copies":220518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41071,"num_objects_misplaced":756,"num_objects_unfound":0,"num_objects_dirty":110259,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196046,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9519,"num_bytes_recovered":36598243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077329920,"data_stored":1077245903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160681,"ondisk_log_size":160681,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018962,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":646916,"kb_used_data":358524,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93724924,"statfs":{"total":96636764160,"available":95974322176,"internally_reserved":0,"allocated":367128576,"data_stored":366860425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,0,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.4490000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.245},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.26500000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602960,"kb_used_data":296520,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93768880,"statfs":{"total":96636764160,"available":96019333120,"internally_reserved":0,"allocated":303636480,"data_stored":303303295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":313781930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.9860000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.2839999999999998},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.2679999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215190,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":778308,"kb_used_data":399732,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93593532,"statfs":{"total":96636764160,"available":95839776768,"internally_reserved":0,"allocated":409325568,"data_stored":409065103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":387641000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,0,5,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":3.6880000000000002},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.72599999999999998},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.0910000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":407941120,"data_stored":407937507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":303120384,"data_stored":303047665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366268416,"data_stored":366260731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:25.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:25.975 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:25.976 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:26.005 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-16T10:29:24.686314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451551777,"num_objects":110261,"num_object_clones":0,"num_object_copies":220522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41071,"num_objects_misplaced":756,"num_objects_unfound":0,"num_objects_dirty":110261,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196048,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9520,"num_bytes_recovered":36598262,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160684,"ondisk_log_size":160684,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2028184,"kb_used_data":1054776,"kb_used_omap":12,"kb_used_meta":973363,"kb_avail":281087336,"statfs":{"total":289910292480,"available":287833432064,"internally_reserved":0,"allocated":1080090624,"data_stored":1079228823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":996723714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,8,9,8,0,0,0,0,0,4,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17420288,"num_objects":-4253,"num_object_clones":0,"num_object_copies":-8506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-37,"num_objects_misplaced":-253,"num_objects_unfound":0,"num_objects_dirty":-4253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4253,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":1036288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004877"},"pg_stats":[{"pgid":"2.f","version":"46'13578","reported_seq":22043,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.774026+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:21.774026+0000","last_peered":"2024-09-16T10:29:21.774026+0000","last_clean":"2024-09-16T10:29:21.774026+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:21.774026+0000","last_undegraded":"2024-09-16T10:29:21.774026+0000","last_fullsized":"2024-09-16T10:29:21.774026+0000","mapping_epoch":45,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28221440,"num_objects":6890,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13578,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13575","reported_seq":13735,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.444229+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:21.444229+0000","last_peered":"2024-09-16T10:29:21.444229+0000","last_clean":"2024-09-16T10:29:21.444229+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:21.444229+0000","last_undegraded":"2024-09-16T10:29:21.444229+0000","last_fullsized":"2024-09-16T10:29:21.444229+0000","mapping_epoch":34,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28209152,"num_objects":6887,"num_object_clones":0,"num_object_copies":13774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9788,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13758","reported_seq":14009,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.543937+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:21.543937+0000","last_peered":"2024-09-16T10:29:21.543937+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:21.543937+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28282904,"num_objects":6906,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6906,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9815,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":911},{"shards":"2","objects":2602}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13605","reported_seq":13663,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.567644+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:21.567644+0000","last_peered":"2024-09-16T10:29:21.567644+0000","last_clean":"2024-09-16T10:29:21.567644+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:21.567644+0000","last_undegraded":"2024-09-16T10:29:21.567644+0000","last_fullsized":"2024-09-16T10:29:21.567644+0000","mapping_epoch":34,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28086272,"num_objects":6857,"num_object_clones":0,"num_object_copies":13714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13605,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13617","reported_seq":13777,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.874599+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:21.874599+0000","last_peered":"2024-09-16T10:29:21.874599+0000","last_clean":"2024-09-16T10:29:21.874599+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:21.874599+0000","last_undegraded":"2024-09-16T10:29:21.874599+0000","last_fullsized":"2024-09-16T10:29:21.874599+0000","mapping_epoch":33,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28213248,"num_objects":6888,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10259,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13494","reported_seq":13781,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.853354+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:21.853354+0000","last_peered":"2024-09-16T10:29:21.853354+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:21.853354+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28168192,"num_objects":6877,"num_object_clones":0,"num_object_copies":13754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10119,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2720},{"shards":"0,2","objects":911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13971","reported_seq":14336,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.888141+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:21.888141+0000","last_peered":"2024-09-16T10:29:21.888141+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:21.888141+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28463104,"num_objects":6949,"num_object_clones":0,"num_object_copies":13898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2640},{"shards":"0,2","objects":895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13522","reported_seq":13700,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.835011+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:21.835011+0000","last_peered":"2024-09-16T10:29:21.835011+0000","last_clean":"2024-09-16T10:29:21.835011+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:21.835011+0000","last_undegraded":"2024-09-16T10:29:21.835011+0000","last_fullsized":"2024-09-16T10:29:21.835011+0000","mapping_epoch":33,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28131328,"num_objects":6868,"num_object_clones":0,"num_object_copies":13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13552","reported_seq":22058,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:20.415412+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:20.415412+0000","last_peered":"2024-09-16T10:29:20.415412+0000","last_clean":"2024-09-16T10:29:20.415412+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:20.415412+0000","last_undegraded":"2024-09-16T10:29:20.415412+0000","last_fullsized":"2024-09-16T10:29:20.415412+0000","mapping_epoch":43,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28300230,"num_objects":6911,"num_object_clones":0,"num_object_copies":13822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13550,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14029","reported_seq":14259,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.784383+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:21.784383+0000","last_peered":"2024-09-16T10:29:21.784383+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:21.784383+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28718022,"num_objects":7013,"num_object_clones":0,"num_object_copies":14026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4362,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14027,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2653},{"shards":"0,2","objects":929}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13576","reported_seq":13742,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.554403+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:21.554403+0000","last_peered":"2024-09-16T10:29:21.554403+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:21.554403+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28029894,"num_objects":6845,"num_object_clones":0,"num_object_copies":13690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13574,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":913},{"shards":"2","objects":2597}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13603","reported_seq":14841,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.568177+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:21.568177+0000","last_peered":"2024-09-16T10:29:21.568177+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:21.568177+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28562374,"num_objects":6975,"num_object_clones":0,"num_object_copies":13950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":756,"num_objects_unfound":0,"num_objects_dirty":6975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13601,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":2232320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":756},{"shards":"2","objects":3355}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13301","reported_seq":13527,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.877515+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:21.877515+0000","last_peered":"2024-09-16T10:29:21.877515+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:21.877515+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27578851,"num_objects":6734,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13300,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2647},{"shards":"0,2","objects":876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13919","reported_seq":14169,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.799109+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:21.799109+0000","last_peered":"2024-09-16T10:29:21.799109+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:21.799109+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28652969,"num_objects":6998,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13916,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2718},{"shards":"0,2","objects":879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13425","reported_seq":13628,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.821844+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:21.821844+0000","last_peered":"2024-09-16T10:29:21.821844+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:21.821844+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27505606,"num_objects":6717,"num_object_clones":0,"num_object_copies":13434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13423,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2611},{"shards":"0,2","objects":814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13956","reported_seq":14166,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.825301+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:21.825301+0000","last_peered":"2024-09-16T10:29:21.825301+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:21.825301+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28428172,"num_objects":6944,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13953,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2606},{"shards":"0,2","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":451551758,"num_objects":110259,"num_object_clones":0,"num_object_copies":220518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41071,"num_objects_misplaced":756,"num_objects_unfound":0,"num_objects_dirty":110259,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196046,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9519,"num_bytes_recovered":36598243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077329920,"data_stored":1077245903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160681,"ondisk_log_size":160681,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018962,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":646916,"kb_used_data":358524,"kb_used_omap":4,"kb_used_meta":288379,"kb_avail":93724924,"statfs":{"total":96636764160,"available":95974322176,"internally_reserved":0,"allocated":367128576,"data_stored":366860425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":295300784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,0,0,0,0,0,0,0,4],"upper_bound":16384},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.4490000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.245},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.26500000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691907,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602960,"kb_used_data":296520,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93768880,"statfs":{"total":96636764160,"available":96019333120,"internally_reserved":0,"allocated":303636480,"data_stored":303303295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":313781930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.9860000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.2839999999999998},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.2679999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215190,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":778308,"kb_used_data":399732,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93593532,"statfs":{"total":96636764160,"available":95839776768,"internally_reserved":0,"allocated":409325568,"data_stored":409065103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":387641000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,0,5,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":3.6880000000000002},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.72599999999999998},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.0910000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":407941120,"data_stored":407937507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":303120384,"data_stored":303047665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":366268416,"data_stored":366260731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:26.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:26.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'13758", 'reported_seq': 14009, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:21.543937+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:21.543937+0000', 'last_peered': '2024-09-16T10:29:21.543937+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:21.543937+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'3700", 'ondisk_log_start': "35'3700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28282904, 'num_objects': 6906, 'num_object_clones': 0, 'num_object_copies': 13812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4305, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6906, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9815, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 911}, {'shards': '2', 'objects': 2602}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:26.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:26.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'13494", 'reported_seq': 13781, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:21.853354+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:21.853354+0000', 'last_peered': '2024-09-16T10:29:21.853354+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:21.853354+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'3400", 'ondisk_log_start': "35'3400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28168192, 'num_objects': 6877, 'num_object_clones': 0, 'num_object_copies': 13754, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4159, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10119, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 60, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2720}, {'shards': '0,2', 'objects': 911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:26.007 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:26.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'13971", 'reported_seq': 14336, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:21.888141+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:21.888141+0000', 'last_peered': '2024-09-16T10:29:21.888141+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:21.888141+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28463104, 'num_objects': 6949, 'num_object_clones': 0, 'num_object_copies': 13898, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4311, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6949, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10213, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 94, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2640}, {'shards': '0,2', 'objects': 895}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:26.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:26.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'14029", 'reported_seq': 14259, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:21.784383+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:21.784383+0000', 'last_peered': '2024-09-16T10:29:21.784383+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:21.784383+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'4000", 'ondisk_log_start': "35'4000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28718022, 'num_objects': 7013, 'num_object_clones': 0, 'num_object_copies': 14026, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4362, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14027, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2653}, {'shards': '0,2', 'objects': 929}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:26.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:26.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'13576", 'reported_seq': 13742, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:21.554403+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:21.554403+0000', 'last_peered': '2024-09-16T10:29:21.554403+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:21.554403+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'3500", 'ondisk_log_start': "35'3500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28029894, 'num_objects': 6845, 'num_object_clones': 0, 'num_object_copies': 13690, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13574, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 913}, {'shards': '2', 'objects': 2597}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:26.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:26.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'13603", 'reported_seq': 14841, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:21.568177+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:29:21.568177+0000', 'last_peered': '2024-09-16T10:29:21.568177+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:21.568177+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'3600", 'ondisk_log_start': "35'3600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28562374, 'num_objects': 6975, 'num_object_clones': 0, 'num_object_copies': 13950, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2864, 'num_objects_misplaced': 756, 'num_objects_unfound': 0, 'num_objects_dirty': 6975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13601, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 576, 'num_bytes_recovered': 2232320, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 756}, {'shards': '2', 'objects': 3355}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:26.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:26.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'13301", 'reported_seq': 13527, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:21.877515+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:21.877515+0000', 'last_peered': '2024-09-16T10:29:21.877515+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:21.877515+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'3300", 'ondisk_log_start': "35'3300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27578851, 'num_objects': 6734, 'num_object_clones': 0, 'num_object_copies': 13468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4088, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6734, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13300, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2647}, {'shards': '0,2', 'objects': 876}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:26.008 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:26.009 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'13919", 'reported_seq': 14169, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:21.799109+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:21.799109+0000', 'last_peered': '2024-09-16T10:29:21.799109+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:21.799109+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28652969, 'num_objects': 6998, 'num_object_clones': 0, 'num_object_copies': 13996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6998, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13916, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 60, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2718}, {'shards': '0,2', 'objects': 879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:26.009 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:26.009 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'13425", 'reported_seq': 13628, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:21.821844+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:21.821844+0000', 'last_peered': '2024-09-16T10:29:21.821844+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:21.821844+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'3400", 'ondisk_log_start': "35'3400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27505606, 'num_objects': 6717, 'num_object_clones': 0, 'num_object_copies': 13434, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4108, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13423, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2611}, {'shards': '0,2', 'objects': 814}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:26.009 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:26.009 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'13956", 'reported_seq': 14166, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:21.825301+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:21.825301+0000', 'last_peered': '2024-09-16T10:29:21.825301+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:21.825301+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28428172, 'num_objects': 6944, 'num_object_clones': 0, 'num_object_copies': 13888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4341, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13953, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2606}, {'shards': '0,2', 'objects': 893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:26.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:26.696 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:26.697 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41071/220522 objects degraded (18.624%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":455,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":110261,"data_bytes":451551777,"bytes_used":2076860416,"bytes_avail":287833432064,"bytes_total":289910292480,"degraded_objects":41071,"degraded_total":220522,"degraded_ratio":0.1862444563354223,"misplaced_objects":756,"misplaced_total":220522,"misplaced_ratio":0.0034282293830094051,"recovering_objects_per_sec":24,"recovering_bytes_per_sec":86322,"recovering_keys_per_sec":0,"num_objects_recovered":290,"num_bytes_recovered":1036288,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":354},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:29:26.719 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:26.719 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:27.298 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:27.299 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:27.329 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2024-09-16T10:29:26.687216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":448397857,"num_objects":109491,"num_object_clones":0,"num_object_copies":218982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41070,"num_objects_misplaced":589,"num_objects_unfound":0,"num_objects_dirty":109491,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196818,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9688,"num_bytes_recovered":37282294,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160654,"ondisk_log_size":160654,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2035444,"kb_used_data":1049744,"kb_used_omap":12,"kb_used_meta":985651,"kb_avail":281080076,"statfs":{"total":289910292480,"available":287825997824,"internally_reserved":0,"allocated":1074937856,"data_stored":1074076055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1009306626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,5,5,3,6,0,0,0,0,0,5],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20574208,"num_objects":-5023,"num_object_clones":0,"num_object_copies":-10046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-38,"num_objects_misplaced":-420,"num_objects_unfound":0,"num_objects_dirty":-5023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5023,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":1720320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004848"},"pg_stats":[{"pgid":"2.f","version":"46'13578","reported_seq":22043,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.774026+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:21.774026+0000","last_peered":"2024-09-16T10:29:21.774026+0000","last_clean":"2024-09-16T10:29:21.774026+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:21.774026+0000","last_undegraded":"2024-09-16T10:29:21.774026+0000","last_fullsized":"2024-09-16T10:29:21.774026+0000","mapping_epoch":45,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28221440,"num_objects":6890,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13578,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13705","reported_seq":13865,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.530063+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:26.530063+0000","last_peered":"2024-09-16T10:29:26.530063+0000","last_clean":"2024-09-16T10:29:26.530063+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:26.530063+0000","last_undegraded":"2024-09-16T10:29:26.530063+0000","last_fullsized":"2024-09-16T10:29:26.530063+0000","mapping_epoch":34,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27676672,"num_objects":6757,"num_object_clones":0,"num_object_copies":13514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9918,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13885","reported_seq":14138,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.529477+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:26.529477+0000","last_peered":"2024-09-16T10:29:26.529477+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:26.529477+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27762712,"num_objects":6779,"num_object_clones":0,"num_object_copies":13558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9942,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":784},{"shards":"2","objects":2476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13718","reported_seq":13777,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.495180+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:26.495180+0000","last_peered":"2024-09-16T10:29:26.495180+0000","last_clean":"2024-09-16T10:29:26.495180+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:26.495180+0000","last_undegraded":"2024-09-16T10:29:26.495180+0000","last_fullsized":"2024-09-16T10:29:26.495180+0000","mapping_epoch":34,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27623424,"num_objects":6744,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13718,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13617","reported_seq":13777,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.874599+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:21.874599+0000","last_peered":"2024-09-16T10:29:21.874599+0000","last_clean":"2024-09-16T10:29:21.874599+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:21.874599+0000","last_undegraded":"2024-09-16T10:29:21.874599+0000","last_fullsized":"2024-09-16T10:29:21.874599+0000","mapping_epoch":33,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28213248,"num_objects":6888,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10259,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13494","reported_seq":13781,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.853354+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:21.853354+0000","last_peered":"2024-09-16T10:29:21.853354+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:21.853354+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28168192,"num_objects":6877,"num_object_clones":0,"num_object_copies":13754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10119,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2720},{"shards":"0,2","objects":911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'13971","reported_seq":14336,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.888141+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:21.888141+0000","last_peered":"2024-09-16T10:29:21.888141+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:21.888141+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28463104,"num_objects":6949,"num_object_clones":0,"num_object_copies":13898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10213,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2640},{"shards":"0,2","objects":895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13522","reported_seq":13700,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:21.835011+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:21.835011+0000","last_peered":"2024-09-16T10:29:21.835011+0000","last_clean":"2024-09-16T10:29:21.835011+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:21.835011+0000","last_undegraded":"2024-09-16T10:29:21.835011+0000","last_fullsized":"2024-09-16T10:29:21.835011+0000","mapping_epoch":33,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28131328,"num_objects":6868,"num_object_clones":0,"num_object_copies":13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13687","reported_seq":22193,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:25.462605+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:25.462605+0000","last_peered":"2024-09-16T10:29:25.462605+0000","last_clean":"2024-09-16T10:29:25.462605+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:25.462605+0000","last_undegraded":"2024-09-16T10:29:25.462605+0000","last_fullsized":"2024-09-16T10:29:25.462605+0000","mapping_epoch":43,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27747270,"num_objects":6776,"num_object_clones":0,"num_object_copies":13552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13685,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14029","reported_seq":14259,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.784383+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:21.784383+0000","last_peered":"2024-09-16T10:29:21.784383+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:21.784383+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28718022,"num_objects":7013,"num_object_clones":0,"num_object_copies":14026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4362,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14027,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2653},{"shards":"0,2","objects":929}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13714","reported_seq":13881,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.501656+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:26.501656+0000","last_peered":"2024-09-16T10:29:26.501656+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:26.501656+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27464646,"num_objects":6707,"num_object_clones":0,"num_object_copies":13414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13712,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":775},{"shards":"2","objects":2459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13730","reported_seq":15303,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.586838+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:26.586838+0000","last_peered":"2024-09-16T10:29:26.586838+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:26.586838+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28042182,"num_objects":6848,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":589,"num_objects_unfound":0,"num_objects_dirty":6848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13728,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":2916352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":589},{"shards":"2","objects":3395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13301","reported_seq":13527,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.877515+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:21.877515+0000","last_peered":"2024-09-16T10:29:21.877515+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:21.877515+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3300","ondisk_log_start":"35'3300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27578851,"num_objects":6734,"num_object_clones":0,"num_object_copies":13468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13300,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2647},{"shards":"0,2","objects":876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'13919","reported_seq":14169,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.799109+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:21.799109+0000","last_peered":"2024-09-16T10:29:21.799109+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:21.799109+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28652969,"num_objects":6998,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13916,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2718},{"shards":"0,2","objects":879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13425","reported_seq":13628,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.821844+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:21.821844+0000","last_peered":"2024-09-16T10:29:21.821844+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:21.821844+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27505606,"num_objects":6717,"num_object_clones":0,"num_object_copies":13434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13423,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2611},{"shards":"0,2","objects":814}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'13956","reported_seq":14166,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:21.825301+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:21.825301+0000","last_peered":"2024-09-16T10:29:21.825301+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:21.825301+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28428172,"num_objects":6944,"num_object_clones":0,"num_object_copies":13888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13953,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2606},{"shards":"0,2","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":448397838,"num_objects":109489,"num_object_clones":0,"num_object_copies":218978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41070,"num_objects_misplaced":589,"num_objects_unfound":0,"num_objects_dirty":109489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196816,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9687,"num_bytes_recovered":37282275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1072263168,"data_stored":1072179151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160651,"ondisk_log_size":160651,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018964,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":646900,"kb_used_data":354408,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93724940,"statfs":{"total":96636764160,"available":95974338560,"internally_reserved":0,"allocated":362913792,"data_stored":362645641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,1,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.194},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.28299999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.318},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.20399999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691908,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610236,"kb_used_data":295604,"kb_used_omap":4,"kb_used_meta":314619,"kb_avail":93761604,"statfs":{"total":96636764160,"available":96011882496,"internally_reserved":0,"allocated":302698496,"data_stored":302365311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":322170538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.2480000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.2170000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.32400000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215190,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":778308,"kb_used_data":399732,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93593532,"statfs":{"total":96636764160,"available":95839776768,"internally_reserved":0,"allocated":409325568,"data_stored":409065103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":387641000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,0,5,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":3.6880000000000002},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.72599999999999998},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.0910000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":407941120,"data_stored":407937507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":302219264,"data_stored":302146545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362102784,"data_stored":362095099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:30.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:30.872 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:30.873 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:30.900 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-16T10:29:30.688598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":442614305,"num_objects":108079,"num_object_clones":0,"num_object_copies":216158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41050,"num_objects_misplaced":589,"num_objects_unfound":0,"num_objects_dirty":108079,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198230,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9708,"num_bytes_recovered":37282294,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160666,"ondisk_log_size":160666,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2020236,"kb_used_data":1042736,"kb_used_omap":12,"kb_used_meta":977459,"kb_avail":281095284,"statfs":{"total":289910292480,"available":287841570816,"internally_reserved":0,"allocated":1067761664,"data_stored":1066899863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1000918018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,6,9,4,4,0,0,0,0,0,12],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17592320,"num_objects":-4295,"num_object_clones":0,"num_object_copies":-8590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-42,"num_objects_misplaced":-303,"num_objects_unfound":0,"num_objects_dirty":-4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":1241088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004447"},"pg_stats":[{"pgid":"2.f","version":"46'13700","reported_seq":22165,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.856398+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:26.856398+0000","last_peered":"2024-09-16T10:29:26.856398+0000","last_clean":"2024-09-16T10:29:26.856398+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:26.856398+0000","last_undegraded":"2024-09-16T10:29:26.856398+0000","last_fullsized":"2024-09-16T10:29:26.856398+0000","mapping_epoch":45,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":6768,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13700,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13705","reported_seq":13865,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.530063+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:26.530063+0000","last_peered":"2024-09-16T10:29:26.530063+0000","last_clean":"2024-09-16T10:29:26.530063+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:26.530063+0000","last_undegraded":"2024-09-16T10:29:26.530063+0000","last_fullsized":"2024-09-16T10:29:26.530063+0000","mapping_epoch":34,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27676672,"num_objects":6757,"num_object_clones":0,"num_object_copies":13514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9918,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13885","reported_seq":14138,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.529477+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:26.529477+0000","last_peered":"2024-09-16T10:29:26.529477+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:26.529477+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27762712,"num_objects":6779,"num_object_clones":0,"num_object_copies":13558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9942,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":784},{"shards":"2","objects":2476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13718","reported_seq":13777,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.495180+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:26.495180+0000","last_peered":"2024-09-16T10:29:26.495180+0000","last_clean":"2024-09-16T10:29:26.495180+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:26.495180+0000","last_undegraded":"2024-09-16T10:29:26.495180+0000","last_fullsized":"2024-09-16T10:29:26.495180+0000","mapping_epoch":34,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27623424,"num_objects":6744,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13718,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13740","reported_seq":13900,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.824085+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:26.824085+0000","last_peered":"2024-09-16T10:29:26.824085+0000","last_clean":"2024-09-16T10:29:26.824085+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:26.824085+0000","last_undegraded":"2024-09-16T10:29:26.824085+0000","last_fullsized":"2024-09-16T10:29:26.824085+0000","mapping_epoch":33,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27709440,"num_objects":6765,"num_object_clones":0,"num_object_copies":13530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10382,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13626","reported_seq":13919,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.880082+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:26.880082+0000","last_peered":"2024-09-16T10:29:26.880082+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:26.880082+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27627520,"num_objects":6745,"num_object_clones":0,"num_object_copies":13490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10251,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2591},{"shards":"0,2","objects":779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14104","reported_seq":14471,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.898832+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:26.898832+0000","last_peered":"2024-09-16T10:29:26.898832+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:26.898832+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'4100","ondisk_log_start":"35'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2508},{"shards":"0,2","objects":762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13651","reported_seq":13829,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.929406+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:26.929406+0000","last_peered":"2024-09-16T10:29:26.929406+0000","last_clean":"2024-09-16T10:29:26.929406+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:26.929406+0000","last_undegraded":"2024-09-16T10:29:26.929406+0000","last_fullsized":"2024-09-16T10:29:26.929406+0000","mapping_epoch":33,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27602944,"num_objects":6739,"num_object_clones":0,"num_object_copies":13478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9454,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13816","reported_seq":22322,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:30.448905+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:30.448905+0000","last_peered":"2024-09-16T10:29:30.448905+0000","last_clean":"2024-09-16T10:29:30.448905+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:30.448905+0000","last_undegraded":"2024-09-16T10:29:30.448905+0000","last_fullsized":"2024-09-16T10:29:30.448905+0000","mapping_epoch":43,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27218886,"num_objects":6647,"num_object_clones":0,"num_object_copies":13294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13814,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14172","reported_seq":14411,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.915687+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:26.915687+0000","last_peered":"2024-09-16T10:29:26.915687+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:26.915687+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'4100","ondisk_log_start":"35'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28132294,"num_objects":6870,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14170,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2515},{"shards":"0,2","objects":786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13714","reported_seq":13881,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.501656+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:26.501656+0000","last_peered":"2024-09-16T10:29:26.501656+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:26.501656+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27464646,"num_objects":6707,"num_object_clones":0,"num_object_copies":13414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13712,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":775},{"shards":"2","objects":2459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13730","reported_seq":15303,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.586838+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:26.586838+0000","last_peered":"2024-09-16T10:29:26.586838+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:26.586838+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28042182,"num_objects":6848,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":589,"num_objects_unfound":0,"num_objects_dirty":6848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13728,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":2916352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":589},{"shards":"2","objects":3395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13441","reported_seq":13667,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.823969+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:26.823969+0000","last_peered":"2024-09-16T10:29:26.823969+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:26.823969+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27005411,"num_objects":6594,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13440,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2507},{"shards":"0,2","objects":736}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14044","reported_seq":14298,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.906790+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:26.906790+0000","last_peered":"2024-09-16T10:29:26.906790+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:26.906790+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28140969,"num_objects":6873,"num_object_clones":0,"num_object_copies":13746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14041,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2595},{"shards":"0,2","objects":754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13530","reported_seq":13741,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.915629+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:26.915629+0000","last_peered":"2024-09-16T10:29:26.915629+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:26.915629+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27075526,"num_objects":6612,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13528,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2509},{"shards":"0,2","objects":709}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14087","reported_seq":14309,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.868902+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:26.868902+0000","last_peered":"2024-09-16T10:29:26.868902+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:26.868902+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27891596,"num_objects":6813,"num_object_clones":0,"num_object_copies":13626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14084,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2481},{"shards":"0,2","objects":762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":442614286,"num_objects":108077,"num_object_clones":0,"num_object_copies":216154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41050,"num_objects_misplaced":589,"num_objects_unfound":0,"num_objects_dirty":108077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198228,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9707,"num_bytes_recovered":37282275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064943616,"data_stored":1064859599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160663,"ondisk_log_size":160663,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018964,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":646900,"kb_used_data":354408,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93724940,"statfs":{"total":96636764160,"available":95974338560,"internally_reserved":0,"allocated":362913792,"data_stored":362645641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,1,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.194},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.28299999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.318},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.20399999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691909,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596936,"kb_used_data":294592,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93774904,"statfs":{"total":96636764160,"available":96025501696,"internally_reserved":0,"allocated":301662208,"data_stored":301329023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":309587626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.0550000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.31}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.149},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.0680000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215191,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776400,"kb_used_data":393736,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93595440,"statfs":{"total":96636764160,"available":95841730560,"internally_reserved":0,"allocated":403185664,"data_stored":402925199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":391835304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,1,0,0,0,0,0,0,11],"upper_bound":8192},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.217},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.436}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.5},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401657856,"data_stored":401654243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301182976,"data_stored":301110257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362102784,"data_stored":362095099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:30.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:31.455 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:31.457 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:31.478 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-16T10:29:30.688598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":442614305,"num_objects":108079,"num_object_clones":0,"num_object_copies":216158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41050,"num_objects_misplaced":589,"num_objects_unfound":0,"num_objects_dirty":108079,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198230,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9708,"num_bytes_recovered":37282294,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160666,"ondisk_log_size":160666,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2020236,"kb_used_data":1042736,"kb_used_omap":12,"kb_used_meta":977459,"kb_avail":281095284,"statfs":{"total":289910292480,"available":287841570816,"internally_reserved":0,"allocated":1067761664,"data_stored":1066899863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1000918018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,6,9,4,4,0,0,0,0,0,12],"upper_bound":8192},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17592320,"num_objects":-4295,"num_object_clones":0,"num_object_copies":-8590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-42,"num_objects_misplaced":-303,"num_objects_unfound":0,"num_objects_dirty":-4295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4295,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":345,"num_bytes_recovered":1241088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004447"},"pg_stats":[{"pgid":"2.f","version":"46'13700","reported_seq":22165,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.856398+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:26.856398+0000","last_peered":"2024-09-16T10:29:26.856398+0000","last_clean":"2024-09-16T10:29:26.856398+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:26.856398+0000","last_undegraded":"2024-09-16T10:29:26.856398+0000","last_fullsized":"2024-09-16T10:29:26.856398+0000","mapping_epoch":45,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":6768,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13700,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13705","reported_seq":13865,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.530063+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:26.530063+0000","last_peered":"2024-09-16T10:29:26.530063+0000","last_clean":"2024-09-16T10:29:26.530063+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:26.530063+0000","last_undegraded":"2024-09-16T10:29:26.530063+0000","last_fullsized":"2024-09-16T10:29:26.530063+0000","mapping_epoch":34,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27676672,"num_objects":6757,"num_object_clones":0,"num_object_copies":13514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9918,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'13885","reported_seq":14138,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.529477+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:26.529477+0000","last_peered":"2024-09-16T10:29:26.529477+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:26.529477+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27762712,"num_objects":6779,"num_object_clones":0,"num_object_copies":13558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9942,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":784},{"shards":"2","objects":2476}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13718","reported_seq":13777,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.495180+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:26.495180+0000","last_peered":"2024-09-16T10:29:26.495180+0000","last_clean":"2024-09-16T10:29:26.495180+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:26.495180+0000","last_undegraded":"2024-09-16T10:29:26.495180+0000","last_fullsized":"2024-09-16T10:29:26.495180+0000","mapping_epoch":34,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27623424,"num_objects":6744,"num_object_clones":0,"num_object_copies":13488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13718,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13740","reported_seq":13900,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.824085+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:26.824085+0000","last_peered":"2024-09-16T10:29:26.824085+0000","last_clean":"2024-09-16T10:29:26.824085+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:26.824085+0000","last_undegraded":"2024-09-16T10:29:26.824085+0000","last_fullsized":"2024-09-16T10:29:26.824085+0000","mapping_epoch":33,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27709440,"num_objects":6765,"num_object_clones":0,"num_object_copies":13530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10382,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13626","reported_seq":13919,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.880082+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:26.880082+0000","last_peered":"2024-09-16T10:29:26.880082+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:26.880082+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27627520,"num_objects":6745,"num_object_clones":0,"num_object_copies":13490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10251,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2591},{"shards":"0,2","objects":779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14104","reported_seq":14471,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.898832+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:26.898832+0000","last_peered":"2024-09-16T10:29:26.898832+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:26.898832+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"35'4100","ondisk_log_start":"35'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4310,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2508},{"shards":"0,2","objects":762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13651","reported_seq":13829,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:26.929406+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:26.929406+0000","last_peered":"2024-09-16T10:29:26.929406+0000","last_clean":"2024-09-16T10:29:26.929406+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:26.929406+0000","last_undegraded":"2024-09-16T10:29:26.929406+0000","last_fullsized":"2024-09-16T10:29:26.929406+0000","mapping_epoch":33,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27602944,"num_objects":6739,"num_object_clones":0,"num_object_copies":13478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9454,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13816","reported_seq":22322,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:30.448905+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:30.448905+0000","last_peered":"2024-09-16T10:29:30.448905+0000","last_clean":"2024-09-16T10:29:30.448905+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:30.448905+0000","last_undegraded":"2024-09-16T10:29:30.448905+0000","last_fullsized":"2024-09-16T10:29:30.448905+0000","mapping_epoch":43,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27218886,"num_objects":6647,"num_object_clones":0,"num_object_copies":13294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13814,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14172","reported_seq":14411,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.915687+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:26.915687+0000","last_peered":"2024-09-16T10:29:26.915687+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:26.915687+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"35'4100","ondisk_log_start":"35'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28132294,"num_objects":6870,"num_object_clones":0,"num_object_copies":13740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14170,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2515},{"shards":"0,2","objects":786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13714","reported_seq":13881,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.501656+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:26.501656+0000","last_peered":"2024-09-16T10:29:26.501656+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:26.501656+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27464646,"num_objects":6707,"num_object_clones":0,"num_object_copies":13414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13712,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":775},{"shards":"2","objects":2459}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13730","reported_seq":15303,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.586838+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:26.586838+0000","last_peered":"2024-09-16T10:29:26.586838+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:26.586838+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28042182,"num_objects":6848,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":589,"num_objects_unfound":0,"num_objects_dirty":6848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13728,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":2916352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":589},{"shards":"2","objects":3395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13441","reported_seq":13667,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.823969+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:26.823969+0000","last_peered":"2024-09-16T10:29:26.823969+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:26.823969+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3400","ondisk_log_start":"35'3400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27005411,"num_objects":6594,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13440,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2507},{"shards":"0,2","objects":736}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14044","reported_seq":14298,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.906790+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:26.906790+0000","last_peered":"2024-09-16T10:29:26.906790+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:26.906790+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28140969,"num_objects":6873,"num_object_clones":0,"num_object_copies":13746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14041,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2595},{"shards":"0,2","objects":754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13530","reported_seq":13741,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.915629+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:26.915629+0000","last_peered":"2024-09-16T10:29:26.915629+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:26.915629+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27075526,"num_objects":6612,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13528,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2509},{"shards":"0,2","objects":709}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14087","reported_seq":14309,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:26.868902+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:26.868902+0000","last_peered":"2024-09-16T10:29:26.868902+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:26.868902+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27891596,"num_objects":6813,"num_object_clones":0,"num_object_copies":13626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14084,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2481},{"shards":"0,2","objects":762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":442614286,"num_objects":108077,"num_object_clones":0,"num_object_copies":216154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41050,"num_objects_misplaced":589,"num_objects_unfound":0,"num_objects_dirty":108077,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":198228,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9707,"num_bytes_recovered":37282275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064943616,"data_stored":1064859599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160663,"ondisk_log_size":160663,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018964,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":646900,"kb_used_data":354408,"kb_used_omap":4,"kb_used_meta":292475,"kb_avail":93724940,"statfs":{"total":96636764160,"available":95974338560,"internally_reserved":0,"allocated":362913792,"data_stored":362645641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":299495088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,1,1,0,0,0,0,0,1],"upper_bound":8192},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.194},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.28299999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.318},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.20399999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691909,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596936,"kb_used_data":294592,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93774904,"statfs":{"total":96636764160,"available":96025501696,"internally_reserved":0,"allocated":301662208,"data_stored":301329023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":309587626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.0550000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.31}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.149},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.0680000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215191,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":776400,"kb_used_data":393736,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93595440,"statfs":{"total":96636764160,"available":95841730560,"internally_reserved":0,"allocated":403185664,"data_stored":402925199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":391835304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,1,0,0,0,0,0,0,11],"upper_bound":8192},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.217},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.436}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.5},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401657856,"data_stored":401654243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301182976,"data_stored":301110257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":362102784,"data_stored":362095099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:31.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:31.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'13885", 'reported_seq': 14138, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:26.529477+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:26.529477+0000', 'last_peered': '2024-09-16T10:29:26.529477+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:26.529477+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "35'3800", 'ondisk_log_start': "35'3800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27762712, 'num_objects': 6779, 'num_object_clones': 0, 'num_object_copies': 13558, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6779, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9942, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 784}, {'shards': '2', 'objects': 2476}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:31.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:31.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'13626", 'reported_seq': 13919, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:26.880082+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:26.880082+0000', 'last_peered': '2024-09-16T10:29:26.880082+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:26.880082+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'3600", 'ondisk_log_start': "35'3600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27627520, 'num_objects': 6745, 'num_object_clones': 0, 'num_object_copies': 13490, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4156, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6745, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10251, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2591}, {'shards': '0,2', 'objects': 779}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:31.480 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:31.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'14104", 'reported_seq': 14471, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:26.898832+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:26.898832+0000', 'last_peered': '2024-09-16T10:29:26.898832+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:26.898832+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27918336, 'num_objects': 6816, 'num_object_clones': 0, 'num_object_copies': 13632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4310, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6816, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10346, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 95, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2508}, {'shards': '0,2', 'objects': 762}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:31.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:31.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'14172", 'reported_seq': 14411, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:26.915687+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:26.915687+0000', 'last_peered': '2024-09-16T10:29:26.915687+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:26.915687+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "35'4100", 'ondisk_log_start': "35'4100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28132294, 'num_objects': 6870, 'num_object_clones': 0, 'num_object_copies': 13740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4357, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14170, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2515}, {'shards': '0,2', 'objects': 786}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:31.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:31.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'13714", 'reported_seq': 13881, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:26.501656+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:26.501656+0000', 'last_peered': '2024-09-16T10:29:26.501656+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:26.501656+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'3700", 'ondisk_log_start': "35'3700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27464646, 'num_objects': 6707, 'num_object_clones': 0, 'num_object_copies': 13414, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4250, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13712, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 31, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 775}, {'shards': '2', 'objects': 2459}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:31.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:31.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'13730", 'reported_seq': 15303, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:26.586838+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:29:26.586838+0000', 'last_peered': '2024-09-16T10:29:26.586838+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:26.586838+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "35'3700", 'ondisk_log_start': "35'3700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28042182, 'num_objects': 6848, 'num_object_clones': 0, 'num_object_copies': 13696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2864, 'num_objects_misplaced': 589, 'num_objects_unfound': 0, 'num_objects_dirty': 6848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13728, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 743, 'num_bytes_recovered': 2916352, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 589}, {'shards': '2', 'objects': 3395}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:31.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:31.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'13441", 'reported_seq': 13667, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:26.823969+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:26.823969+0000', 'last_peered': '2024-09-16T10:29:26.823969+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:26.823969+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'3400", 'ondisk_log_start': "35'3400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27005411, 'num_objects': 6594, 'num_object_clones': 0, 'num_object_copies': 13188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4088, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6594, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13440, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2507}, {'shards': '0,2', 'objects': 736}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:31.481 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:31.482 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'14044", 'reported_seq': 14298, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:26.906790+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:26.906790+0000', 'last_peered': '2024-09-16T10:29:26.906790+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:26.906790+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "35'4000", 'ondisk_log_start': "35'4000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28140969, 'num_objects': 6873, 'num_object_clones': 0, 'num_object_copies': 13746, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6873, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14041, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 62, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2595}, {'shards': '0,2', 'objects': 754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:31.482 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:31.482 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'13530", 'reported_seq': 13741, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:26.915629+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:26.915629+0000', 'last_peered': '2024-09-16T10:29:26.915629+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:26.915629+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'3500", 'ondisk_log_start': "35'3500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27075526, 'num_objects': 6612, 'num_object_clones': 0, 'num_object_copies': 13224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4105, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6612, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13528, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 66, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2509}, {'shards': '0,2', 'objects': 709}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:31.482 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:31.482 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'14087", 'reported_seq': 14309, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:26.868902+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:26.868902+0000', 'last_peered': '2024-09-16T10:29:26.868902+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:26.868902+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "35'4000", 'ondisk_log_start': "35'4000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27891596, 'num_objects': 6813, 'num_object_clones': 0, 'num_object_copies': 13626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14084, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2481}, {'shards': '0,2', 'objects': 762}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:31.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:32.178 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:32.179 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41050/216158 objects degraded (18.991%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":460,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":108079,"data_bytes":442614305,"bytes_used":2068721664,"bytes_avail":287841570816,"bytes_total":289910292480,"degraded_objects":41050,"degraded_total":216158,"degraded_ratio":0.18990738256275502,"misplaced_objects":589,"misplaced_total":216158,"misplaced_ratio":0.002724858668196412,"recovering_objects_per_sec":28,"recovering_bytes_per_sec":103385,"recovering_keys_per_sec":0,"num_objects_recovered":345,"num_bytes_recovered":1241088,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":357},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-16T10:24:52.588313+0000","services":{}},"progress_events":{}} 2024-09-16T10:29:32.201 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:32.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:32.748 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:32.750 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:32.771 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-16T10:29:32.692028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":434389537,"num_objects":106071,"num_object_clones":0,"num_object_copies":212142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41032,"num_objects_misplaced":463,"num_objects_unfound":0,"num_objects_dirty":106071,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200238,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9852,"num_bytes_recovered":37798390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2014304,"kb_used_data":1031028,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281101216,"statfs":{"total":289910292480,"available":287847645184,"internally_reserved":0,"allocated":1055772672,"data_stored":1054910871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1006816258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,8,4,3,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25313280,"num_objects":-6180,"num_object_clones":0,"num_object_copies":-12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-60,"num_objects_misplaced":-429,"num_objects_unfound":0,"num_objects_dirty":-6180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":489,"num_bytes_recovered":1757184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007048"},"pg_stats":[{"pgid":"2.f","version":"46'13830","reported_seq":22295,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.843503+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:31.843503+0000","last_peered":"2024-09-16T10:29:31.843503+0000","last_clean":"2024-09-16T10:29:31.843503+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:31.843503+0000","last_undegraded":"2024-09-16T10:29:31.843503+0000","last_fullsized":"2024-09-16T10:29:31.843503+0000","mapping_epoch":45,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27189248,"num_objects":6638,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13830,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13831","reported_seq":13991,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.534491+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:31.534491+0000","last_peered":"2024-09-16T10:29:31.534491+0000","last_clean":"2024-09-16T10:29:31.534491+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:31.534491+0000","last_undegraded":"2024-09-16T10:29:31.534491+0000","last_fullsized":"2024-09-16T10:29:31.534491+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160576,"num_objects":6631,"num_object_clones":0,"num_object_copies":13262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10044,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14032","reported_seq":14285,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.506858+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:31.506858+0000","last_peered":"2024-09-16T10:29:31.506858+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:31.506858+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160600,"num_objects":6632,"num_object_clones":0,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10089,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":637},{"shards":"2","objects":2329}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13854","reported_seq":13913,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.595116+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:31.595116+0000","last_peered":"2024-09-16T10:29:31.595116+0000","last_clean":"2024-09-16T10:29:31.595116+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:31.595116+0000","last_undegraded":"2024-09-16T10:29:31.595116+0000","last_fullsized":"2024-09-16T10:29:31.595116+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27066368,"num_objects":6608,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13854,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13871","reported_seq":14031,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.839428+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:31.839428+0000","last_peered":"2024-09-16T10:29:31.839428+0000","last_clean":"2024-09-16T10:29:31.839428+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:31.839428+0000","last_undegraded":"2024-09-16T10:29:31.839428+0000","last_fullsized":"2024-09-16T10:29:31.839428+0000","mapping_epoch":33,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27172864,"num_objects":6634,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10513,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13767","reported_seq":14070,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.806597+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:31.806597+0000","last_peered":"2024-09-16T10:29:31.806597+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:31.806597+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27049984,"num_objects":6604,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2455},{"shards":"0,2","objects":638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14240","reported_seq":14615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.924513+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:31.924513+0000","last_peered":"2024-09-16T10:29:31.924513+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:31.924513+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"38'4200","ondisk_log_start":"38'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27361280,"num_objects":6680,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10482,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2376},{"shards":"0,2","objects":626}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13793","reported_seq":13971,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.871270+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:31.871270+0000","last_peered":"2024-09-16T10:29:31.871270+0000","last_clean":"2024-09-16T10:29:31.871270+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:31.871270+0000","last_undegraded":"2024-09-16T10:29:31.871270+0000","last_fullsized":"2024-09-16T10:29:31.871270+0000","mapping_epoch":33,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27021312,"num_objects":6597,"num_object_clones":0,"num_object_copies":13194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9596,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13816","reported_seq":22322,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:30.448905+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:30.448905+0000","last_peered":"2024-09-16T10:29:30.448905+0000","last_clean":"2024-09-16T10:29:30.448905+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:30.448905+0000","last_undegraded":"2024-09-16T10:29:30.448905+0000","last_fullsized":"2024-09-16T10:29:30.448905+0000","mapping_epoch":43,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27218886,"num_objects":6647,"num_object_clones":0,"num_object_copies":13294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13814,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14304","reported_seq":14550,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.946564+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:31.946564+0000","last_peered":"2024-09-16T10:29:31.946564+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:31.946564+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"38'4300","ondisk_log_start":"38'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27591622,"num_objects":6738,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14302,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2386},{"shards":"0,2","objects":654}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13849","reported_seq":14016,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.570045+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:31.570045+0000","last_peered":"2024-09-16T10:29:31.570045+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:31.570045+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26911686,"num_objects":6572,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13847,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":640},{"shards":"2","objects":2325}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13872","reported_seq":15697,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.607893+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:31.607893+0000","last_peered":"2024-09-16T10:29:31.607893+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:31.607893+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27460550,"num_objects":6706,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":463,"num_objects_unfound":0,"num_objects_dirty":6706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13870,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":3432448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":463},{"shards":"2","objects":3379}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13558","reported_seq":13785,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.843609+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:31.843609+0000","last_peered":"2024-09-16T10:29:31.843609+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:31.843609+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26526179,"num_objects":6477,"num_object_clones":0,"num_object_copies":12954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13557,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2390},{"shards":"0,2","objects":619}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14170","reported_seq":14426,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.938465+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:31.938465+0000","last_peered":"2024-09-16T10:29:31.938465+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:31.938465+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27624873,"num_objects":6747,"num_object_clones":0,"num_object_copies":13494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2470},{"shards":"0,2","objects":628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13659","reported_seq":13871,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.898492+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:31.898492+0000","last_peered":"2024-09-16T10:29:31.898492+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:31.898492+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26547142,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13657,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2380},{"shards":"0,2","objects":580}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14225","reported_seq":14455,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.919050+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:31.919050+0000","last_peered":"2024-09-16T10:29:31.919050+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:31.919050+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"38'4200","ondisk_log_start":"38'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27326348,"num_objects":6675,"num_object_clones":0,"num_object_copies":13350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14222,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2347},{"shards":"0,2","objects":624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":434389518,"num_objects":106069,"num_object_clones":0,"num_object_copies":212138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41032,"num_objects_misplaced":463,"num_objects_unfound":0,"num_objects_dirty":106069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200236,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9851,"num_bytes_recovered":37798371,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1053999104,"data_stored":1053915087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018965,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660640,"kb_used_data":350100,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93711200,"statfs":{"total":96636764160,"available":95960268800,"internally_reserved":0,"allocated":358502400,"data_stored":358234249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,1,0,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.802},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.3199999999999998},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.7290000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691909,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596936,"kb_used_data":294592,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93774904,"statfs":{"total":96636764160,"available":96025501696,"internally_reserved":0,"allocated":301662208,"data_stored":301329023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":309587626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.0550000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.31}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.149},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.0680000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215192,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756728,"kb_used_data":386336,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93615112,"statfs":{"total":96636764160,"available":95861874688,"internally_reserved":0,"allocated":395608064,"data_stored":395347599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":7.726},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.5389999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":1.988},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":7.7679999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395186176,"data_stored":395182563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301182976,"data_stored":301110257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357629952,"data_stored":357622267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:35.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:36.311 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:36.314 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:36.336 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-16T10:29:34.693021+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":434389537,"num_objects":106071,"num_object_clones":0,"num_object_copies":212142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41032,"num_objects_misplaced":463,"num_objects_unfound":0,"num_objects_dirty":106071,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200238,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9852,"num_bytes_recovered":37798390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160774,"ondisk_log_size":160774,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2014304,"kb_used_data":1031028,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281101216,"statfs":{"total":289910292480,"available":287847645184,"internally_reserved":0,"allocated":1055772672,"data_stored":1054910871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1006816258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,5,8,4,3,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17162240,"num_objects":-4190,"num_object_clones":0,"num_object_copies":-8380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-39,"num_objects_misplaced":-293,"num_objects_unfound":0,"num_objects_dirty":-4190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4190,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":1200128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007422"},"pg_stats":[{"pgid":"2.f","version":"46'13830","reported_seq":22295,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.843503+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:31.843503+0000","last_peered":"2024-09-16T10:29:31.843503+0000","last_clean":"2024-09-16T10:29:31.843503+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:31.843503+0000","last_undegraded":"2024-09-16T10:29:31.843503+0000","last_fullsized":"2024-09-16T10:29:31.843503+0000","mapping_epoch":45,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27189248,"num_objects":6638,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13830,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13831","reported_seq":13991,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.534491+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:31.534491+0000","last_peered":"2024-09-16T10:29:31.534491+0000","last_clean":"2024-09-16T10:29:31.534491+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:31.534491+0000","last_undegraded":"2024-09-16T10:29:31.534491+0000","last_fullsized":"2024-09-16T10:29:31.534491+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160576,"num_objects":6631,"num_object_clones":0,"num_object_copies":13262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10044,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14032","reported_seq":14285,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.506858+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:31.506858+0000","last_peered":"2024-09-16T10:29:31.506858+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:31.506858+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160600,"num_objects":6632,"num_object_clones":0,"num_object_copies":13264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10089,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":637},{"shards":"2","objects":2329}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'13854","reported_seq":13913,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.595116+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:31.595116+0000","last_peered":"2024-09-16T10:29:31.595116+0000","last_clean":"2024-09-16T10:29:31.595116+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:31.595116+0000","last_undegraded":"2024-09-16T10:29:31.595116+0000","last_fullsized":"2024-09-16T10:29:31.595116+0000","mapping_epoch":34,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27066368,"num_objects":6608,"num_object_clones":0,"num_object_copies":13216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13854,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13871","reported_seq":14031,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.839428+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:31.839428+0000","last_peered":"2024-09-16T10:29:31.839428+0000","last_clean":"2024-09-16T10:29:31.839428+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:31.839428+0000","last_undegraded":"2024-09-16T10:29:31.839428+0000","last_fullsized":"2024-09-16T10:29:31.839428+0000","mapping_epoch":33,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27172864,"num_objects":6634,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10513,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13767","reported_seq":14070,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.806597+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:31.806597+0000","last_peered":"2024-09-16T10:29:31.806597+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:31.806597+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27049984,"num_objects":6604,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2455},{"shards":"0,2","objects":638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14240","reported_seq":14615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.924513+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:31.924513+0000","last_peered":"2024-09-16T10:29:31.924513+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:31.924513+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"38'4200","ondisk_log_start":"38'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27361280,"num_objects":6680,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10482,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2376},{"shards":"0,2","objects":626}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13793","reported_seq":13971,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.871270+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:31.871270+0000","last_peered":"2024-09-16T10:29:31.871270+0000","last_clean":"2024-09-16T10:29:31.871270+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:31.871270+0000","last_undegraded":"2024-09-16T10:29:31.871270+0000","last_fullsized":"2024-09-16T10:29:31.871270+0000","mapping_epoch":33,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27021312,"num_objects":6597,"num_object_clones":0,"num_object_copies":13194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9596,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13816","reported_seq":22322,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:30.448905+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:30.448905+0000","last_peered":"2024-09-16T10:29:30.448905+0000","last_clean":"2024-09-16T10:29:30.448905+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:30.448905+0000","last_undegraded":"2024-09-16T10:29:30.448905+0000","last_fullsized":"2024-09-16T10:29:30.448905+0000","mapping_epoch":43,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27218886,"num_objects":6647,"num_object_clones":0,"num_object_copies":13294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13814,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14304","reported_seq":14550,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.946564+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:31.946564+0000","last_peered":"2024-09-16T10:29:31.946564+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:31.946564+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"38'4300","ondisk_log_start":"38'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27591622,"num_objects":6738,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14302,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2386},{"shards":"0,2","objects":654}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13849","reported_seq":14016,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.570045+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:31.570045+0000","last_peered":"2024-09-16T10:29:31.570045+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:31.570045+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26911686,"num_objects":6572,"num_object_clones":0,"num_object_copies":13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13847,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":640},{"shards":"2","objects":2325}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'13872","reported_seq":15697,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.607893+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:31.607893+0000","last_peered":"2024-09-16T10:29:31.607893+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:31.607893+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27460550,"num_objects":6706,"num_object_clones":0,"num_object_copies":13412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":463,"num_objects_unfound":0,"num_objects_dirty":6706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13870,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":869,"num_bytes_recovered":3432448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":463},{"shards":"2","objects":3379}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13558","reported_seq":13785,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.843609+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:31.843609+0000","last_peered":"2024-09-16T10:29:31.843609+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:31.843609+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26526179,"num_objects":6477,"num_object_clones":0,"num_object_copies":12954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13557,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2390},{"shards":"0,2","objects":619}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14170","reported_seq":14426,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.938465+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:31.938465+0000","last_peered":"2024-09-16T10:29:31.938465+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:31.938465+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27624873,"num_objects":6747,"num_object_clones":0,"num_object_copies":13494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2470},{"shards":"0,2","objects":628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13659","reported_seq":13871,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.898492+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:31.898492+0000","last_peered":"2024-09-16T10:29:31.898492+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:31.898492+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26547142,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13657,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2380},{"shards":"0,2","objects":580}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14225","reported_seq":14455,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.919050+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:31.919050+0000","last_peered":"2024-09-16T10:29:31.919050+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:31.919050+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"38'4200","ondisk_log_start":"38'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27326348,"num_objects":6675,"num_object_clones":0,"num_object_copies":13350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14222,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2347},{"shards":"0,2","objects":624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":434389518,"num_objects":106069,"num_object_clones":0,"num_object_copies":212138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41032,"num_objects_misplaced":463,"num_objects_unfound":0,"num_objects_dirty":106069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200236,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9851,"num_bytes_recovered":37798371,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1053999104,"data_stored":1053915087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160771,"ondisk_log_size":160771,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018965,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":660640,"kb_used_data":350100,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93711200,"statfs":{"total":96636764160,"available":95960268800,"internally_reserved":0,"allocated":358502400,"data_stored":358234249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,1,0,0,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.802},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.3199999999999998},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.7290000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691909,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596936,"kb_used_data":294592,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93774904,"statfs":{"total":96636764160,"available":96025501696,"internally_reserved":0,"allocated":301662208,"data_stored":301329023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":309587626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,5,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.0550000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.31}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.149},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.0680000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215192,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756728,"kb_used_data":386336,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93615112,"statfs":{"total":96636764160,"available":95861874688,"internally_reserved":0,"allocated":395608064,"data_stored":395347599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":7.726},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.5389999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":1.988},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":7.7679999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395186176,"data_stored":395182563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301182976,"data_stored":301110257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357629952,"data_stored":357622267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:36.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:36.910 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:36.913 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:36.939 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-16T10:29:36.693709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":431063585,"num_objects":105259,"num_object_clones":0,"num_object_copies":210518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41024,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":105259,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201050,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9982,"num_bytes_recovered":38298102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2000700,"kb_used_data":1025620,"kb_used_omap":12,"kb_used_meta":975027,"kb_avail":281114820,"statfs":{"total":289910292480,"available":287861575680,"internally_reserved":0,"allocated":1050234880,"data_stored":1049373079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":998427650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,2,7,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20488192,"num_objects":-5002,"num_object_clones":0,"num_object_copies":-10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-47,"num_objects_misplaced":-415,"num_objects_unfound":0,"num_objects_dirty":-5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5002,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":462,"num_bytes_recovered":1699840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007395"},"pg_stats":[{"pgid":"2.f","version":"46'13830","reported_seq":22295,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.843503+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:31.843503+0000","last_peered":"2024-09-16T10:29:31.843503+0000","last_clean":"2024-09-16T10:29:31.843503+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:31.843503+0000","last_undegraded":"2024-09-16T10:29:31.843503+0000","last_fullsized":"2024-09-16T10:29:31.843503+0000","mapping_epoch":45,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27189248,"num_objects":6638,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13830,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13963","reported_seq":14123,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.558678+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:36.558678+0000","last_peered":"2024-09-16T10:29:36.558678+0000","last_clean":"2024-09-16T10:29:36.558678+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:36.558678+0000","last_undegraded":"2024-09-16T10:29:36.558678+0000","last_fullsized":"2024-09-16T10:29:36.558678+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26619904,"num_objects":6499,"num_object_clones":0,"num_object_copies":12998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10176,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14143","reported_seq":14395,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.552721+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:36.552721+0000","last_peered":"2024-09-16T10:29:36.552721+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:36.552721+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26705944,"num_objects":6521,"num_object_clones":0,"num_object_copies":13042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10200,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":526},{"shards":"2","objects":2218}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14004","reported_seq":14063,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.652611+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:36.652611+0000","last_peered":"2024-09-16T10:29:36.652611+0000","last_clean":"2024-09-16T10:29:36.652611+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:36.652611+0000","last_undegraded":"2024-09-16T10:29:36.652611+0000","last_fullsized":"2024-09-16T10:29:36.652611+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26451968,"num_objects":6458,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14004,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13871","reported_seq":14031,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.839428+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:31.839428+0000","last_peered":"2024-09-16T10:29:31.839428+0000","last_clean":"2024-09-16T10:29:31.839428+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:31.839428+0000","last_undegraded":"2024-09-16T10:29:31.839428+0000","last_fullsized":"2024-09-16T10:29:31.839428+0000","mapping_epoch":33,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27172864,"num_objects":6634,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10513,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13767","reported_seq":14070,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.806597+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:31.806597+0000","last_peered":"2024-09-16T10:29:31.806597+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:31.806597+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27049984,"num_objects":6604,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2455},{"shards":"0,2","objects":638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14240","reported_seq":14615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.924513+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:31.924513+0000","last_peered":"2024-09-16T10:29:31.924513+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:31.924513+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"38'4200","ondisk_log_start":"38'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27361280,"num_objects":6680,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10482,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2376},{"shards":"0,2","objects":626}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13793","reported_seq":13971,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.871270+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:31.871270+0000","last_peered":"2024-09-16T10:29:31.871270+0000","last_clean":"2024-09-16T10:29:31.871270+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:31.871270+0000","last_undegraded":"2024-09-16T10:29:31.871270+0000","last_fullsized":"2024-09-16T10:29:31.871270+0000","mapping_epoch":33,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27021312,"num_objects":6597,"num_object_clones":0,"num_object_copies":13194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9596,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13946","reported_seq":22452,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:35.457141+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:35.457141+0000","last_peered":"2024-09-16T10:29:35.457141+0000","last_clean":"2024-09-16T10:29:35.457141+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:35.457141+0000","last_undegraded":"2024-09-16T10:29:35.457141+0000","last_fullsized":"2024-09-16T10:29:35.457141+0000","mapping_epoch":43,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686406,"num_objects":6517,"num_object_clones":0,"num_object_copies":13034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13944,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14304","reported_seq":14550,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.946564+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:31.946564+0000","last_peered":"2024-09-16T10:29:31.946564+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:31.946564+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"38'4300","ondisk_log_start":"38'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27591622,"num_objects":6738,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14302,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2386},{"shards":"0,2","objects":654}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13990","reported_seq":14175,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.536348+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:36.536348+0000","last_peered":"2024-09-16T10:29:36.536348+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:36.536348+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26334150,"num_objects":6431,"num_object_clones":0,"num_object_copies":12862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13988,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":499},{"shards":"2","objects":2192}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14020","reported_seq":16089,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.626766+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:36.626766+0000","last_peered":"2024-09-16T10:29:36.626766+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:36.626766+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26854342,"num_objects":6558,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":6558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14018,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":991,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":341},{"shards":"2","objects":3353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13558","reported_seq":13785,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.843609+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:31.843609+0000","last_peered":"2024-09-16T10:29:31.843609+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:31.843609+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26526179,"num_objects":6477,"num_object_clones":0,"num_object_copies":12954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13557,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2390},{"shards":"0,2","objects":619}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14170","reported_seq":14426,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.938465+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:31.938465+0000","last_peered":"2024-09-16T10:29:31.938465+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:31.938465+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27624873,"num_objects":6747,"num_object_clones":0,"num_object_copies":13494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2470},{"shards":"0,2","objects":628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13659","reported_seq":13871,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.898492+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:31.898492+0000","last_peered":"2024-09-16T10:29:31.898492+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:31.898492+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26547142,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13657,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2380},{"shards":"0,2","objects":580}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14225","reported_seq":14455,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.919050+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:31.919050+0000","last_peered":"2024-09-16T10:29:31.919050+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:31.919050+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"38'4200","ondisk_log_start":"38'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27326348,"num_objects":6675,"num_object_clones":0,"num_object_copies":13350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14222,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2347},{"shards":"0,2","objects":624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":431063566,"num_objects":105257,"num_object_clones":0,"num_object_copies":210514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41024,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":105257,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201048,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9981,"num_bytes_recovered":38298083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048199168,"data_stored":1048115151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018966,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643956,"kb_used_data":345708,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93727884,"statfs":{"total":96636764160,"available":95977353216,"internally_reserved":0,"allocated":354004992,"data_stored":353736841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.1680000000000001},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.24299999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.375},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.28799999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691910,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600016,"kb_used_data":293576,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93771824,"statfs":{"total":96636764160,"available":96022347776,"internally_reserved":0,"allocated":300621824,"data_stored":300288639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":313781930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":12.061999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.5640000000000001},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.4409999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215192,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756728,"kb_used_data":386336,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93615112,"statfs":{"total":96636764160,"available":95861874688,"internally_reserved":0,"allocated":395608064,"data_stored":395347599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":7.726},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.5389999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":1.988},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":7.7679999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395186176,"data_stored":395182563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":299958272,"data_stored":299885553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":353054720,"data_stored":353047035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:36.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:36.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'14143", 'reported_seq': 14395, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.552721+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:36.552721+0000', 'last_peered': '2024-09-16T10:29:36.552721+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:36.552721+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "38'4100", 'ondisk_log_start': "38'4100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26705944, 'num_objects': 6521, 'num_object_clones': 0, 'num_object_copies': 13042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6521, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10200, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 526}, {'shards': '2', 'objects': 2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:36.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:36.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'13767", 'reported_seq': 14070, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:31.806597+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:31.806597+0000', 'last_peered': '2024-09-16T10:29:31.806597+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:31.806597+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'3700", 'ondisk_log_start': "35'3700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27049984, 'num_objects': 6604, 'num_object_clones': 0, 'num_object_copies': 13208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4151, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10392, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 68, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2455}, {'shards': '0,2', 'objects': 638}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:36.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:36.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'14240", 'reported_seq': 14615, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:31.924513+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:31.924513+0000', 'last_peered': '2024-09-16T10:29:31.924513+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:31.924513+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "38'4200", 'ondisk_log_start': "38'4200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27361280, 'num_objects': 6680, 'num_object_clones': 0, 'num_object_copies': 13360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4306, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6680, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10482, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2376}, {'shards': '0,2', 'objects': 626}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'14304", 'reported_seq': 14550, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:31.946564+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:31.946564+0000', 'last_peered': '2024-09-16T10:29:31.946564+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:31.946564+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "38'4300", 'ondisk_log_start': "38'4300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27591622, 'num_objects': 6738, 'num_object_clones': 0, 'num_object_copies': 13476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4354, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6738, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14302, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2386}, {'shards': '0,2', 'objects': 654}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'13990", 'reported_seq': 14175, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.536348+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:36.536348+0000', 'last_peered': '2024-09-16T10:29:36.536348+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:36.536348+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26334150, 'num_objects': 6431, 'num_object_clones': 0, 'num_object_copies': 12862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13988, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 499}, {'shards': '2', 'objects': 2192}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'14020", 'reported_seq': 16089, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.626766+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:29:36.626766+0000', 'last_peered': '2024-09-16T10:29:36.626766+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:36.626766+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "38'4000", 'ondisk_log_start': "38'4000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26854342, 'num_objects': 6558, 'num_object_clones': 0, 'num_object_copies': 13116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2864, 'num_objects_misplaced': 341, 'num_objects_unfound': 0, 'num_objects_dirty': 6558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14018, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 991, 'num_bytes_recovered': 3932160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 341}, {'shards': '2', 'objects': 3353}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'13558", 'reported_seq': 13785, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:31.843609+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:31.843609+0000', 'last_peered': '2024-09-16T10:29:31.843609+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:31.843609+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'3500", 'ondisk_log_start': "35'3500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26526179, 'num_objects': 6477, 'num_object_clones': 0, 'num_object_copies': 12954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4088, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13557, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2390}, {'shards': '0,2', 'objects': 619}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'14170", 'reported_seq': 14426, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:31.938465+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:31.938465+0000', 'last_peered': '2024-09-16T10:29:31.938465+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:31.938465+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "38'4100", 'ondisk_log_start': "38'4100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27624873, 'num_objects': 6747, 'num_object_clones': 0, 'num_object_copies': 13494, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14167, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2470}, {'shards': '0,2', 'objects': 628}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:36.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:36.943 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'13659", 'reported_seq': 13871, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:31.898492+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:31.898492+0000', 'last_peered': '2024-09-16T10:29:31.898492+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:31.898492+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'3600", 'ondisk_log_start': "35'3600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26547142, 'num_objects': 6483, 'num_object_clones': 0, 'num_object_copies': 12966, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4105, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13657, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 66, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2380}, {'shards': '0,2', 'objects': 580}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:36.943 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:36.943 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'14225", 'reported_seq': 14455, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:31.919050+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:31.919050+0000', 'last_peered': '2024-09-16T10:29:31.919050+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:31.919050+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "38'4200", 'ondisk_log_start': "38'4200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27326348, 'num_objects': 6675, 'num_object_clones': 0, 'num_object_copies': 13350, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4331, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14222, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 72, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2347}, {'shards': '0,2', 'objects': 624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:36.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:37.636 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:37.636 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41024/210518 objects degraded (19.487%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":466,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":105259,"data_bytes":431063585,"bytes_used":2048716800,"bytes_avail":287861575680,"bytes_total":289910292480,"degraded_objects":41024,"degraded_total":210518,"degraded_ratio":0.1948716974320486,"misplaced_objects":341,"misplaced_total":210518,"misplaced_ratio":0.0016198139826523147,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":141566,"recovering_keys_per_sec":0,"num_objects_recovered":462,"num_bytes_recovered":1699840,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":416},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T10:29:36.693808+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:29:37.656 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:37.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:38.181 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:38.181 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:38.203 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-16T10:29:36.693709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":431063585,"num_objects":105259,"num_object_clones":0,"num_object_copies":210518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41024,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":105259,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201050,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9982,"num_bytes_recovered":38298102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160786,"ondisk_log_size":160786,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2000700,"kb_used_data":1025620,"kb_used_omap":12,"kb_used_meta":975027,"kb_avail":281114820,"statfs":{"total":289910292480,"available":287861575680,"internally_reserved":0,"allocated":1050234880,"data_stored":1049373079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":998427650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,2,7,5,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20488192,"num_objects":-5002,"num_object_clones":0,"num_object_copies":-10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-47,"num_objects_misplaced":-415,"num_objects_unfound":0,"num_objects_dirty":-5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5002,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":462,"num_bytes_recovered":1699840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007395"},"pg_stats":[{"pgid":"2.f","version":"46'13830","reported_seq":22295,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.843503+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:31.843503+0000","last_peered":"2024-09-16T10:29:31.843503+0000","last_clean":"2024-09-16T10:29:31.843503+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:31.843503+0000","last_undegraded":"2024-09-16T10:29:31.843503+0000","last_fullsized":"2024-09-16T10:29:31.843503+0000","mapping_epoch":45,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27189248,"num_objects":6638,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13830,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13963","reported_seq":14123,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.558678+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:36.558678+0000","last_peered":"2024-09-16T10:29:36.558678+0000","last_clean":"2024-09-16T10:29:36.558678+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:36.558678+0000","last_undegraded":"2024-09-16T10:29:36.558678+0000","last_fullsized":"2024-09-16T10:29:36.558678+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26619904,"num_objects":6499,"num_object_clones":0,"num_object_copies":12998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10176,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14143","reported_seq":14395,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.552721+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:36.552721+0000","last_peered":"2024-09-16T10:29:36.552721+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:36.552721+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26705944,"num_objects":6521,"num_object_clones":0,"num_object_copies":13042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10200,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":526},{"shards":"2","objects":2218}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14004","reported_seq":14063,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.652611+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:36.652611+0000","last_peered":"2024-09-16T10:29:36.652611+0000","last_clean":"2024-09-16T10:29:36.652611+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:36.652611+0000","last_undegraded":"2024-09-16T10:29:36.652611+0000","last_fullsized":"2024-09-16T10:29:36.652611+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26451968,"num_objects":6458,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14004,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'13871","reported_seq":14031,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.839428+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:31.839428+0000","last_peered":"2024-09-16T10:29:31.839428+0000","last_clean":"2024-09-16T10:29:31.839428+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:31.839428+0000","last_undegraded":"2024-09-16T10:29:31.839428+0000","last_fullsized":"2024-09-16T10:29:31.839428+0000","mapping_epoch":33,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27172864,"num_objects":6634,"num_object_clones":0,"num_object_copies":13268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10513,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13767","reported_seq":14070,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.806597+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:31.806597+0000","last_peered":"2024-09-16T10:29:31.806597+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:31.806597+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27049984,"num_objects":6604,"num_object_clones":0,"num_object_copies":13208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2455},{"shards":"0,2","objects":638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14240","reported_seq":14615,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.924513+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:31.924513+0000","last_peered":"2024-09-16T10:29:31.924513+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:31.924513+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"38'4200","ondisk_log_start":"38'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27361280,"num_objects":6680,"num_object_clones":0,"num_object_copies":13360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10482,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2376},{"shards":"0,2","objects":626}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13793","reported_seq":13971,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:31.871270+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:31.871270+0000","last_peered":"2024-09-16T10:29:31.871270+0000","last_clean":"2024-09-16T10:29:31.871270+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:31.871270+0000","last_undegraded":"2024-09-16T10:29:31.871270+0000","last_fullsized":"2024-09-16T10:29:31.871270+0000","mapping_epoch":33,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27021312,"num_objects":6597,"num_object_clones":0,"num_object_copies":13194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9596,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'13946","reported_seq":22452,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:35.457141+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:35.457141+0000","last_peered":"2024-09-16T10:29:35.457141+0000","last_clean":"2024-09-16T10:29:35.457141+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:35.457141+0000","last_undegraded":"2024-09-16T10:29:35.457141+0000","last_fullsized":"2024-09-16T10:29:35.457141+0000","mapping_epoch":43,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686406,"num_objects":6517,"num_object_clones":0,"num_object_copies":13034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13944,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14304","reported_seq":14550,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.946564+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:31.946564+0000","last_peered":"2024-09-16T10:29:31.946564+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:31.946564+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"38'4300","ondisk_log_start":"38'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27591622,"num_objects":6738,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14302,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2386},{"shards":"0,2","objects":654}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13990","reported_seq":14175,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.536348+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:36.536348+0000","last_peered":"2024-09-16T10:29:36.536348+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:36.536348+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26334150,"num_objects":6431,"num_object_clones":0,"num_object_copies":12862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13988,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":499},{"shards":"2","objects":2192}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14020","reported_seq":16089,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.626766+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:36.626766+0000","last_peered":"2024-09-16T10:29:36.626766+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:36.626766+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26854342,"num_objects":6558,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":6558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14018,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":991,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":341},{"shards":"2","objects":3353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13558","reported_seq":13785,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.843609+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:31.843609+0000","last_peered":"2024-09-16T10:29:31.843609+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:31.843609+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3500","ondisk_log_start":"35'3500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26526179,"num_objects":6477,"num_object_clones":0,"num_object_copies":12954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13557,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2390},{"shards":"0,2","objects":619}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14170","reported_seq":14426,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.938465+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:31.938465+0000","last_peered":"2024-09-16T10:29:31.938465+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:31.938465+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27624873,"num_objects":6747,"num_object_clones":0,"num_object_copies":13494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14167,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2470},{"shards":"0,2","objects":628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13659","reported_seq":13871,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.898492+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:31.898492+0000","last_peered":"2024-09-16T10:29:31.898492+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:31.898492+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26547142,"num_objects":6483,"num_object_clones":0,"num_object_copies":12966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13657,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2380},{"shards":"0,2","objects":580}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14225","reported_seq":14455,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:31.919050+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:31.919050+0000","last_peered":"2024-09-16T10:29:31.919050+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:31.919050+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"38'4200","ondisk_log_start":"38'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27326348,"num_objects":6675,"num_object_clones":0,"num_object_copies":13350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14222,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2347},{"shards":"0,2","objects":624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":431063566,"num_objects":105257,"num_object_clones":0,"num_object_copies":210514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41024,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":105257,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201048,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9981,"num_bytes_recovered":38298083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048199168,"data_stored":1048115151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160783,"ondisk_log_size":160783,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018966,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643956,"kb_used_data":345708,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93727884,"statfs":{"total":96636764160,"available":95977353216,"internally_reserved":0,"allocated":354004992,"data_stored":353736841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.1680000000000001},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.24299999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.375},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.28799999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691910,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":600016,"kb_used_data":293576,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93771824,"statfs":{"total":96636764160,"available":96022347776,"internally_reserved":0,"allocated":300621824,"data_stored":300288639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":313781930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":12.061999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.5640000000000001},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.4409999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215192,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756728,"kb_used_data":386336,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93615112,"statfs":{"total":96636764160,"available":95861874688,"internally_reserved":0,"allocated":395608064,"data_stored":395347599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":7.726},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.5389999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":1.988},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":7.7679999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":395186176,"data_stored":395182563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":299958272,"data_stored":299885553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":353054720,"data_stored":353047035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:41.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:41.750 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:41.750 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:41.780 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-16T10:29:40.695088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":424862241,"num_objects":103745,"num_object_clones":0,"num_object_copies":207490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41014,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":103745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202564,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9992,"num_bytes_recovered":38298102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2009352,"kb_used_data":1017896,"kb_used_omap":12,"kb_used_meta":991411,"kb_avail":281106168,"statfs":{"total":289910292480,"available":287852716032,"internally_reserved":0,"allocated":1042325504,"data_stored":1041463703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1015204866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1,3,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18280448,"num_objects":-4463,"num_object_clones":0,"num_object_copies":-8926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-36,"num_objects_misplaced":-248,"num_objects_unfound":0,"num_objects_dirty":-4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4463,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":1015808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007087"},"pg_stats":[{"pgid":"2.f","version":"46'13976","reported_seq":22440,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.912086+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:36.912086+0000","last_peered":"2024-09-16T10:29:36.912086+0000","last_clean":"2024-09-16T10:29:36.912086+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:36.912086+0000","last_undegraded":"2024-09-16T10:29:36.912086+0000","last_fullsized":"2024-09-16T10:29:36.912086+0000","mapping_epoch":45,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26591232,"num_objects":6492,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13976,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13963","reported_seq":14123,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.558678+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:36.558678+0000","last_peered":"2024-09-16T10:29:36.558678+0000","last_clean":"2024-09-16T10:29:36.558678+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:36.558678+0000","last_undegraded":"2024-09-16T10:29:36.558678+0000","last_fullsized":"2024-09-16T10:29:36.558678+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26619904,"num_objects":6499,"num_object_clones":0,"num_object_copies":12998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10176,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14143","reported_seq":14395,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.552721+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:36.552721+0000","last_peered":"2024-09-16T10:29:36.552721+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:36.552721+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26705944,"num_objects":6521,"num_object_clones":0,"num_object_copies":13042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10200,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":526},{"shards":"2","objects":2218}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14004","reported_seq":14063,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.652611+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:36.652611+0000","last_peered":"2024-09-16T10:29:36.652611+0000","last_clean":"2024-09-16T10:29:36.652611+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:36.652611+0000","last_undegraded":"2024-09-16T10:29:36.652611+0000","last_fullsized":"2024-09-16T10:29:36.652611+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26451968,"num_objects":6458,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14004,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14020","reported_seq":14180,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.835000+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:36.835000+0000","last_peered":"2024-09-16T10:29:36.835000+0000","last_clean":"2024-09-16T10:29:36.835000+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:36.835000+0000","last_undegraded":"2024-09-16T10:29:36.835000+0000","last_fullsized":"2024-09-16T10:29:36.835000+0000","mapping_epoch":33,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26562560,"num_objects":6485,"num_object_clones":0,"num_object_copies":12970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10662,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13892","reported_seq":14197,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.940083+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:36.940083+0000","last_peered":"2024-09-16T10:29:36.940083+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:36.940083+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26537984,"num_objects":6479,"num_object_clones":0,"num_object_copies":12958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10517,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2331},{"shards":"0,2","objects":513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14363","reported_seq":14754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.805477+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:36.805477+0000","last_peered":"2024-09-16T10:29:36.805477+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:36.805477+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"38'4300","ondisk_log_start":"38'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26857472,"num_objects":6557,"num_object_clones":0,"num_object_copies":13114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2261},{"shards":"0,2","objects":503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13945","reported_seq":14123,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.874788+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:36.874788+0000","last_peered":"2024-09-16T10:29:36.874788+0000","last_clean":"2024-09-16T10:29:36.874788+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:36.874788+0000","last_undegraded":"2024-09-16T10:29:36.874788+0000","last_fullsized":"2024-09-16T10:29:36.874788+0000","mapping_epoch":33,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26398720,"num_objects":6445,"num_object_clones":0,"num_object_copies":12890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9748,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14075","reported_seq":22581,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.477939+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:40.477939+0000","last_peered":"2024-09-16T10:29:40.477939+0000","last_clean":"2024-09-16T10:29:40.477939+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:40.477939+0000","last_undegraded":"2024-09-16T10:29:40.477939+0000","last_fullsized":"2024-09-16T10:29:40.477939+0000","mapping_epoch":43,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26158022,"num_objects":6388,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14073,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14440","reported_seq":14685,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.947429+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:36.947429+0000","last_peered":"2024-09-16T10:29:36.947429+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:36.947429+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"39'4400","ondisk_log_start":"39'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27034566,"num_objects":6602,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14438,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2250},{"shards":"0,2","objects":518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13990","reported_seq":14175,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.536348+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:36.536348+0000","last_peered":"2024-09-16T10:29:36.536348+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:36.536348+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26334150,"num_objects":6431,"num_object_clones":0,"num_object_copies":12862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13988,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":499},{"shards":"2","objects":2192}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14020","reported_seq":16089,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.626766+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:36.626766+0000","last_peered":"2024-09-16T10:29:36.626766+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:36.626766+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26854342,"num_objects":6558,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":6558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14018,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":991,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":341},{"shards":"2","objects":3353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13686","reported_seq":13913,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.868242+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:36.868242+0000","last_peered":"2024-09-16T10:29:36.868242+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:36.868242+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26001891,"num_objects":6349,"num_object_clones":0,"num_object_copies":12698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13685,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2262},{"shards":"0,2","objects":491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14326","reported_seq":14582,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.867214+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:36.867214+0000","last_peered":"2024-09-16T10:29:36.867214+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:36.867214+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"39'4300","ondisk_log_start":"39'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26985897,"num_objects":6591,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14323,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2314},{"shards":"0,2","objects":472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13782","reported_seq":13996,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.946538+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:36.946538+0000","last_peered":"2024-09-16T10:29:36.946538+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:36.946538+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26043334,"num_objects":6360,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13780,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2258},{"shards":"0,2","objects":457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14372","reported_seq":14600,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.935162+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:36.935162+0000","last_peered":"2024-09-16T10:29:36.935162+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:36.935162+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"39'4300","ondisk_log_start":"39'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26724236,"num_objects":6528,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14369,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2200},{"shards":"0,2","objects":477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":424862222,"num_objects":103743,"num_object_clones":0,"num_object_copies":207486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41014,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":103743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202562,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9991,"num_bytes_recovered":38298083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1040359424,"data_stored":1040275407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160897,"ondisk_log_size":160897,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018966,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643956,"kb_used_data":345708,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93727884,"statfs":{"total":96636764160,"available":95977353216,"internally_reserved":0,"allocated":354004992,"data_stored":353736841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.1680000000000001},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.24299999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.375},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.28799999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691911,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607084,"kb_used_data":292452,"kb_used_omap":4,"kb_used_meta":314619,"kb_avail":93764756,"statfs":{"total":96636764160,"available":96015110144,"internally_reserved":0,"allocated":299470848,"data_stored":299137663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":322170538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":4.0220000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.0519999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":5.968},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.39500000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215193,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758312,"kb_used_data":379736,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93613528,"statfs":{"total":96636764160,"available":95860252672,"internally_reserved":0,"allocated":388849664,"data_stored":388589199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":387641000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":7.726},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.5389999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":1.988},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":7.7679999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":388354048,"data_stored":388350435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298950656,"data_stored":298877937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":353054720,"data_stored":353047035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:41.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:42.348 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:42.382 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-16T10:29:40.695088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":424862241,"num_objects":103745,"num_object_clones":0,"num_object_copies":207490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41014,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":103745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202564,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9992,"num_bytes_recovered":38298102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2009352,"kb_used_data":1017896,"kb_used_omap":12,"kb_used_meta":991411,"kb_avail":281106168,"statfs":{"total":289910292480,"available":287852716032,"internally_reserved":0,"allocated":1042325504,"data_stored":1041463703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1015204866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1,3,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18280448,"num_objects":-4463,"num_object_clones":0,"num_object_copies":-8926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-36,"num_objects_misplaced":-248,"num_objects_unfound":0,"num_objects_dirty":-4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4463,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":1015808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007087"},"pg_stats":[{"pgid":"2.f","version":"46'13976","reported_seq":22440,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.912086+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:36.912086+0000","last_peered":"2024-09-16T10:29:36.912086+0000","last_clean":"2024-09-16T10:29:36.912086+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:36.912086+0000","last_undegraded":"2024-09-16T10:29:36.912086+0000","last_fullsized":"2024-09-16T10:29:36.912086+0000","mapping_epoch":45,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26591232,"num_objects":6492,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13976,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'13963","reported_seq":14123,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.558678+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:36.558678+0000","last_peered":"2024-09-16T10:29:36.558678+0000","last_clean":"2024-09-16T10:29:36.558678+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:36.558678+0000","last_undegraded":"2024-09-16T10:29:36.558678+0000","last_fullsized":"2024-09-16T10:29:36.558678+0000","mapping_epoch":34,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26619904,"num_objects":6499,"num_object_clones":0,"num_object_copies":12998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10176,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14143","reported_seq":14395,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.552721+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:36.552721+0000","last_peered":"2024-09-16T10:29:36.552721+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:36.552721+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26705944,"num_objects":6521,"num_object_clones":0,"num_object_copies":13042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6521,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10200,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":526},{"shards":"2","objects":2218}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14004","reported_seq":14063,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.652611+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:36.652611+0000","last_peered":"2024-09-16T10:29:36.652611+0000","last_clean":"2024-09-16T10:29:36.652611+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:36.652611+0000","last_undegraded":"2024-09-16T10:29:36.652611+0000","last_fullsized":"2024-09-16T10:29:36.652611+0000","mapping_epoch":34,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26451968,"num_objects":6458,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14004,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14020","reported_seq":14180,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.835000+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:36.835000+0000","last_peered":"2024-09-16T10:29:36.835000+0000","last_clean":"2024-09-16T10:29:36.835000+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:36.835000+0000","last_undegraded":"2024-09-16T10:29:36.835000+0000","last_fullsized":"2024-09-16T10:29:36.835000+0000","mapping_epoch":33,"log_start":"35'4000","ondisk_log_start":"35'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26562560,"num_objects":6485,"num_object_clones":0,"num_object_copies":12970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10662,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'13892","reported_seq":14197,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.940083+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:36.940083+0000","last_peered":"2024-09-16T10:29:36.940083+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:36.940083+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26537984,"num_objects":6479,"num_object_clones":0,"num_object_copies":12958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10517,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2331},{"shards":"0,2","objects":513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14363","reported_seq":14754,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.805477+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:36.805477+0000","last_peered":"2024-09-16T10:29:36.805477+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:36.805477+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"38'4300","ondisk_log_start":"38'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26857472,"num_objects":6557,"num_object_clones":0,"num_object_copies":13114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2261},{"shards":"0,2","objects":503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'13945","reported_seq":14123,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:36.874788+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:36.874788+0000","last_peered":"2024-09-16T10:29:36.874788+0000","last_clean":"2024-09-16T10:29:36.874788+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:36.874788+0000","last_undegraded":"2024-09-16T10:29:36.874788+0000","last_fullsized":"2024-09-16T10:29:36.874788+0000","mapping_epoch":33,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26398720,"num_objects":6445,"num_object_clones":0,"num_object_copies":12890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9748,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14075","reported_seq":22581,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.477939+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:40.477939+0000","last_peered":"2024-09-16T10:29:40.477939+0000","last_clean":"2024-09-16T10:29:40.477939+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:40.477939+0000","last_undegraded":"2024-09-16T10:29:40.477939+0000","last_fullsized":"2024-09-16T10:29:40.477939+0000","mapping_epoch":43,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26158022,"num_objects":6388,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14073,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14440","reported_seq":14685,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.947429+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:36.947429+0000","last_peered":"2024-09-16T10:29:36.947429+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:36.947429+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"39'4400","ondisk_log_start":"39'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27034566,"num_objects":6602,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14438,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2250},{"shards":"0,2","objects":518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'13990","reported_seq":14175,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.536348+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:36.536348+0000","last_peered":"2024-09-16T10:29:36.536348+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:36.536348+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26334150,"num_objects":6431,"num_object_clones":0,"num_object_copies":12862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13988,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":499},{"shards":"2","objects":2192}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14020","reported_seq":16089,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.626766+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:36.626766+0000","last_peered":"2024-09-16T10:29:36.626766+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:36.626766+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26854342,"num_objects":6558,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":6558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14018,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":991,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":341},{"shards":"2","objects":3353}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13686","reported_seq":13913,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.868242+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:36.868242+0000","last_peered":"2024-09-16T10:29:36.868242+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:36.868242+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3600","ondisk_log_start":"35'3600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26001891,"num_objects":6349,"num_object_clones":0,"num_object_copies":12698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13685,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2262},{"shards":"0,2","objects":491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14326","reported_seq":14582,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.867214+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:36.867214+0000","last_peered":"2024-09-16T10:29:36.867214+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:36.867214+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"39'4300","ondisk_log_start":"39'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26985897,"num_objects":6591,"num_object_clones":0,"num_object_copies":13182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14323,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2314},{"shards":"0,2","objects":472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13782","reported_seq":13996,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.946538+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:36.946538+0000","last_peered":"2024-09-16T10:29:36.946538+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:36.946538+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3700","ondisk_log_start":"35'3700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26043334,"num_objects":6360,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13780,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2258},{"shards":"0,2","objects":457}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14372","reported_seq":14600,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:36.935162+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:36.935162+0000","last_peered":"2024-09-16T10:29:36.935162+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:36.935162+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"39'4300","ondisk_log_start":"39'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26724236,"num_objects":6528,"num_object_clones":0,"num_object_copies":13056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4331,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14369,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2200},{"shards":"0,2","objects":477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":424862222,"num_objects":103743,"num_object_clones":0,"num_object_copies":207486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41014,"num_objects_misplaced":341,"num_objects_unfound":0,"num_objects_dirty":103743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202562,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9991,"num_bytes_recovered":38298083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1040359424,"data_stored":1040275407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160897,"ondisk_log_size":160897,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018966,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643956,"kb_used_data":345708,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93727884,"statfs":{"total":96636764160,"available":95977353216,"internally_reserved":0,"allocated":354004992,"data_stored":353736841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.1680000000000001},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.24299999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.375},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.28799999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691911,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607084,"kb_used_data":292452,"kb_used_omap":4,"kb_used_meta":314619,"kb_avail":93764756,"statfs":{"total":96636764160,"available":96015110144,"internally_reserved":0,"allocated":299470848,"data_stored":299137663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":322170538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":4.0220000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.0519999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":5.968},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.39500000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215193,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758312,"kb_used_data":379736,"kb_used_omap":4,"kb_used_meta":378555,"kb_avail":93613528,"statfs":{"total":96636764160,"available":95860252672,"internally_reserved":0,"allocated":388849664,"data_stored":388589199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":387641000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":7.726},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.5389999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":1.988},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":7.7679999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":388354048,"data_stored":388350435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298950656,"data_stored":298877937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":353054720,"data_stored":353047035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:42.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:42.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'14143", 'reported_seq': 14395, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.552721+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:36.552721+0000', 'last_peered': '2024-09-16T10:29:36.552721+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:36.552721+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "38'4100", 'ondisk_log_start': "38'4100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26705944, 'num_objects': 6521, 'num_object_clones': 0, 'num_object_copies': 13042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6521, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10200, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 526}, {'shards': '2', 'objects': 2218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:42.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:42.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'13892", 'reported_seq': 14197, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.940083+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:36.940083+0000', 'last_peered': '2024-09-16T10:29:36.940083+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:36.940083+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "35'3800", 'ondisk_log_start': "35'3800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26537984, 'num_objects': 6479, 'num_object_clones': 0, 'num_object_copies': 12958, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4150, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10517, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 69, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2331}, {'shards': '0,2', 'objects': 513}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:42.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:42.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'14363", 'reported_seq': 14754, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.805477+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:36.805477+0000', 'last_peered': '2024-09-16T10:29:36.805477+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:36.805477+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "38'4300", 'ondisk_log_start': "38'4300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26857472, 'num_objects': 6557, 'num_object_clones': 0, 'num_object_copies': 13114, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4298, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10605, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 107, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2261}, {'shards': '0,2', 'objects': 503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:42.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:42.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'14440", 'reported_seq': 14685, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.947429+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:36.947429+0000', 'last_peered': '2024-09-16T10:29:36.947429+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:36.947429+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "39'4400", 'ondisk_log_start': "39'4400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27034566, 'num_objects': 6602, 'num_object_clones': 0, 'num_object_copies': 13204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4354, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6602, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14438, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 84, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2250}, {'shards': '0,2', 'objects': 518}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:42.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:42.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'13990", 'reported_seq': 14175, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.536348+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:36.536348+0000', 'last_peered': '2024-09-16T10:29:36.536348+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:36.536348+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26334150, 'num_objects': 6431, 'num_object_clones': 0, 'num_object_copies': 12862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13988, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 499}, {'shards': '2', 'objects': 2192}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:42.384 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:42.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'14020", 'reported_seq': 16089, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.626766+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:29:36.626766+0000', 'last_peered': '2024-09-16T10:29:36.626766+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:36.626766+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "38'4000", 'ondisk_log_start': "38'4000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26854342, 'num_objects': 6558, 'num_object_clones': 0, 'num_object_copies': 13116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2864, 'num_objects_misplaced': 341, 'num_objects_unfound': 0, 'num_objects_dirty': 6558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14018, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 991, 'num_bytes_recovered': 3932160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 341}, {'shards': '2', 'objects': 3353}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:42.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:42.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'13686", 'reported_seq': 13913, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.868242+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:36.868242+0000', 'last_peered': '2024-09-16T10:29:36.868242+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:36.868242+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'3600", 'ondisk_log_start': "35'3600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26001891, 'num_objects': 6349, 'num_object_clones': 0, 'num_object_copies': 12698, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4088, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13685, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2262}, {'shards': '0,2', 'objects': 491}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:42.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:42.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'14326", 'reported_seq': 14582, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.867214+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:36.867214+0000', 'last_peered': '2024-09-16T10:29:36.867214+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:36.867214+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "39'4300", 'ondisk_log_start': "39'4300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26985897, 'num_objects': 6591, 'num_object_clones': 0, 'num_object_copies': 13182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4280, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14323, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 63, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2314}, {'shards': '0,2', 'objects': 472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:42.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:42.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'13782", 'reported_seq': 13996, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.946538+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:36.946538+0000', 'last_peered': '2024-09-16T10:29:36.946538+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:36.946538+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'3700", 'ondisk_log_start': "35'3700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26043334, 'num_objects': 6360, 'num_object_clones': 0, 'num_object_copies': 12720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4104, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6360, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13780, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 67, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2258}, {'shards': '0,2', 'objects': 457}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:42.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:42.385 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'14372", 'reported_seq': 14600, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:36.935162+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:36.935162+0000', 'last_peered': '2024-09-16T10:29:36.935162+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:36.935162+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "39'4300", 'ondisk_log_start': "39'4300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26724236, 'num_objects': 6528, 'num_object_clones': 0, 'num_object_copies': 13056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4331, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14369, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 72, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2200}, {'shards': '0,2', 'objects': 477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:42.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:43.096 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:43.096 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 41014/207490 objects degraded (19.767%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":471,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":103745,"data_bytes":424862241,"bytes_used":2057576448,"bytes_avail":287852716032,"bytes_total":289910292480,"degraded_objects":41014,"degraded_total":207490,"degraded_ratio":0.19766735746301026,"misplaced_objects":341,"misplaced_total":207490,"misplaced_ratio":0.0016434526965154947,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":84600,"recovering_keys_per_sec":0,"num_objects_recovered":284,"num_bytes_recovered":1015808,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":371},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T10:29:36.693808+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:29:43.121 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:43.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:43.665 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:43.666 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:43.688 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-16T10:29:42.696005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416670241,"num_objects":101745,"num_object_clones":0,"num_object_copies":203490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40988,"num_objects_misplaced":198,"num_objects_unfound":0,"num_objects_dirty":101745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204564,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10161,"num_bytes_recovered":38883830,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2006820,"kb_used_data":1007180,"kb_used_omap":12,"kb_used_meta":999603,"kb_avail":281108700,"statfs":{"total":289910292480,"available":287855308800,"internally_reserved":0,"allocated":1031352320,"data_stored":1030490519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1023593474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,2,4,5,3,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25944064,"num_objects":-6334,"num_object_clones":0,"num_object_copies":-12668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-62,"num_objects_misplaced":-391,"num_objects_unfound":0,"num_objects_dirty":-6334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6334,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":453,"num_bytes_recovered":1601536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007407"},"pg_stats":[{"pgid":"2.f","version":"46'14091","reported_seq":22556,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.832305+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:41.832305+0000","last_peered":"2024-09-16T10:29:41.832305+0000","last_clean":"2024-09-16T10:29:41.832305+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:41.832305+0000","last_undegraded":"2024-09-16T10:29:41.832305+0000","last_fullsized":"2024-09-16T10:29:41.832305+0000","mapping_epoch":45,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26120192,"num_objects":6377,"num_object_clones":0,"num_object_copies":12754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14091,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14100","reported_seq":14260,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.664059+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:41.664059+0000","last_peered":"2024-09-16T10:29:41.664059+0000","last_clean":"2024-09-16T10:29:41.664059+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:41.664059+0000","last_undegraded":"2024-09-16T10:29:41.664059+0000","last_fullsized":"2024-09-16T10:29:41.664059+0000","mapping_epoch":34,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26058752,"num_objects":6362,"num_object_clones":0,"num_object_copies":12724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10313,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14304","reported_seq":14554,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.690897+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:41.690897+0000","last_peered":"2024-09-16T10:29:41.690897+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:41.690897+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"39'4300","ondisk_log_start":"39'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26046488,"num_objects":6360,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6360,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10361,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":365},{"shards":"2","objects":2057}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14140","reported_seq":14198,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.706854+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:41.706854+0000","last_peered":"2024-09-16T10:29:41.706854+0000","last_clean":"2024-09-16T10:29:41.706854+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:41.706854+0000","last_undegraded":"2024-09-16T10:29:41.706854+0000","last_fullsized":"2024-09-16T10:29:41.706854+0000","mapping_epoch":34,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25894912,"num_objects":6322,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14140,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14150","reported_seq":14310,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.916968+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:41.916968+0000","last_peered":"2024-09-16T10:29:41.916968+0000","last_clean":"2024-09-16T10:29:41.916968+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:41.916968+0000","last_undegraded":"2024-09-16T10:29:41.916968+0000","last_fullsized":"2024-09-16T10:29:41.916968+0000","mapping_epoch":33,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26030080,"num_objects":6355,"num_object_clones":0,"num_object_copies":12710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14025","reported_seq":14340,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.986016+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:41.986016+0000","last_peered":"2024-09-16T10:29:41.986016+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:41.986016+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25993216,"num_objects":6346,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10650,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2204},{"shards":"0,2","objects":380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14491","reported_seq":14886,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.890253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:41.890253+0000","last_peered":"2024-09-16T10:29:41.890253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:41.890253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"39'4400","ondisk_log_start":"39'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26333184,"num_objects":6429,"num_object_clones":0,"num_object_copies":12858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2135},{"shards":"0,2","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14090","reported_seq":14268,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.889123+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:41.889123+0000","last_peered":"2024-09-16T10:29:41.889123+0000","last_clean":"2024-09-16T10:29:41.889123+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:41.889123+0000","last_undegraded":"2024-09-16T10:29:41.889123+0000","last_fullsized":"2024-09-16T10:29:41.889123+0000","mapping_epoch":33,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25804800,"num_objects":6300,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9893,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14075","reported_seq":22581,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:40.477939+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:40.477939+0000","last_peered":"2024-09-16T10:29:40.477939+0000","last_clean":"2024-09-16T10:29:40.477939+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:40.477939+0000","last_undegraded":"2024-09-16T10:29:40.477939+0000","last_fullsized":"2024-09-16T10:29:40.477939+0000","mapping_epoch":43,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26158022,"num_objects":6388,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14073,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14576","reported_seq":14838,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.986749+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:41.986749+0000","last_peered":"2024-09-16T10:29:41.986749+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:41.986749+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26477510,"num_objects":6466,"num_object_clones":0,"num_object_copies":12932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14574,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2122},{"shards":"0,2","objects":382}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14131","reported_seq":14316,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.679603+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:41.679603+0000","last_peered":"2024-09-16T10:29:41.679603+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:41.679603+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25756614,"num_objects":6290,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14129,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":358},{"shards":"2","objects":2051}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14140","reported_seq":16494,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.709666+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:41.709666+0000","last_peered":"2024-09-16T10:29:41.709666+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:41.709666+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"39'4100","ondisk_log_start":"39'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26362822,"num_objects":6438,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":198,"num_objects_unfound":0,"num_objects_dirty":6438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14138,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1134,"num_bytes_recovered":4517888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":198},{"shards":"2","objects":3376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13816","reported_seq":14047,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.980145+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:41.980145+0000","last_peered":"2024-09-16T10:29:41.980145+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:41.980145+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25469411,"num_objects":6219,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13815,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2134},{"shards":"0,2","objects":361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14475","reported_seq":14735,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.954652+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:41.954652+0000","last_peered":"2024-09-16T10:29:41.954652+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:41.954652+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26375593,"num_objects":6442,"num_object_clones":0,"num_object_copies":12884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14472,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2167},{"shards":"0,2","objects":323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13903","reported_seq":14125,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.787052+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:41.787052+0000","last_peered":"2024-09-16T10:29:41.787052+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:41.787052+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25547718,"num_objects":6239,"num_object_clones":0,"num_object_copies":12478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13901,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2141},{"shards":"0,2","objects":336}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14490","reported_seq":14724,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.889689+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:41.889689+0000","last_peered":"2024-09-16T10:29:41.889689+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:41.889689+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"39'4400","ondisk_log_start":"39'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26240908,"num_objects":6410,"num_object_clones":0,"num_object_copies":12820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14487,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2084},{"shards":"0,2","objects":359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":416670222,"num_objects":101743,"num_object_clones":0,"num_object_copies":203486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40988,"num_objects_misplaced":198,"num_objects_unfound":0,"num_objects_dirty":101743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204562,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10160,"num_bytes_recovered":38883811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1029431296,"data_stored":1029347279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160897,"ondisk_log_size":160897,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018967,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643656,"kb_used_data":341312,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93728184,"statfs":{"total":96636764160,"available":95977660416,"internally_reserved":0,"allocated":349503488,"data_stored":349235337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":5.1260000000000003},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.2859999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691911,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607084,"kb_used_data":292452,"kb_used_omap":4,"kb_used_meta":314619,"kb_avail":93764756,"statfs":{"total":96636764160,"available":96015110144,"internally_reserved":0,"allocated":299470848,"data_stored":299137663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":322170538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":4.0220000000000002},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.0519999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":5.968},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.39500000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215194,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756080,"kb_used_data":373416,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93615760,"statfs":{"total":96636764160,"available":95862538240,"internally_reserved":0,"allocated":382377984,"data_stored":382117519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":391835304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.331}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.88700000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.45000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":381923328,"data_stored":381919715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298950656,"data_stored":298877937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":348557312,"data_stored":348549627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:46.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:47.252 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:47.254 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:47.284 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-16T10:29:46.697529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416170529,"num_objects":101623,"num_object_clones":0,"num_object_copies":203246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40988,"num_objects_misplaced":198,"num_objects_unfound":0,"num_objects_dirty":101623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204686,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10161,"num_bytes_recovered":38883830,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160922,"ondisk_log_size":160922,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1993696,"kb_used_data":1006344,"kb_used_omap":12,"kb_used_meta":987315,"kb_avail":281121824,"statfs":{"total":289910292480,"available":287868747776,"internally_reserved":0,"allocated":1030496256,"data_stored":1029634455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1011010562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,4,6,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18219008,"num_objects":-4448,"num_object_clones":0,"num_object_copies":-8896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-44,"num_objects_misplaced":-265,"num_objects_unfound":0,"num_objects_dirty":-4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4448,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":1085440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004508"},"pg_stats":[{"pgid":"2.f","version":"46'14091","reported_seq":22556,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.832305+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:41.832305+0000","last_peered":"2024-09-16T10:29:41.832305+0000","last_clean":"2024-09-16T10:29:41.832305+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:41.832305+0000","last_undegraded":"2024-09-16T10:29:41.832305+0000","last_fullsized":"2024-09-16T10:29:41.832305+0000","mapping_epoch":45,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26120192,"num_objects":6377,"num_object_clones":0,"num_object_copies":12754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14091,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14100","reported_seq":14260,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.664059+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:41.664059+0000","last_peered":"2024-09-16T10:29:41.664059+0000","last_clean":"2024-09-16T10:29:41.664059+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:41.664059+0000","last_undegraded":"2024-09-16T10:29:41.664059+0000","last_fullsized":"2024-09-16T10:29:41.664059+0000","mapping_epoch":34,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26058752,"num_objects":6362,"num_object_clones":0,"num_object_copies":12724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10313,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14304","reported_seq":14554,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.690897+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:41.690897+0000","last_peered":"2024-09-16T10:29:41.690897+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:41.690897+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"39'4300","ondisk_log_start":"39'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26046488,"num_objects":6360,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6360,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10361,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":365},{"shards":"2","objects":2057}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14140","reported_seq":14198,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.706854+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:41.706854+0000","last_peered":"2024-09-16T10:29:41.706854+0000","last_clean":"2024-09-16T10:29:41.706854+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:41.706854+0000","last_undegraded":"2024-09-16T10:29:41.706854+0000","last_fullsized":"2024-09-16T10:29:41.706854+0000","mapping_epoch":34,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25894912,"num_objects":6322,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14140,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14150","reported_seq":14310,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.916968+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:41.916968+0000","last_peered":"2024-09-16T10:29:41.916968+0000","last_clean":"2024-09-16T10:29:41.916968+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:41.916968+0000","last_undegraded":"2024-09-16T10:29:41.916968+0000","last_fullsized":"2024-09-16T10:29:41.916968+0000","mapping_epoch":33,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26030080,"num_objects":6355,"num_object_clones":0,"num_object_copies":12710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14025","reported_seq":14340,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.986016+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:41.986016+0000","last_peered":"2024-09-16T10:29:41.986016+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:41.986016+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25993216,"num_objects":6346,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10650,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2204},{"shards":"0,2","objects":380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14491","reported_seq":14886,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.890253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:41.890253+0000","last_peered":"2024-09-16T10:29:41.890253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:41.890253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"39'4400","ondisk_log_start":"39'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26333184,"num_objects":6429,"num_object_clones":0,"num_object_copies":12858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2135},{"shards":"0,2","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14090","reported_seq":14268,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.889123+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:41.889123+0000","last_peered":"2024-09-16T10:29:41.889123+0000","last_clean":"2024-09-16T10:29:41.889123+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:41.889123+0000","last_undegraded":"2024-09-16T10:29:41.889123+0000","last_fullsized":"2024-09-16T10:29:41.889123+0000","mapping_epoch":33,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25804800,"num_objects":6300,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9893,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14197","reported_seq":22703,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:45.537072+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:45.537072+0000","last_peered":"2024-09-16T10:29:45.537072+0000","last_clean":"2024-09-16T10:29:45.537072+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:45.537072+0000","last_undegraded":"2024-09-16T10:29:45.537072+0000","last_fullsized":"2024-09-16T10:29:45.537072+0000","mapping_epoch":43,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25658310,"num_objects":6266,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14195,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14576","reported_seq":14838,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.986749+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:41.986749+0000","last_peered":"2024-09-16T10:29:41.986749+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:41.986749+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26477510,"num_objects":6466,"num_object_clones":0,"num_object_copies":12932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14574,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2122},{"shards":"0,2","objects":382}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14131","reported_seq":14316,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.679603+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:41.679603+0000","last_peered":"2024-09-16T10:29:41.679603+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:41.679603+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25756614,"num_objects":6290,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14129,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":358},{"shards":"2","objects":2051}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14140","reported_seq":16494,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.709666+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:41.709666+0000","last_peered":"2024-09-16T10:29:41.709666+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:41.709666+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"39'4100","ondisk_log_start":"39'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26362822,"num_objects":6438,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":198,"num_objects_unfound":0,"num_objects_dirty":6438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14138,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1134,"num_bytes_recovered":4517888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":198},{"shards":"2","objects":3376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13816","reported_seq":14047,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.980145+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:41.980145+0000","last_peered":"2024-09-16T10:29:41.980145+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:41.980145+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25469411,"num_objects":6219,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13815,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2134},{"shards":"0,2","objects":361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14475","reported_seq":14735,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.954652+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:41.954652+0000","last_peered":"2024-09-16T10:29:41.954652+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:41.954652+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26375593,"num_objects":6442,"num_object_clones":0,"num_object_copies":12884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14472,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2167},{"shards":"0,2","objects":323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13903","reported_seq":14125,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.787052+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:41.787052+0000","last_peered":"2024-09-16T10:29:41.787052+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:41.787052+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25547718,"num_objects":6239,"num_object_clones":0,"num_object_copies":12478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13901,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2141},{"shards":"0,2","objects":336}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14490","reported_seq":14724,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.889689+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:41.889689+0000","last_peered":"2024-09-16T10:29:41.889689+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:41.889689+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"39'4400","ondisk_log_start":"39'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26240908,"num_objects":6410,"num_object_clones":0,"num_object_copies":12820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14487,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2084},{"shards":"0,2","objects":359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":416170510,"num_objects":101621,"num_object_clones":0,"num_object_copies":203242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40988,"num_objects_misplaced":198,"num_objects_unfound":0,"num_objects_dirty":101621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204684,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10160,"num_bytes_recovered":38883811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1028476928,"data_stored":1028392911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160919,"ondisk_log_size":160919,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018967,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643656,"kb_used_data":341312,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93728184,"statfs":{"total":96636764160,"available":95977660416,"internally_reserved":0,"allocated":349503488,"data_stored":349235337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":5.1260000000000003},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.2859999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691912,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593960,"kb_used_data":291616,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93777880,"statfs":{"total":96636764160,"available":96028549120,"internally_reserved":0,"allocated":298614784,"data_stored":298281599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":309587626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":5.3150000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.625},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":7.9189999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215194,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756080,"kb_used_data":373416,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93615760,"statfs":{"total":96636764160,"available":95862538240,"internally_reserved":0,"allocated":382377984,"data_stored":382117519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":391835304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.331}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.88700000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.45000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":381923328,"data_stored":381919715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297996288,"data_stored":297923569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":348557312,"data_stored":348549627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:47.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:47.836 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:47.837 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:47.861 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-16T10:29:46.697529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416170529,"num_objects":101623,"num_object_clones":0,"num_object_copies":203246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40988,"num_objects_misplaced":198,"num_objects_unfound":0,"num_objects_dirty":101623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204686,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10161,"num_bytes_recovered":38883830,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160922,"ondisk_log_size":160922,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1993696,"kb_used_data":1006344,"kb_used_omap":12,"kb_used_meta":987315,"kb_avail":281121824,"statfs":{"total":289910292480,"available":287868747776,"internally_reserved":0,"allocated":1030496256,"data_stored":1029634455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1011010562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,4,6,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18219008,"num_objects":-4448,"num_object_clones":0,"num_object_copies":-8896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-44,"num_objects_misplaced":-265,"num_objects_unfound":0,"num_objects_dirty":-4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4448,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":1085440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004508"},"pg_stats":[{"pgid":"2.f","version":"46'14091","reported_seq":22556,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.832305+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:41.832305+0000","last_peered":"2024-09-16T10:29:41.832305+0000","last_clean":"2024-09-16T10:29:41.832305+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:41.832305+0000","last_undegraded":"2024-09-16T10:29:41.832305+0000","last_fullsized":"2024-09-16T10:29:41.832305+0000","mapping_epoch":45,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26120192,"num_objects":6377,"num_object_clones":0,"num_object_copies":12754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14091,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14100","reported_seq":14260,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.664059+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:41.664059+0000","last_peered":"2024-09-16T10:29:41.664059+0000","last_clean":"2024-09-16T10:29:41.664059+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:41.664059+0000","last_undegraded":"2024-09-16T10:29:41.664059+0000","last_fullsized":"2024-09-16T10:29:41.664059+0000","mapping_epoch":34,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26058752,"num_objects":6362,"num_object_clones":0,"num_object_copies":12724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10313,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14304","reported_seq":14554,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.690897+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:41.690897+0000","last_peered":"2024-09-16T10:29:41.690897+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:41.690897+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"39'4300","ondisk_log_start":"39'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26046488,"num_objects":6360,"num_object_clones":0,"num_object_copies":12720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6360,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10361,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":365},{"shards":"2","objects":2057}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14140","reported_seq":14198,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.706854+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:41.706854+0000","last_peered":"2024-09-16T10:29:41.706854+0000","last_clean":"2024-09-16T10:29:41.706854+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:41.706854+0000","last_undegraded":"2024-09-16T10:29:41.706854+0000","last_fullsized":"2024-09-16T10:29:41.706854+0000","mapping_epoch":34,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25894912,"num_objects":6322,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14140,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14150","reported_seq":14310,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.916968+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:41.916968+0000","last_peered":"2024-09-16T10:29:41.916968+0000","last_clean":"2024-09-16T10:29:41.916968+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:41.916968+0000","last_undegraded":"2024-09-16T10:29:41.916968+0000","last_fullsized":"2024-09-16T10:29:41.916968+0000","mapping_epoch":33,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26030080,"num_objects":6355,"num_object_clones":0,"num_object_copies":12710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14025","reported_seq":14340,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.986016+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:41.986016+0000","last_peered":"2024-09-16T10:29:41.986016+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:41.986016+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25993216,"num_objects":6346,"num_object_clones":0,"num_object_copies":12692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10650,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2204},{"shards":"0,2","objects":380}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14491","reported_seq":14886,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.890253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:41.890253+0000","last_peered":"2024-09-16T10:29:41.890253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:41.890253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"39'4400","ondisk_log_start":"39'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26333184,"num_objects":6429,"num_object_clones":0,"num_object_copies":12858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4296,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10733,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2135},{"shards":"0,2","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14090","reported_seq":14268,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:41.889123+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:41.889123+0000","last_peered":"2024-09-16T10:29:41.889123+0000","last_clean":"2024-09-16T10:29:41.889123+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:41.889123+0000","last_undegraded":"2024-09-16T10:29:41.889123+0000","last_fullsized":"2024-09-16T10:29:41.889123+0000","mapping_epoch":33,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25804800,"num_objects":6300,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9893,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14197","reported_seq":22703,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:45.537072+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:45.537072+0000","last_peered":"2024-09-16T10:29:45.537072+0000","last_clean":"2024-09-16T10:29:45.537072+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:45.537072+0000","last_undegraded":"2024-09-16T10:29:45.537072+0000","last_fullsized":"2024-09-16T10:29:45.537072+0000","mapping_epoch":43,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25658310,"num_objects":6266,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14195,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14576","reported_seq":14838,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.986749+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:41.986749+0000","last_peered":"2024-09-16T10:29:41.986749+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:41.986749+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26477510,"num_objects":6466,"num_object_clones":0,"num_object_copies":12932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14574,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2122},{"shards":"0,2","objects":382}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14131","reported_seq":14316,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.679603+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:41.679603+0000","last_peered":"2024-09-16T10:29:41.679603+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:41.679603+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25756614,"num_objects":6290,"num_object_clones":0,"num_object_copies":12580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14129,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":358},{"shards":"2","objects":2051}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14140","reported_seq":16494,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.709666+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:41.709666+0000","last_peered":"2024-09-16T10:29:41.709666+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:41.709666+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"39'4100","ondisk_log_start":"39'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26362822,"num_objects":6438,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":198,"num_objects_unfound":0,"num_objects_dirty":6438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14138,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1134,"num_bytes_recovered":4517888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":198},{"shards":"2","objects":3376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13816","reported_seq":14047,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.980145+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:41.980145+0000","last_peered":"2024-09-16T10:29:41.980145+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:41.980145+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"35'3800","ondisk_log_start":"35'3800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25469411,"num_objects":6219,"num_object_clones":0,"num_object_copies":12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13815,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2134},{"shards":"0,2","objects":361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14475","reported_seq":14735,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.954652+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:41.954652+0000","last_peered":"2024-09-16T10:29:41.954652+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:41.954652+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26375593,"num_objects":6442,"num_object_clones":0,"num_object_copies":12884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4278,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14472,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2167},{"shards":"0,2","objects":323}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'13903","reported_seq":14125,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.787052+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:41.787052+0000","last_peered":"2024-09-16T10:29:41.787052+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:41.787052+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"35'3900","ondisk_log_start":"35'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25547718,"num_objects":6239,"num_object_clones":0,"num_object_copies":12478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13901,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2141},{"shards":"0,2","objects":336}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14490","reported_seq":14724,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:41.889689+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:41.889689+0000","last_peered":"2024-09-16T10:29:41.889689+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:41.889689+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"39'4400","ondisk_log_start":"39'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26240908,"num_objects":6410,"num_object_clones":0,"num_object_copies":12820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4329,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14487,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2084},{"shards":"0,2","objects":359}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":416170510,"num_objects":101621,"num_object_clones":0,"num_object_copies":203242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40988,"num_objects_misplaced":198,"num_objects_unfound":0,"num_objects_dirty":101621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204684,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10160,"num_bytes_recovered":38883811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1028476928,"data_stored":1028392911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160919,"ondisk_log_size":160919,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018967,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643656,"kb_used_data":341312,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93728184,"statfs":{"total":96636764160,"available":95977660416,"internally_reserved":0,"allocated":349503488,"data_stored":349235337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":5.1260000000000003},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.2859999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691912,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593960,"kb_used_data":291616,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93777880,"statfs":{"total":96636764160,"available":96028549120,"internally_reserved":0,"allocated":298614784,"data_stored":298281599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":309587626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":5.3150000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.625},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":7.9189999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215194,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756080,"kb_used_data":373416,"kb_used_omap":4,"kb_used_meta":382651,"kb_avail":93615760,"statfs":{"total":96636764160,"available":95862538240,"internally_reserved":0,"allocated":382377984,"data_stored":382117519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":391835304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,2,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.331}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.88700000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.45000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":381923328,"data_stored":381919715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297996288,"data_stored":297923569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":348557312,"data_stored":348549627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:47.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:47.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'14304", 'reported_seq': 14554, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:41.690897+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:41.690897+0000', 'last_peered': '2024-09-16T10:29:41.690897+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:41.690897+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "39'4300", 'ondisk_log_start': "39'4300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26046488, 'num_objects': 6360, 'num_object_clones': 0, 'num_object_copies': 12720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6360, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10361, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 365}, {'shards': '2', 'objects': 2057}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:47.862 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:47.863 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'14025", 'reported_seq': 14340, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:41.986016+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:41.986016+0000', 'last_peered': '2024-09-16T10:29:41.986016+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:41.986016+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "38'4000", 'ondisk_log_start': "38'4000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25993216, 'num_objects': 6346, 'num_object_clones': 0, 'num_object_copies': 12692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4144, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10650, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 75, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2204}, {'shards': '0,2', 'objects': 380}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:47.863 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:47.863 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'14491", 'reported_seq': 14886, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:41.890253+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:41.890253+0000', 'last_peered': '2024-09-16T10:29:41.890253+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:41.890253+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "39'4400", 'ondisk_log_start': "39'4400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26333184, 'num_objects': 6429, 'num_object_clones': 0, 'num_object_copies': 12858, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4296, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10733, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 109, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2135}, {'shards': '0,2', 'objects': 375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:47.863 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:47.863 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'14576", 'reported_seq': 14838, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:41.986749+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:41.986749+0000', 'last_peered': '2024-09-16T10:29:41.986749+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:41.986749+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'4500", 'ondisk_log_start': "42'4500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26477510, 'num_objects': 6466, 'num_object_clones': 0, 'num_object_copies': 12932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4346, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14574, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 92, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2122}, {'shards': '0,2', 'objects': 382}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:47.863 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:47.863 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'14131", 'reported_seq': 14316, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:41.679603+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:41.679603+0000', 'last_peered': '2024-09-16T10:29:41.679603+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:41.679603+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "38'4100", 'ondisk_log_start': "38'4100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25756614, 'num_objects': 6290, 'num_object_clones': 0, 'num_object_copies': 12580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14129, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 40, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 358}, {'shards': '2', 'objects': 2051}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:47.863 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:47.863 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'14140", 'reported_seq': 16494, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:41.709666+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:29:41.709666+0000', 'last_peered': '2024-09-16T10:29:41.709666+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:41.709666+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "39'4100", 'ondisk_log_start': "39'4100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26362822, 'num_objects': 6438, 'num_object_clones': 0, 'num_object_copies': 12876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2864, 'num_objects_misplaced': 198, 'num_objects_unfound': 0, 'num_objects_dirty': 6438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14138, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1134, 'num_bytes_recovered': 4517888, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 198}, {'shards': '2', 'objects': 3376}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:47.863 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:47.864 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'13816", 'reported_seq': 14047, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:41.980145+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:41.980145+0000', 'last_peered': '2024-09-16T10:29:41.980145+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:41.980145+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "35'3800", 'ondisk_log_start': "35'3800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25469411, 'num_objects': 6219, 'num_object_clones': 0, 'num_object_copies': 12438, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4086, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13815, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 75, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2134}, {'shards': '0,2', 'objects': 361}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:47.864 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:47.864 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'14475", 'reported_seq': 14735, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:41.954652+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:41.954652+0000', 'last_peered': '2024-09-16T10:29:41.954652+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:41.954652+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'4400", 'ondisk_log_start': "42'4400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26375593, 'num_objects': 6442, 'num_object_clones': 0, 'num_object_copies': 12884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4278, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14472, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 65, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2167}, {'shards': '0,2', 'objects': 323}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:47.864 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:47.864 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'13903", 'reported_seq': 14125, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:41.787052+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:41.787052+0000', 'last_peered': '2024-09-16T10:29:41.787052+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:41.787052+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "35'3900", 'ondisk_log_start': "35'3900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25547718, 'num_objects': 6239, 'num_object_clones': 0, 'num_object_copies': 12478, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4100, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6239, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13901, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2141}, {'shards': '0,2', 'objects': 336}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:47.864 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:47.864 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'14490", 'reported_seq': 14724, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:41.889689+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:41.889689+0000', 'last_peered': '2024-09-16T10:29:41.889689+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:41.889689+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "39'4400", 'ondisk_log_start': "39'4400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26240908, 'num_objects': 6410, 'num_object_clones': 0, 'num_object_copies': 12820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4329, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14487, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 74, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 2084}, {'shards': '0,2', 'objects': 359}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:47.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:48.517 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:48.517 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 40988/203246 objects degraded (20.167%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":477,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":101623,"data_bytes":416170529,"bytes_used":2041544704,"bytes_avail":287868747776,"bytes_total":289910292480,"degraded_objects":40988,"degraded_total":203246,"degraded_ratio":0.20166694547494168,"misplaced_objects":198,"misplaced_total":203246,"misplaced_ratio":0.00097418891392696532,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":90419,"recovering_keys_per_sec":0,"num_objects_recovered":309,"num_bytes_recovered":1085440,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":370},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T10:29:36.693808+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:29:48.544 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:48.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:49.081 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:49.083 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:49.104 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-09-16T10:29:48.698481+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":408023585,"num_objects":99634,"num_object_clones":0,"num_object_copies":199268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40965,"num_objects_misplaced":74,"num_objects_unfound":0,"num_objects_dirty":99634,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206675,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10308,"num_bytes_recovered":39391734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160611,"ondisk_log_size":160611,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1974692,"kb_used_data":995512,"kb_used_omap":12,"kb_used_meta":979123,"kb_avail":281140828,"statfs":{"total":289910292480,"available":287888207872,"internally_reserved":0,"allocated":1019404288,"data_stored":1018542487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1002621954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,9,6,5,1,0,0,1,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23040000,"num_objects":-5625,"num_object_clones":0,"num_object_copies":-11250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-59,"num_objects_misplaced":-267,"num_objects_unfound":0,"num_objects_dirty":-5625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5625,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":1093632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004771"},"pg_stats":[{"pgid":"2.f","version":"46'14209","reported_seq":22674,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:46.926073+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:46.926073+0000","last_peered":"2024-09-16T10:29:46.926073+0000","last_clean":"2024-09-16T10:29:46.926073+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:46.926073+0000","last_undegraded":"2024-09-16T10:29:46.926073+0000","last_fullsized":"2024-09-16T10:29:46.926073+0000","mapping_epoch":45,"log_start":"42'4200","ondisk_log_start":"42'4200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25636864,"num_objects":6259,"num_object_clones":0,"num_object_copies":12518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14209,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14254","reported_seq":14414,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:46.716051+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:46.716051+0000","last_peered":"2024-09-16T10:29:46.716051+0000","last_clean":"2024-09-16T10:29:46.716051+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:46.716051+0000","last_undegraded":"2024-09-16T10:29:46.716051+0000","last_fullsized":"2024-09-16T10:29:46.716051+0000","mapping_epoch":34,"log_start":"39'4200","ondisk_log_start":"39'4200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":6208,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10467,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14430","reported_seq":14683,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.669774+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:46.669774+0000","last_peered":"2024-09-16T10:29:46.669774+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:46.669774+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25530392,"num_objects":6234,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10487,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":239},{"shards":"2","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14301","reported_seq":14361,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:46.742105+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:46.742105+0000","last_peered":"2024-09-16T10:29:46.742105+0000","last_clean":"2024-09-16T10:29:46.742105+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:46.742105+0000","last_undegraded":"2024-09-16T10:29:46.742105+0000","last_fullsized":"2024-09-16T10:29:46.742105+0000","mapping_epoch":34,"log_start":"39'4300","ondisk_log_start":"39'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25235456,"num_objects":6161,"num_object_clones":0,"num_object_copies":12322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14301,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14303","reported_seq":14463,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:46.975907+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:46.975907+0000","last_peered":"2024-09-16T10:29:46.975907+0000","last_clean":"2024-09-16T10:29:46.975907+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:46.975907+0000","last_undegraded":"2024-09-16T10:29:46.975907+0000","last_fullsized":"2024-09-16T10:29:46.975907+0000","mapping_epoch":33,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25403392,"num_objects":6202,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14156","reported_seq":14477,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.976445+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:46.976445+0000","last_peered":"2024-09-16T10:29:46.976445+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:46.976445+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"39'4100","ondisk_log_start":"39'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25456640,"num_objects":6215,"num_object_clones":0,"num_object_copies":12430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10781,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2075},{"shards":"0,2","objects":249}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14628","reported_seq":15027,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.816896+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:46.816896+0000","last_peered":"2024-09-16T10:29:46.816896+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:46.816896+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25772032,"num_objects":6292,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10870,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2000},{"shards":"0,2","objects":238}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14213","reported_seq":14391,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:46.928418+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:46.928418+0000","last_peered":"2024-09-16T10:29:46.928418+0000","last_clean":"2024-09-16T10:29:46.928418+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:46.928418+0000","last_undegraded":"2024-09-16T10:29:46.928418+0000","last_fullsized":"2024-09-16T10:29:46.928418+0000","mapping_epoch":33,"log_start":"39'4200","ondisk_log_start":"39'4200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25300992,"num_objects":6177,"num_object_clones":0,"num_object_copies":12354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14197","reported_seq":22703,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:45.537072+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:45.537072+0000","last_peered":"2024-09-16T10:29:45.537072+0000","last_clean":"2024-09-16T10:29:45.537072+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:45.537072+0000","last_undegraded":"2024-09-16T10:29:45.537072+0000","last_fullsized":"2024-09-16T10:29:45.537072+0000","mapping_epoch":43,"log_start":"38'4100","ondisk_log_start":"38'4100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25658310,"num_objects":6266,"num_object_clones":0,"num_object_copies":12532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14195,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14711","reported_seq":14981,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.903765+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:46.903765+0000","last_peered":"2024-09-16T10:29:46.903765+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:46.903765+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25924550,"num_objects":6331,"num_object_clones":0,"num_object_copies":12662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14709,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1991},{"shards":"0,2","objects":247}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14260","reported_seq":14449,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.638412+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:46.638412+0000","last_peered":"2024-09-16T10:29:46.638412+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:46.638412+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"39'4200","ondisk_log_start":"39'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228230,"num_objects":6161,"num_object_clones":0,"num_object_copies":12322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14258,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":229},{"shards":"2","objects":1924}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14256","reported_seq":16859,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.715218+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:46.715218+0000","last_peered":"2024-09-16T10:29:46.715218+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:46.715218+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4200","ondisk_log_start":"42'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25887686,"num_objects":6322,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":74,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14254,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":5025792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":74},{"shards":"2","objects":3384}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13941","reported_seq":14182,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.957829+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:46.957829+0000","last_peered":"2024-09-16T10:29:46.957829+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:46.957829+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"38'3900","ondisk_log_start":"38'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24957411,"num_objects":6094,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13940,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2014},{"shards":"0,2","objects":236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14590","reported_seq":14858,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.936351+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:46.936351+0000","last_peered":"2024-09-16T10:29:46.936351+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:46.936351+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25904553,"num_objects":6327,"num_object_clones":0,"num_object_copies":12654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14587,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2056},{"shards":"0,2","objects":208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14038","reported_seq":14264,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.837294+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:46.837294+0000","last_peered":"2024-09-16T10:29:46.837294+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:46.837294+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24994758,"num_objects":6104,"num_object_clones":0,"num_object_copies":12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14036,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2008},{"shards":"0,2","objects":201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14621","reported_seq":14859,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.929281+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:46.929281+0000","last_peered":"2024-09-16T10:29:46.929281+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:46.929281+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25704332,"num_objects":6279,"num_object_clones":0,"num_object_copies":12558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14618,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1955},{"shards":"0,2","objects":228}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":408023566,"num_objects":99632,"num_object_clones":0,"num_object_copies":199264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40965,"num_objects_misplaced":74,"num_objects_unfound":0,"num_objects_dirty":99632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206673,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10307,"num_bytes_recovered":39391715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1017323520,"data_stored":1017239503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160608,"ondisk_log_size":160608,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018968,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643432,"kb_used_data":336980,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93728408,"statfs":{"total":96636764160,"available":95977889792,"internally_reserved":0,"allocated":345067520,"data_stored":344799369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":313781936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,1,1,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.956},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":13.917}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":13.973000000000001},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":17.219999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691912,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":593960,"kb_used_data":291616,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93777880,"statfs":{"total":96636764160,"available":96028549120,"internally_reserved":0,"allocated":298614784,"data_stored":298281599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":309587626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":5.3150000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.625},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":7.9189999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215195,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737300,"kb_used_data":366916,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93634540,"statfs":{"total":96636764160,"available":95881768960,"internally_reserved":0,"allocated":375721984,"data_stored":375461519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,2,4,1,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.3319999999999999},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.3370000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":2.5390000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.0230000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":375242752,"data_stored":375239139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297996288,"data_stored":297923569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":344084480,"data_stored":344076795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:52.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:52.663 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:52.668 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:52.693 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-16T10:29:50.699246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":407532065,"num_objects":99514,"num_object_clones":0,"num_object_copies":199028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40965,"num_objects_misplaced":74,"num_objects_unfound":0,"num_objects_dirty":99514,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206795,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10308,"num_bytes_recovered":39391734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160531,"ondisk_log_size":160531,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1977820,"kb_used_data":994544,"kb_used_omap":12,"kb_used_meta":983219,"kb_avail":281137700,"statfs":{"total":289910292480,"available":287885004800,"internally_reserved":0,"allocated":1018413056,"data_stored":1017551255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1006816258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,3,10,8,5,1,0,0,1,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17858560,"num_objects":-4360,"num_object_clones":0,"num_object_copies":-8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-49,"num_objects_misplaced":-267,"num_objects_unfound":0,"num_objects_dirty":-4360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4360,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":1093632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005012"},"pg_stats":[{"pgid":"2.f","version":"46'14209","reported_seq":22674,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:46.926073+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:46.926073+0000","last_peered":"2024-09-16T10:29:46.926073+0000","last_clean":"2024-09-16T10:29:46.926073+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:46.926073+0000","last_undegraded":"2024-09-16T10:29:46.926073+0000","last_fullsized":"2024-09-16T10:29:46.926073+0000","mapping_epoch":45,"log_start":"42'4200","ondisk_log_start":"42'4200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25636864,"num_objects":6259,"num_object_clones":0,"num_object_copies":12518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14209,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14254","reported_seq":14414,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:46.716051+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:46.716051+0000","last_peered":"2024-09-16T10:29:46.716051+0000","last_clean":"2024-09-16T10:29:46.716051+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:46.716051+0000","last_undegraded":"2024-09-16T10:29:46.716051+0000","last_fullsized":"2024-09-16T10:29:46.716051+0000","mapping_epoch":34,"log_start":"39'4200","ondisk_log_start":"39'4200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":6208,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10467,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14430","reported_seq":14683,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.669774+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:46.669774+0000","last_peered":"2024-09-16T10:29:46.669774+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:46.669774+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25530392,"num_objects":6234,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6234,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10487,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":239},{"shards":"2","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14301","reported_seq":14361,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:46.742105+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:46.742105+0000","last_peered":"2024-09-16T10:29:46.742105+0000","last_clean":"2024-09-16T10:29:46.742105+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:46.742105+0000","last_undegraded":"2024-09-16T10:29:46.742105+0000","last_fullsized":"2024-09-16T10:29:46.742105+0000","mapping_epoch":34,"log_start":"39'4300","ondisk_log_start":"39'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25235456,"num_objects":6161,"num_object_clones":0,"num_object_copies":12322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14301,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14303","reported_seq":14463,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:46.975907+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:46.975907+0000","last_peered":"2024-09-16T10:29:46.975907+0000","last_clean":"2024-09-16T10:29:46.975907+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:46.975907+0000","last_undegraded":"2024-09-16T10:29:46.975907+0000","last_fullsized":"2024-09-16T10:29:46.975907+0000","mapping_epoch":33,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25403392,"num_objects":6202,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10945,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14156","reported_seq":14477,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.976445+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:46.976445+0000","last_peered":"2024-09-16T10:29:46.976445+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:46.976445+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"39'4100","ondisk_log_start":"39'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25456640,"num_objects":6215,"num_object_clones":0,"num_object_copies":12430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4142,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10781,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2075},{"shards":"0,2","objects":249}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14628","reported_seq":15027,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.816896+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:46.816896+0000","last_peered":"2024-09-16T10:29:46.816896+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:46.816896+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25772032,"num_objects":6292,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10870,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2000},{"shards":"0,2","objects":238}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14213","reported_seq":14391,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:46.928418+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:46.928418+0000","last_peered":"2024-09-16T10:29:46.928418+0000","last_clean":"2024-09-16T10:29:46.928418+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:46.928418+0000","last_undegraded":"2024-09-16T10:29:46.928418+0000","last_fullsized":"2024-09-16T10:29:46.928418+0000","mapping_epoch":33,"log_start":"39'4200","ondisk_log_start":"39'4200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25300992,"num_objects":6177,"num_object_clones":0,"num_object_copies":12354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10016,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14317","reported_seq":22823,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:50.670460+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:50.670460+0000","last_peered":"2024-09-16T10:29:50.670460+0000","last_clean":"2024-09-16T10:29:50.670460+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:50.670460+0000","last_undegraded":"2024-09-16T10:29:50.670460+0000","last_fullsized":"2024-09-16T10:29:50.670460+0000","mapping_epoch":43,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166790,"num_objects":6146,"num_object_clones":0,"num_object_copies":12292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14315,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14711","reported_seq":14981,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.903765+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:46.903765+0000","last_peered":"2024-09-16T10:29:46.903765+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:46.903765+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25924550,"num_objects":6331,"num_object_clones":0,"num_object_copies":12662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14709,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1991},{"shards":"0,2","objects":247}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14260","reported_seq":14449,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.638412+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:46.638412+0000","last_peered":"2024-09-16T10:29:46.638412+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:46.638412+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"39'4200","ondisk_log_start":"39'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25228230,"num_objects":6161,"num_object_clones":0,"num_object_copies":12322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14258,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":229},{"shards":"2","objects":1924}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14256","reported_seq":16859,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.715218+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:46.715218+0000","last_peered":"2024-09-16T10:29:46.715218+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:46.715218+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4200","ondisk_log_start":"42'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25887686,"num_objects":6322,"num_object_clones":0,"num_object_copies":12644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2864,"num_objects_misplaced":74,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14254,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":5025792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":74},{"shards":"2","objects":3384}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'13941","reported_seq":14182,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.957829+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:46.957829+0000","last_peered":"2024-09-16T10:29:46.957829+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:46.957829+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"38'3900","ondisk_log_start":"38'3900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24957411,"num_objects":6094,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13940,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2014},{"shards":"0,2","objects":236}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14590","reported_seq":14858,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.936351+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:46.936351+0000","last_peered":"2024-09-16T10:29:46.936351+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:46.936351+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25904553,"num_objects":6327,"num_object_clones":0,"num_object_copies":12654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14587,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2056},{"shards":"0,2","objects":208}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14038","reported_seq":14264,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.837294+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:46.837294+0000","last_peered":"2024-09-16T10:29:46.837294+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:46.837294+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24994758,"num_objects":6104,"num_object_clones":0,"num_object_copies":12208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14036,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":2008},{"shards":"0,2","objects":201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14621","reported_seq":14859,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:46.929281+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:46.929281+0000","last_peered":"2024-09-16T10:29:46.929281+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:46.929281+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25704332,"num_objects":6279,"num_object_clones":0,"num_object_copies":12558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14618,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1955},{"shards":"0,2","objects":228}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":407532046,"num_objects":99512,"num_object_clones":0,"num_object_copies":199024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40965,"num_objects_misplaced":74,"num_objects_unfound":0,"num_objects_dirty":99512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206793,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10307,"num_bytes_recovered":39391715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1016287232,"data_stored":1016203215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160528,"ondisk_log_size":160528,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018968,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643432,"kb_used_data":336980,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93728408,"statfs":{"total":96636764160,"available":95977889792,"internally_reserved":0,"allocated":345067520,"data_stored":344799369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":313781936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,1,1,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.956},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":13.917}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":13.973000000000001},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":17.219999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691913,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597088,"kb_used_data":290648,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93774752,"statfs":{"total":96636764160,"available":96025346048,"internally_reserved":0,"allocated":297623552,"data_stored":297290367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":313781930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":6.9930000000000003},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.6600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.8519999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.21199999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215195,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737300,"kb_used_data":366916,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93634540,"statfs":{"total":96636764160,"available":95881768960,"internally_reserved":0,"allocated":375721984,"data_stored":375461519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":379252392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,2,4,1,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.3319999999999999},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.3370000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":2.5390000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.0230000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":375242752,"data_stored":375239139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296960000,"data_stored":296887281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":344084480,"data_stored":344076795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:52.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:53.248 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:53.248 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:53.273 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-16T10:29:52.700489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":399004193,"num_objects":97432,"num_object_clones":0,"num_object_copies":194864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208877,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10462,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160813,"ondisk_log_size":160813,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1997304,"kb_used_data":983704,"kb_used_omap":12,"kb_used_meta":1013555,"kb_avail":281118216,"statfs":{"total":289910292480,"available":287865053184,"internally_reserved":0,"allocated":1007312896,"data_stored":1006451095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1037880322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,7,6,9,4,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25858048,"num_objects":-6313,"num_object_clones":0,"num_object_copies":-12626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":-341,"num_objects_unfound":0,"num_objects_dirty":-6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":1396736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005401"},"pg_stats":[{"pgid":"2.f","version":"46'14332","reported_seq":22797,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:52.036516+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:52.036516+0000","last_peered":"2024-09-16T10:29:52.036516+0000","last_clean":"2024-09-16T10:29:52.036516+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:52.036516+0000","last_undegraded":"2024-09-16T10:29:52.036516+0000","last_fullsized":"2024-09-16T10:29:52.036516+0000","mapping_epoch":45,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25133056,"num_objects":6136,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14332,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14390","reported_seq":14550,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.646292+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:51.646292+0000","last_peered":"2024-09-16T10:29:51.646292+0000","last_clean":"2024-09-16T10:29:51.646292+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:51.646292+0000","last_undegraded":"2024-09-16T10:29:51.646292+0000","last_fullsized":"2024-09-16T10:29:51.646292+0000","mapping_epoch":34,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24870912,"num_objects":6072,"num_object_clones":0,"num_object_copies":12144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10603,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14570","reported_seq":14822,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.717770+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:51.717770+0000","last_peered":"2024-09-16T10:29:51.717770+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:51.717770+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24956952,"num_objects":6094,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10627,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":99},{"shards":"2","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14436","reported_seq":14496,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.726867+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:51.726867+0000","last_peered":"2024-09-16T10:29:51.726867+0000","last_clean":"2024-09-16T10:29:51.726867+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:51.726867+0000","last_undegraded":"2024-09-16T10:29:51.726867+0000","last_fullsized":"2024-09-16T10:29:51.726867+0000","mapping_epoch":34,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24682496,"num_objects":6026,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14436,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14443","reported_seq":14603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.964557+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:51.964557+0000","last_peered":"2024-09-16T10:29:51.964557+0000","last_clean":"2024-09-16T10:29:51.964557+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:51.964557+0000","last_undegraded":"2024-09-16T10:29:51.964557+0000","last_fullsized":"2024-09-16T10:29:51.964557+0000","mapping_epoch":33,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24829952,"num_objects":6062,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14309","reported_seq":14640,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.042977+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:52.042977+0000","last_peered":"2024-09-16T10:29:52.042977+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:52.042977+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24829952,"num_objects":6062,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10934,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1927},{"shards":"0,2","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14782","reported_seq":15191,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.982004+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:51.982004+0000","last_peered":"2024-09-16T10:29:51.982004+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:51.982004+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":6138,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11024,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1851},{"shards":"0,2","objects":84}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14349","reported_seq":14527,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.996575+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:51.996575+0000","last_peered":"2024-09-16T10:29:51.996575+0000","last_clean":"2024-09-16T10:29:51.996575+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:51.996575+0000","last_undegraded":"2024-09-16T10:29:51.996575+0000","last_fullsized":"2024-09-16T10:29:51.996575+0000","mapping_epoch":33,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24743936,"num_objects":6041,"num_object_clones":0,"num_object_copies":12082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14317","reported_seq":22823,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:50.670460+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:50.670460+0000","last_peered":"2024-09-16T10:29:50.670460+0000","last_clean":"2024-09-16T10:29:50.670460+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:50.670460+0000","last_undegraded":"2024-09-16T10:29:50.670460+0000","last_fullsized":"2024-09-16T10:29:50.670460+0000","mapping_epoch":43,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166790,"num_objects":6146,"num_object_clones":0,"num_object_copies":12292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14315,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14862","reported_seq":15136,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.820152+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:51.820152+0000","last_peered":"2024-09-16T10:29:51.820152+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:51.820152+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25306054,"num_objects":6180,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14860,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1842},{"shards":"0,2","objects":96}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14401","reported_seq":14590,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.669278+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:51.669278+0000","last_peered":"2024-09-16T10:29:51.669278+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:51.669278+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24650694,"num_objects":6020,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14399,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":88},{"shards":"2","objects":1783}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14393","reported_seq":17256,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.718861+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:51.718861+0000","last_peered":"2024-09-16T10:29:51.718861+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:51.718861+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25326534,"num_objects":6185,"num_object_clones":0,"num_object_copies":12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14391,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1388,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14082","reported_seq":14333,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.956550+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:51.956550+0000","last_peered":"2024-09-16T10:29:51.956550+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:51.956550+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379875,"num_objects":5953,"num_object_clones":0,"num_object_copies":11906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1878},{"shards":"0,2","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14728","reported_seq":15004,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.018029+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:52.018029+0000","last_peered":"2024-09-16T10:29:52.018029+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:52.018029+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25339305,"num_objects":6189,"num_object_clones":0,"num_object_copies":12378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14725,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1922},{"shards":"0,2","objects":70}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14153","reported_seq":14385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.959883+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:51.959883+0000","last_peered":"2024-09-16T10:29:51.959883+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:51.959883+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"39'4100","ondisk_log_start":"39'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24523718,"num_objects":5989,"num_object_clones":0,"num_object_copies":11978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14151,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1896},{"shards":"0,2","objects":86}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14763","reported_seq":15001,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.038641+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:52.038641+0000","last_peered":"2024-09-16T10:29:52.038641+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:52.038641+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25122700,"num_objects":6137,"num_object_clones":0,"num_object_copies":12274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14760,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1813},{"shards":"0,2","objects":86}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":399004174,"num_objects":97430,"num_object_clones":0,"num_object_copies":194860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97430,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208875,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10461,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1005068288,"data_stored":1004984271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018969,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643392,"kb_used_data":332856,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93728448,"statfs":{"total":96636764160,"available":95977930752,"internally_reserved":0,"allocated":340844544,"data_stored":340576393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.309},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.76}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.415},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.1960000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691913,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597088,"kb_used_data":290648,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93774752,"statfs":{"total":96636764160,"available":96025346048,"internally_reserved":0,"allocated":297623552,"data_stored":297290367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":313781930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":6.9930000000000003},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.6600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.8519999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.21199999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215196,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756824,"kb_used_data":360200,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93615016,"statfs":{"total":96636764160,"available":95861776384,"internally_reserved":0,"allocated":368844800,"data_stored":368584335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":406122152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,2,4,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.4240000000000004},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":5.7149999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":3.6539999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.5140000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368431104,"data_stored":368427491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296960000,"data_stored":296887281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339677184,"data_stored":339669499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:53.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:53.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'14570", 'reported_seq': 14822, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.717770+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:51.717770+0000', 'last_peered': '2024-09-16T10:29:51.717770+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:51.717770+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'4500", 'ondisk_log_start': "42'4500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24956952, 'num_objects': 6094, 'num_object_clones': 0, 'num_object_copies': 12188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6094, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10627, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 99}, {'shards': '2', 'objects': 1791}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:53.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:53.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'14309", 'reported_seq': 14640, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:52.042977+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:52.042977+0000', 'last_peered': '2024-09-16T10:29:52.042977+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:52.042977+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'4300", 'ondisk_log_start': "42'4300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24829952, 'num_objects': 6062, 'num_object_clones': 0, 'num_object_copies': 12124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4137, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10934, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 82, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1927}, {'shards': '0,2', 'objects': 96}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:53.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:53.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'14782", 'reported_seq': 15191, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.982004+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:51.982004+0000', 'last_peered': '2024-09-16T10:29:51.982004+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:51.982004+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25141248, 'num_objects': 6138, 'num_object_clones': 0, 'num_object_copies': 12276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11024, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1851}, {'shards': '0,2', 'objects': 84}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:53.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:53.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'14862", 'reported_seq': 15136, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.820152+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:51.820152+0000', 'last_peered': '2024-09-16T10:29:51.820152+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:51.820152+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'4800", 'ondisk_log_start': "42'4800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25306054, 'num_objects': 6180, 'num_object_clones': 0, 'num_object_copies': 12360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14860, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 98, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1842}, {'shards': '0,2', 'objects': 96}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:53.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:53.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'14401", 'reported_seq': 14590, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.669278+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:51.669278+0000', 'last_peered': '2024-09-16T10:29:51.669278+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:51.669278+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'4400", 'ondisk_log_start': "42'4400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24650694, 'num_objects': 6020, 'num_object_clones': 0, 'num_object_copies': 12040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14399, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 88}, {'shards': '2', 'objects': 1783}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:53.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:53.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'14393", 'reported_seq': 17256, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.718861+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:29:51.718861+0000', 'last_peered': '2024-09-16T10:29:51.718861+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:51.718861+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'4300", 'ondisk_log_start': "42'4300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25326534, 'num_objects': 6185, 'num_object_clones': 0, 'num_object_copies': 12370, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2808, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14391, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1388, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3377}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:53.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:53.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'14082", 'reported_seq': 14333, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.956550+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:51.956550+0000', 'last_peered': '2024-09-16T10:29:51.956550+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:51.956550+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "38'4000", 'ondisk_log_start': "38'4000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24379875, 'num_objects': 5953, 'num_object_clones': 0, 'num_object_copies': 11906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14081, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1878}, {'shards': '0,2', 'objects': 95}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:53.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:53.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'14728", 'reported_seq': 15004, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:52.018029+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:52.018029+0000', 'last_peered': '2024-09-16T10:29:52.018029+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:52.018029+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25339305, 'num_objects': 6189, 'num_object_clones': 0, 'num_object_copies': 12378, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14725, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1922}, {'shards': '0,2', 'objects': 70}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:53.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:53.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'14153", 'reported_seq': 14385, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.959883+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:51.959883+0000', 'last_peered': '2024-09-16T10:29:51.959883+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:51.959883+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "39'4100", 'ondisk_log_start': "39'4100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24523718, 'num_objects': 5989, 'num_object_clones': 0, 'num_object_copies': 11978, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14151, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1896}, {'shards': '0,2', 'objects': 86}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:53.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:53.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'14763", 'reported_seq': 15001, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:52.038641+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:52.038641+0000', 'last_peered': '2024-09-16T10:29:52.038641+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:52.038641+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25122700, 'num_objects': 6137, 'num_object_clones': 0, 'num_object_copies': 12274, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14760, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1813}, {'shards': '0,2', 'objects': 86}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:53.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:53.934 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:53.935 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 40885/194864 objects degraded (20.981%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":482,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":97432,"data_bytes":399004193,"bytes_used":2045239296,"bytes_avail":287865053184,"bytes_total":289910292480,"degraded_objects":40885,"degraded_total":194864,"degraded_ratio":0.20981299778306922,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":116342,"recovering_keys_per_sec":0,"num_objects_recovered":470,"num_bytes_recovered":1396736,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":525},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T10:29:36.693808+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:29:53.956 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:53.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:54.473 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:54.476 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:54.494 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-16T10:29:52.700489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":399004193,"num_objects":97432,"num_object_clones":0,"num_object_copies":194864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208877,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10462,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160813,"ondisk_log_size":160813,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1997304,"kb_used_data":983704,"kb_used_omap":12,"kb_used_meta":1013555,"kb_avail":281118216,"statfs":{"total":289910292480,"available":287865053184,"internally_reserved":0,"allocated":1007312896,"data_stored":1006451095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1037880322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,7,6,9,4,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25858048,"num_objects":-6313,"num_object_clones":0,"num_object_copies":-12626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":-341,"num_objects_unfound":0,"num_objects_dirty":-6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":1396736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005401"},"pg_stats":[{"pgid":"2.f","version":"46'14332","reported_seq":22797,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:52.036516+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:52.036516+0000","last_peered":"2024-09-16T10:29:52.036516+0000","last_clean":"2024-09-16T10:29:52.036516+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:52.036516+0000","last_undegraded":"2024-09-16T10:29:52.036516+0000","last_fullsized":"2024-09-16T10:29:52.036516+0000","mapping_epoch":45,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25133056,"num_objects":6136,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14332,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14390","reported_seq":14550,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.646292+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:51.646292+0000","last_peered":"2024-09-16T10:29:51.646292+0000","last_clean":"2024-09-16T10:29:51.646292+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:51.646292+0000","last_undegraded":"2024-09-16T10:29:51.646292+0000","last_fullsized":"2024-09-16T10:29:51.646292+0000","mapping_epoch":34,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24870912,"num_objects":6072,"num_object_clones":0,"num_object_copies":12144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10603,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14570","reported_seq":14822,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.717770+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:51.717770+0000","last_peered":"2024-09-16T10:29:51.717770+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:51.717770+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24956952,"num_objects":6094,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10627,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":99},{"shards":"2","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14436","reported_seq":14496,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.726867+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:51.726867+0000","last_peered":"2024-09-16T10:29:51.726867+0000","last_clean":"2024-09-16T10:29:51.726867+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:51.726867+0000","last_undegraded":"2024-09-16T10:29:51.726867+0000","last_fullsized":"2024-09-16T10:29:51.726867+0000","mapping_epoch":34,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24682496,"num_objects":6026,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14436,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14443","reported_seq":14603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.964557+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:51.964557+0000","last_peered":"2024-09-16T10:29:51.964557+0000","last_clean":"2024-09-16T10:29:51.964557+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:51.964557+0000","last_undegraded":"2024-09-16T10:29:51.964557+0000","last_fullsized":"2024-09-16T10:29:51.964557+0000","mapping_epoch":33,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24829952,"num_objects":6062,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14309","reported_seq":14640,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.042977+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:52.042977+0000","last_peered":"2024-09-16T10:29:52.042977+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:52.042977+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24829952,"num_objects":6062,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10934,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1927},{"shards":"0,2","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14782","reported_seq":15191,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.982004+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:51.982004+0000","last_peered":"2024-09-16T10:29:51.982004+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:51.982004+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":6138,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11024,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1851},{"shards":"0,2","objects":84}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14349","reported_seq":14527,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.996575+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:51.996575+0000","last_peered":"2024-09-16T10:29:51.996575+0000","last_clean":"2024-09-16T10:29:51.996575+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:51.996575+0000","last_undegraded":"2024-09-16T10:29:51.996575+0000","last_fullsized":"2024-09-16T10:29:51.996575+0000","mapping_epoch":33,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24743936,"num_objects":6041,"num_object_clones":0,"num_object_copies":12082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973424+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:02.973424+0000","last_peered":"2024-09-16T10:29:02.973424+0000","last_clean":"2024-09-16T10:29:02.973424+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:02.973424+0000","last_undegraded":"2024-09-16T10:29:02.973424+0000","last_fullsized":"2024-09-16T10:29:02.973424+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14317","reported_seq":22823,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:50.670460+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:50.670460+0000","last_peered":"2024-09-16T10:29:50.670460+0000","last_clean":"2024-09-16T10:29:50.670460+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:50.670460+0000","last_undegraded":"2024-09-16T10:29:50.670460+0000","last_fullsized":"2024-09-16T10:29:50.670460+0000","mapping_epoch":43,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25166790,"num_objects":6146,"num_object_clones":0,"num_object_copies":12292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14315,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14862","reported_seq":15136,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.820152+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:51.820152+0000","last_peered":"2024-09-16T10:29:51.820152+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:51.820152+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25306054,"num_objects":6180,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14860,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1842},{"shards":"0,2","objects":96}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14401","reported_seq":14590,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.669278+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:51.669278+0000","last_peered":"2024-09-16T10:29:51.669278+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:51.669278+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24650694,"num_objects":6020,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14399,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":88},{"shards":"2","objects":1783}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14393","reported_seq":17256,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.718861+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:51.718861+0000","last_peered":"2024-09-16T10:29:51.718861+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:51.718861+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25326534,"num_objects":6185,"num_object_clones":0,"num_object_copies":12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14391,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1388,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14082","reported_seq":14333,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.956550+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:51.956550+0000","last_peered":"2024-09-16T10:29:51.956550+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:51.956550+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379875,"num_objects":5953,"num_object_clones":0,"num_object_copies":11906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1878},{"shards":"0,2","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14728","reported_seq":15004,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.018029+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:52.018029+0000","last_peered":"2024-09-16T10:29:52.018029+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:52.018029+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25339305,"num_objects":6189,"num_object_clones":0,"num_object_copies":12378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14725,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1922},{"shards":"0,2","objects":70}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14153","reported_seq":14385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.959883+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:51.959883+0000","last_peered":"2024-09-16T10:29:51.959883+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:51.959883+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"39'4100","ondisk_log_start":"39'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24523718,"num_objects":5989,"num_object_clones":0,"num_object_copies":11978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14151,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1896},{"shards":"0,2","objects":86}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14763","reported_seq":15001,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.038641+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:52.038641+0000","last_peered":"2024-09-16T10:29:52.038641+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:52.038641+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25122700,"num_objects":6137,"num_object_clones":0,"num_object_copies":12274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14760,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1813},{"shards":"0,2","objects":86}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":399004174,"num_objects":97430,"num_object_clones":0,"num_object_copies":194860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97430,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208875,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10461,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1005068288,"data_stored":1004984271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160810,"ondisk_log_size":160810,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018969,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643392,"kb_used_data":332856,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93728448,"statfs":{"total":96636764160,"available":95977930752,"internally_reserved":0,"allocated":340844544,"data_stored":340576393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.309},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.76}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.415},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.1960000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691913,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":597088,"kb_used_data":290648,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93774752,"statfs":{"total":96636764160,"available":96025346048,"internally_reserved":0,"allocated":297623552,"data_stored":297290367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":313781930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":6.9930000000000003},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.6600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.8519999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.21199999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215196,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756824,"kb_used_data":360200,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93615016,"statfs":{"total":96636764160,"available":95861776384,"internally_reserved":0,"allocated":368844800,"data_stored":368584335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":406122152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,2,4,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.4240000000000004},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":5.7149999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":3.6539999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.5140000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368431104,"data_stored":368427491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":296960000,"data_stored":296887281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339677184,"data_stored":339669499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:57.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:58.057 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:58.059 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:58.081 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-16T10:29:56.702151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398496289,"num_objects":97308,"num_object_clones":0,"num_object_copies":194616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209001,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10462,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160837,"ondisk_log_size":160837,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2005620,"kb_used_data":982160,"kb_used_omap":12,"kb_used_meta":1023411,"kb_avail":281109900,"statfs":{"total":289910292480,"available":287856537600,"internally_reserved":0,"allocated":1005731840,"data_stored":1004870039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1047972866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,7,6,9,4,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18173952,"num_objects":-4437,"num_object_clones":0,"num_object_copies":-8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":-198,"num_objects_unfound":0,"num_objects_dirty":-4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005396"},"pg_stats":[{"pgid":"2.f","version":"46'14332","reported_seq":22797,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:52.036516+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:52.036516+0000","last_peered":"2024-09-16T10:29:52.036516+0000","last_clean":"2024-09-16T10:29:52.036516+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:52.036516+0000","last_undegraded":"2024-09-16T10:29:52.036516+0000","last_fullsized":"2024-09-16T10:29:52.036516+0000","mapping_epoch":45,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25133056,"num_objects":6136,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14332,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14390","reported_seq":14550,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.646292+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:51.646292+0000","last_peered":"2024-09-16T10:29:51.646292+0000","last_clean":"2024-09-16T10:29:51.646292+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:51.646292+0000","last_undegraded":"2024-09-16T10:29:51.646292+0000","last_fullsized":"2024-09-16T10:29:51.646292+0000","mapping_epoch":34,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24870912,"num_objects":6072,"num_object_clones":0,"num_object_copies":12144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10603,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14570","reported_seq":14822,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.717770+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:51.717770+0000","last_peered":"2024-09-16T10:29:51.717770+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:51.717770+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24956952,"num_objects":6094,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10627,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":99},{"shards":"2","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14436","reported_seq":14496,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.726867+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:51.726867+0000","last_peered":"2024-09-16T10:29:51.726867+0000","last_clean":"2024-09-16T10:29:51.726867+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:51.726867+0000","last_undegraded":"2024-09-16T10:29:51.726867+0000","last_fullsized":"2024-09-16T10:29:51.726867+0000","mapping_epoch":34,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24682496,"num_objects":6026,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14436,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14443","reported_seq":14603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.964557+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:51.964557+0000","last_peered":"2024-09-16T10:29:51.964557+0000","last_clean":"2024-09-16T10:29:51.964557+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:51.964557+0000","last_undegraded":"2024-09-16T10:29:51.964557+0000","last_fullsized":"2024-09-16T10:29:51.964557+0000","mapping_epoch":33,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24829952,"num_objects":6062,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14309","reported_seq":14640,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.042977+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:52.042977+0000","last_peered":"2024-09-16T10:29:52.042977+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:52.042977+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24829952,"num_objects":6062,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10934,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1927},{"shards":"0,2","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14782","reported_seq":15191,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.982004+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:51.982004+0000","last_peered":"2024-09-16T10:29:51.982004+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:51.982004+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":6138,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11024,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1851},{"shards":"0,2","objects":84}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14349","reported_seq":14527,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.996575+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:51.996575+0000","last_peered":"2024-09-16T10:29:51.996575+0000","last_clean":"2024-09-16T10:29:51.996575+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:51.996575+0000","last_undegraded":"2024-09-16T10:29:51.996575+0000","last_fullsized":"2024-09-16T10:29:51.996575+0000","mapping_epoch":33,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24743936,"num_objects":6041,"num_object_clones":0,"num_object_copies":12082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14441","reported_seq":22947,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:55.613140+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:55.613140+0000","last_peered":"2024-09-16T10:29:55.613140+0000","last_clean":"2024-09-16T10:29:55.613140+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:55.613140+0000","last_undegraded":"2024-09-16T10:29:55.613140+0000","last_fullsized":"2024-09-16T10:29:55.613140+0000","mapping_epoch":43,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24658886,"num_objects":6022,"num_object_clones":0,"num_object_copies":12044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14439,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14862","reported_seq":15136,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.820152+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:51.820152+0000","last_peered":"2024-09-16T10:29:51.820152+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:51.820152+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25306054,"num_objects":6180,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14860,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1842},{"shards":"0,2","objects":96}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14401","reported_seq":14590,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.669278+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:51.669278+0000","last_peered":"2024-09-16T10:29:51.669278+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:51.669278+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24650694,"num_objects":6020,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14399,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":88},{"shards":"2","objects":1783}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14393","reported_seq":17256,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.718861+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:51.718861+0000","last_peered":"2024-09-16T10:29:51.718861+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:51.718861+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25326534,"num_objects":6185,"num_object_clones":0,"num_object_copies":12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14391,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1388,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14082","reported_seq":14333,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.956550+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:51.956550+0000","last_peered":"2024-09-16T10:29:51.956550+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:51.956550+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379875,"num_objects":5953,"num_object_clones":0,"num_object_copies":11906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1878},{"shards":"0,2","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14728","reported_seq":15004,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.018029+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:52.018029+0000","last_peered":"2024-09-16T10:29:52.018029+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:52.018029+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25339305,"num_objects":6189,"num_object_clones":0,"num_object_copies":12378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14725,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1922},{"shards":"0,2","objects":70}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14153","reported_seq":14385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.959883+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:51.959883+0000","last_peered":"2024-09-16T10:29:51.959883+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:51.959883+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"39'4100","ondisk_log_start":"39'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24523718,"num_objects":5989,"num_object_clones":0,"num_object_copies":11978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14151,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1896},{"shards":"0,2","objects":86}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14763","reported_seq":15001,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.038641+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:52.038641+0000","last_peered":"2024-09-16T10:29:52.038641+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:52.038641+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25122700,"num_objects":6137,"num_object_clones":0,"num_object_copies":12274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14760,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1813},{"shards":"0,2","objects":86}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":398496270,"num_objects":97306,"num_object_clones":0,"num_object_copies":194612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97306,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208999,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10461,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1003053056,"data_stored":1002969039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018969,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643392,"kb_used_data":332856,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93728448,"statfs":{"total":96636764160,"available":95977930752,"internally_reserved":0,"allocated":340844544,"data_stored":340576393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.309},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.76}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.415},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.1960000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605404,"kb_used_data":289104,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93766436,"statfs":{"total":96636764160,"available":96016830464,"internally_reserved":0,"allocated":296042496,"data_stored":295709311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":323874474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":6.9930000000000003},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.6600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.8519999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.21199999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215196,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756824,"kb_used_data":360200,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93615016,"statfs":{"total":96636764160,"available":95861776384,"internally_reserved":0,"allocated":368844800,"data_stored":368584335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":406122152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,2,4,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.4240000000000004},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":5.7149999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":3.6539999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.5140000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368431104,"data_stored":368427491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294944768,"data_stored":294872049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339677184,"data_stored":339669499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:58.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:58.646 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:58.647 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:58.673 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-16T10:29:56.702151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":398496289,"num_objects":97308,"num_object_clones":0,"num_object_copies":194616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":209001,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10462,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160837,"ondisk_log_size":160837,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2005620,"kb_used_data":982160,"kb_used_omap":12,"kb_used_meta":1023411,"kb_avail":281109900,"statfs":{"total":289910292480,"available":287856537600,"internally_reserved":0,"allocated":1005731840,"data_stored":1004870039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1047972866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,7,6,9,4,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18173952,"num_objects":-4437,"num_object_clones":0,"num_object_copies":-8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":-198,"num_objects_unfound":0,"num_objects_dirty":-4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005396"},"pg_stats":[{"pgid":"2.f","version":"46'14332","reported_seq":22797,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:52.036516+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:52.036516+0000","last_peered":"2024-09-16T10:29:52.036516+0000","last_clean":"2024-09-16T10:29:52.036516+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:52.036516+0000","last_undegraded":"2024-09-16T10:29:52.036516+0000","last_fullsized":"2024-09-16T10:29:52.036516+0000","mapping_epoch":45,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25133056,"num_objects":6136,"num_object_clones":0,"num_object_copies":12272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14332,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14390","reported_seq":14550,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.646292+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:51.646292+0000","last_peered":"2024-09-16T10:29:51.646292+0000","last_clean":"2024-09-16T10:29:51.646292+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:51.646292+0000","last_undegraded":"2024-09-16T10:29:51.646292+0000","last_fullsized":"2024-09-16T10:29:51.646292+0000","mapping_epoch":34,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24870912,"num_objects":6072,"num_object_clones":0,"num_object_copies":12144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10603,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14570","reported_seq":14822,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.717770+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:51.717770+0000","last_peered":"2024-09-16T10:29:51.717770+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:51.717770+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24956952,"num_objects":6094,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10627,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":99},{"shards":"2","objects":1791}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14436","reported_seq":14496,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.726867+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:51.726867+0000","last_peered":"2024-09-16T10:29:51.726867+0000","last_clean":"2024-09-16T10:29:51.726867+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:51.726867+0000","last_undegraded":"2024-09-16T10:29:51.726867+0000","last_fullsized":"2024-09-16T10:29:51.726867+0000","mapping_epoch":34,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24682496,"num_objects":6026,"num_object_clones":0,"num_object_copies":12052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14436,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14443","reported_seq":14603,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.964557+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:51.964557+0000","last_peered":"2024-09-16T10:29:51.964557+0000","last_clean":"2024-09-16T10:29:51.964557+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:51.964557+0000","last_undegraded":"2024-09-16T10:29:51.964557+0000","last_fullsized":"2024-09-16T10:29:51.964557+0000","mapping_epoch":33,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24829952,"num_objects":6062,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14309","reported_seq":14640,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.042977+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:52.042977+0000","last_peered":"2024-09-16T10:29:52.042977+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:52.042977+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24829952,"num_objects":6062,"num_object_clones":0,"num_object_copies":12124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10934,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1927},{"shards":"0,2","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14782","reported_seq":15191,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.982004+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:51.982004+0000","last_peered":"2024-09-16T10:29:51.982004+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:51.982004+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":6138,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11024,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1851},{"shards":"0,2","objects":84}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14349","reported_seq":14527,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:51.996575+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:51.996575+0000","last_peered":"2024-09-16T10:29:51.996575+0000","last_clean":"2024-09-16T10:29:51.996575+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:51.996575+0000","last_undegraded":"2024-09-16T10:29:51.996575+0000","last_fullsized":"2024-09-16T10:29:51.996575+0000","mapping_epoch":33,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24743936,"num_objects":6041,"num_object_clones":0,"num_object_copies":12082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10152,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14441","reported_seq":22947,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:55.613140+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:55.613140+0000","last_peered":"2024-09-16T10:29:55.613140+0000","last_clean":"2024-09-16T10:29:55.613140+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:55.613140+0000","last_undegraded":"2024-09-16T10:29:55.613140+0000","last_fullsized":"2024-09-16T10:29:55.613140+0000","mapping_epoch":43,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24658886,"num_objects":6022,"num_object_clones":0,"num_object_copies":12044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14439,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'14862","reported_seq":15136,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.820152+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:51.820152+0000","last_peered":"2024-09-16T10:29:51.820152+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:51.820152+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25306054,"num_objects":6180,"num_object_clones":0,"num_object_copies":12360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14860,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1842},{"shards":"0,2","objects":96}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14401","reported_seq":14590,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.669278+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:51.669278+0000","last_peered":"2024-09-16T10:29:51.669278+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:51.669278+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24650694,"num_objects":6020,"num_object_clones":0,"num_object_copies":12040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14399,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"0,2","objects":88},{"shards":"2","objects":1783}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14393","reported_seq":17256,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.718861+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:51.718861+0000","last_peered":"2024-09-16T10:29:51.718861+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:51.718861+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25326534,"num_objects":6185,"num_object_clones":0,"num_object_copies":12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14391,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1388,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3377}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14082","reported_seq":14333,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.956550+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:51.956550+0000","last_peered":"2024-09-16T10:29:51.956550+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:51.956550+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"38'4000","ondisk_log_start":"38'4000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24379875,"num_objects":5953,"num_object_clones":0,"num_object_copies":11906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14081,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1878},{"shards":"0,2","objects":95}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14728","reported_seq":15004,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.018029+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:52.018029+0000","last_peered":"2024-09-16T10:29:52.018029+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:52.018029+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25339305,"num_objects":6189,"num_object_clones":0,"num_object_copies":12378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14725,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1922},{"shards":"0,2","objects":70}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14153","reported_seq":14385,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:51.959883+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:51.959883+0000","last_peered":"2024-09-16T10:29:51.959883+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:51.959883+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"39'4100","ondisk_log_start":"39'4100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24523718,"num_objects":5989,"num_object_clones":0,"num_object_copies":11978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14151,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1896},{"shards":"0,2","objects":86}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14763","reported_seq":15001,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:52.038641+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:52.038641+0000","last_peered":"2024-09-16T10:29:52.038641+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:52.038641+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25122700,"num_objects":6137,"num_object_clones":0,"num_object_copies":12274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14760,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1813},{"shards":"0,2","objects":86}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":398496270,"num_objects":97306,"num_object_clones":0,"num_object_copies":194612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97306,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":208999,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10461,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1003053056,"data_stored":1002969039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018969,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":643392,"kb_used_data":332856,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93728448,"statfs":{"total":96636764160,"available":95977930752,"internally_reserved":0,"allocated":340844544,"data_stored":340576393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.309},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":1.76}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.415},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.1960000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605404,"kb_used_data":289104,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93766436,"statfs":{"total":96636764160,"available":96016830464,"internally_reserved":0,"allocated":296042496,"data_stored":295709311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":323874474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":6.9930000000000003},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.6600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.8519999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.21199999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215196,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756824,"kb_used_data":360200,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93615016,"statfs":{"total":96636764160,"available":95861776384,"internally_reserved":0,"allocated":368844800,"data_stored":368584335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":406122152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,3,2,4,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.4240000000000004},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":5.7149999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":3.6539999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.5140000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":368431104,"data_stored":368427491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294944768,"data_stored":294872049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339677184,"data_stored":339669499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:29:58.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:29:58.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'14570", 'reported_seq': 14822, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.717770+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:29:51.717770+0000', 'last_peered': '2024-09-16T10:29:51.717770+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:29:51.717770+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'4500", 'ondisk_log_start': "42'4500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24956952, 'num_objects': 6094, 'num_object_clones': 0, 'num_object_copies': 12188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6094, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10627, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 99}, {'shards': '2', 'objects': 1791}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:58.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:29:58.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'14309", 'reported_seq': 14640, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:52.042977+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:29:52.042977+0000', 'last_peered': '2024-09-16T10:29:52.042977+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:29:52.042977+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'4300", 'ondisk_log_start': "42'4300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24829952, 'num_objects': 6062, 'num_object_clones': 0, 'num_object_copies': 12124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4137, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10934, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 82, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1927}, {'shards': '0,2', 'objects': 96}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:58.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:29:58.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'14782", 'reported_seq': 15191, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.982004+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:29:51.982004+0000', 'last_peered': '2024-09-16T10:29:51.982004+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:29:51.982004+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25141248, 'num_objects': 6138, 'num_object_clones': 0, 'num_object_copies': 12276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11024, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1851}, {'shards': '0,2', 'objects': 84}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:58.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:29:58.676 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'14862", 'reported_seq': 15136, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.820152+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:29:51.820152+0000', 'last_peered': '2024-09-16T10:29:51.820152+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:29:51.820152+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'4800", 'ondisk_log_start': "42'4800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25306054, 'num_objects': 6180, 'num_object_clones': 0, 'num_object_copies': 12360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14860, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 98, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1842}, {'shards': '0,2', 'objects': 96}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:58.677 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:29:58.677 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'14401", 'reported_seq': 14590, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.669278+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:29:51.669278+0000', 'last_peered': '2024-09-16T10:29:51.669278+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:29:51.669278+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'4400", 'ondisk_log_start': "42'4400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24650694, 'num_objects': 6020, 'num_object_clones': 0, 'num_object_copies': 12040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4239, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14399, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '0,2', 'objects': 88}, {'shards': '2', 'objects': 1783}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:58.677 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:29:58.677 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'14393", 'reported_seq': 17256, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.718861+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:29:51.718861+0000', 'last_peered': '2024-09-16T10:29:51.718861+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:29:51.718861+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'4300", 'ondisk_log_start': "42'4300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25326534, 'num_objects': 6185, 'num_object_clones': 0, 'num_object_copies': 12370, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2808, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14391, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1388, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3377}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:29:58.677 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:29:58.678 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'14082", 'reported_seq': 14333, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.956550+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:29:51.956550+0000', 'last_peered': '2024-09-16T10:29:51.956550+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:29:51.956550+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "38'4000", 'ondisk_log_start': "38'4000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24379875, 'num_objects': 5953, 'num_object_clones': 0, 'num_object_copies': 11906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14081, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1878}, {'shards': '0,2', 'objects': 95}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:58.678 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:29:58.678 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'14728", 'reported_seq': 15004, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:52.018029+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:29:52.018029+0000', 'last_peered': '2024-09-16T10:29:52.018029+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:29:52.018029+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25339305, 'num_objects': 6189, 'num_object_clones': 0, 'num_object_copies': 12378, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6189, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14725, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1922}, {'shards': '0,2', 'objects': 70}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:58.678 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:29:58.678 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'14153", 'reported_seq': 14385, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:51.959883+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:29:51.959883+0000', 'last_peered': '2024-09-16T10:29:51.959883+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:29:51.959883+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "39'4100", 'ondisk_log_start': "39'4100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24523718, 'num_objects': 5989, 'num_object_clones': 0, 'num_object_copies': 11978, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5989, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14151, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1896}, {'shards': '0,2', 'objects': 86}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:58.678 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:29:58.678 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'14763", 'reported_seq': 15001, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:29:52.038641+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:29:52.038641+0000', 'last_peered': '2024-09-16T10:29:52.038641+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:29:52.038641+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25122700, 'num_objects': 6137, 'num_object_clones': 0, 'num_object_copies': 12274, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4327, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14760, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1813}, {'shards': '0,2', 'objects': 86}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:29:58.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:29:59.351 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:59.351 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 40710/190510 objects degraded (21.369%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":488,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":95255,"data_bytes":390087201,"bytes_used":2034237440,"bytes_avail":287876055040,"bytes_total":289910292480,"degraded_objects":40710,"degraded_total":190510,"degraded_ratio":0.21368957010130699,"recovering_objects_per_sec":39,"recovering_bytes_per_sec":67552,"recovering_keys_per_sec":0,"num_objects_recovered":476,"num_bytes_recovered":811008,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":530},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T10:29:36.693808+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:29:59.369 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:29:59.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:29:59.932 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:29:59.932 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:29:59.958 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-16T10:29:58.703095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":390087201,"num_objects":95255,"num_object_clones":0,"num_object_copies":190510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211054,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10637,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1986560,"kb_used_data":971300,"kb_used_omap":12,"kb_used_meta":1015219,"kb_avail":281128960,"statfs":{"total":289910292480,"available":287876055040,"internally_reserved":0,"allocated":994611200,"data_stored":993749399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1039584258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,9,9,3,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26083328,"num_objects":-6368,"num_object_clones":0,"num_object_copies":-12736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-278,"num_objects_misplaced":-198,"num_objects_unfound":0,"num_objects_dirty":-6368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6368,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":476,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005566"},"pg_stats":[{"pgid":"2.f","version":"46'14467","reported_seq":22932,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:57.112099+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:29:57.112099+0000","last_peered":"2024-09-16T10:29:57.112099+0000","last_clean":"2024-09-16T10:29:57.112099+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:29:57.112099+0000","last_undegraded":"2024-09-16T10:29:57.112099+0000","last_fullsized":"2024-09-16T10:29:57.112099+0000","mapping_epoch":45,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24580096,"num_objects":6001,"num_object_clones":0,"num_object_copies":12002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14467,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14539","reported_seq":14699,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:56.681885+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:29:56.681885+0000","last_peered":"2024-09-16T10:29:56.681885+0000","last_clean":"2024-09-16T10:29:56.681885+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:29:56.681885+0000","last_undegraded":"2024-09-16T10:29:56.681885+0000","last_fullsized":"2024-09-16T10:29:56.681885+0000","mapping_epoch":34,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24260608,"num_objects":5923,"num_object_clones":0,"num_object_copies":11846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10752,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14712","reported_seq":14966,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:56.704052+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:29:56.704052+0000","last_peered":"2024-09-16T10:29:56.704052+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:29:56.704052+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24375320,"num_objects":5952,"num_object_clones":0,"num_object_copies":11904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5952,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10769,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14579","reported_seq":14639,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:56.713205+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:29:56.713205+0000","last_peered":"2024-09-16T10:29:56.713205+0000","last_clean":"2024-09-16T10:29:56.713205+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:29:56.713205+0000","last_undegraded":"2024-09-16T10:29:56.713205+0000","last_fullsized":"2024-09-16T10:29:56.713205+0000","mapping_epoch":34,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24096768,"num_objects":5883,"num_object_clones":0,"num_object_copies":11766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14579,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14563","reported_seq":14723,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:57.094708+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:29:57.094708+0000","last_peered":"2024-09-16T10:29:57.094708+0000","last_clean":"2024-09-16T10:29:57.094708+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:29:57.094708+0000","last_undegraded":"2024-09-16T10:29:57.094708+0000","last_fullsized":"2024-09-16T10:29:57.094708+0000","mapping_epoch":33,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24338432,"num_objects":5942,"num_object_clones":0,"num_object_copies":11884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11205,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14462","reported_seq":14795,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:57.074099+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:29:57.074099+0000","last_peered":"2024-09-16T10:29:57.074099+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:29:57.074099+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24203264,"num_objects":5909,"num_object_clones":0,"num_object_copies":11818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11087,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1775}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'14916","reported_seq":15322,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:57.013679+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:29:57.013679+0000","last_peered":"2024-09-16T10:29:57.013679+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:29:57.013679+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24592384,"num_objects":6004,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11158,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1717}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14485","reported_seq":14663,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:56.895215+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:29:56.895215+0000","last_peered":"2024-09-16T10:29:56.895215+0000","last_clean":"2024-09-16T10:29:56.895215+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:29:56.895215+0000","last_undegraded":"2024-09-16T10:29:56.895215+0000","last_fullsized":"2024-09-16T10:29:56.895215+0000","mapping_epoch":33,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24186880,"num_objects":5905,"num_object_clones":0,"num_object_copies":11810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14441","reported_seq":22947,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:55.613140+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:29:55.613140+0000","last_peered":"2024-09-16T10:29:55.613140+0000","last_clean":"2024-09-16T10:29:55.613140+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:29:55.613140+0000","last_undegraded":"2024-09-16T10:29:55.613140+0000","last_fullsized":"2024-09-16T10:29:55.613140+0000","mapping_epoch":43,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24658886,"num_objects":6022,"num_object_clones":0,"num_object_copies":12044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14439,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15019","reported_seq":15292,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:57.082816+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:29:57.082816+0000","last_peered":"2024-09-16T10:29:57.082816+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:29:57.082816+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24662982,"num_objects":6023,"num_object_clones":0,"num_object_copies":12046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15017,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1686}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14527","reported_seq":14715,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:56.751010+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:29:56.751010+0000","last_peered":"2024-09-16T10:29:56.751010+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:29:56.751010+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24134598,"num_objects":5894,"num_object_clones":0,"num_object_copies":11788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4239,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14525,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14515","reported_seq":17720,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:56.755096+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:29:56.755096+0000","last_peered":"2024-09-16T10:29:56.755096+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:29:56.755096+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24826822,"num_objects":6063,"num_object_clones":0,"num_object_copies":12126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2637,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14513,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1559,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3426}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14225","reported_seq":14476,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:57.047209+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:29:57.047209+0000","last_peered":"2024-09-16T10:29:57.047209+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:29:57.047209+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4200","ondisk_log_start":"42'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23794147,"num_objects":5810,"num_object_clones":0,"num_object_copies":11620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14224,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1735}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14845","reported_seq":15121,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:57.064378+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:29:57.064378+0000","last_peered":"2024-09-16T10:29:57.064378+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:29:57.064378+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24860073,"num_objects":6072,"num_object_clones":0,"num_object_copies":12144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14842,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1805}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14291","reported_seq":14523,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:57.059124+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:29:57.059124+0000","last_peered":"2024-09-16T10:29:57.059124+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:29:57.059124+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4200","ondisk_log_start":"42'4200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23958470,"num_objects":5851,"num_object_clones":0,"num_object_copies":11702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14289,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1758}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'14901","reported_seq":15143,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:29:56.958887+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:29:56.958887+0000","last_peered":"2024-09-16T10:29:56.958887+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:29:56.958887+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24557452,"num_objects":5999,"num_object_clones":0,"num_object_copies":11998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14898,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":390087182,"num_objects":95253,"num_object_clones":0,"num_object_copies":190506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95253,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211052,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10636,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991936512,"data_stored":991852495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160687,"ondisk_log_size":160687,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018970,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626828,"kb_used_data":328580,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93745012,"statfs":{"total":96636764160,"available":95994892288,"internally_reserved":0,"allocated":336465920,"data_stored":336197769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.51600000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691914,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605404,"kb_used_data":289104,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93766436,"statfs":{"total":96636764160,"available":96016830464,"internally_reserved":0,"allocated":296042496,"data_stored":295709311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":323874474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":6.9930000000000003},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.6600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.8519999999999999},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.21199999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215197,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754328,"kb_used_data":353616,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93617512,"statfs":{"total":96636764160,"available":95864332288,"internally_reserved":0,"allocated":362102784,"data_stored":361842319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":410316456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.0150000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.22900000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.0289999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":361627648,"data_stored":361624035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294944768,"data_stored":294872049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":335364096,"data_stored":335356411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:02.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:03.532 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:03.533 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:03.556 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-16T10:30:02.704420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":380891681,"num_objects":93010,"num_object_clones":0,"num_object_copies":186020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213299,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10777,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160935,"ondisk_log_size":160935,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1969304,"kb_used_data":954044,"kb_used_omap":12,"kb_used_meta":1015219,"kb_avail":281146216,"statfs":{"total":289910292480,"available":287893725184,"internally_reserved":0,"allocated":976941056,"data_stored":976079255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1039584258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,5,3,11,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26640384,"num_objects":-6504,"num_object_clones":0,"num_object_copies":-13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-395,"num_objects_misplaced":-74,"num_objects_unfound":0,"num_objects_dirty":-6504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6504,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":303104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005173"},"pg_stats":[{"pgid":"2.f","version":"46'14619","reported_seq":23084,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:02.088648+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:02.088648+0000","last_peered":"2024-09-16T10:30:02.088648+0000","last_clean":"2024-09-16T10:30:02.088648+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:02.088648+0000","last_undegraded":"2024-09-16T10:30:02.088648+0000","last_fullsized":"2024-09-16T10:30:02.088648+0000","mapping_epoch":45,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23957504,"num_objects":5849,"num_object_clones":0,"num_object_copies":11698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14619,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14688","reported_seq":14846,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:01.831758+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:01.831758+0000","last_peered":"2024-09-16T10:30:01.831758+0000","last_clean":"2024-09-16T10:30:01.831758+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:01.831758+0000","last_undegraded":"2024-09-16T10:30:01.831758+0000","last_fullsized":"2024-09-16T10:30:01.831758+0000","mapping_epoch":34,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":5774,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10901,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14847","reported_seq":15100,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:01.811562+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:01.811562+0000","last_peered":"2024-09-16T10:30:01.811562+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:01.811562+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23822360,"num_objects":5817,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10904,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1514}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14716","reported_seq":14776,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:01.755078+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:01.755078+0000","last_peered":"2024-09-16T10:30:01.755078+0000","last_clean":"2024-09-16T10:30:01.755078+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:01.755078+0000","last_undegraded":"2024-09-16T10:30:01.755078+0000","last_fullsized":"2024-09-16T10:30:01.755078+0000","mapping_epoch":34,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23535616,"num_objects":5746,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14716,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14688","reported_seq":14848,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:02.140439+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:02.140439+0000","last_peered":"2024-09-16T10:30:02.140439+0000","last_clean":"2024-09-16T10:30:02.140439+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:02.140439+0000","last_undegraded":"2024-09-16T10:30:02.140439+0000","last_fullsized":"2024-09-16T10:30:02.140439+0000","mapping_epoch":33,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23826432,"num_objects":5817,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11330,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14589","reported_seq":14922,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.095491+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:02.095491+0000","last_peered":"2024-09-16T10:30:02.095491+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:02.095491+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23683072,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15063","reported_seq":15472,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.102214+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:02.102214+0000","last_peered":"2024-09-16T10:30:02.102214+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:02.102214+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23990272,"num_objects":5857,"num_object_clones":0,"num_object_copies":11714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11305,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14616","reported_seq":14794,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:02.127444+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:02.127444+0000","last_peered":"2024-09-16T10:30:02.127444+0000","last_clean":"2024-09-16T10:30:02.127444+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:02.127444+0000","last_undegraded":"2024-09-16T10:30:02.127444+0000","last_fullsized":"2024-09-16T10:30:02.127444+0000","mapping_epoch":33,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":5774,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14598","reported_seq":23104,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:00.784515+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:00.784515+0000","last_peered":"2024-09-16T10:30:00.784515+0000","last_clean":"2024-09-16T10:30:00.784515+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:00.784515+0000","last_undegraded":"2024-09-16T10:30:00.784515+0000","last_fullsized":"2024-09-16T10:30:00.784515+0000","mapping_epoch":43,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24015814,"num_objects":5865,"num_object_clones":0,"num_object_copies":11730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14596,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15172","reported_seq":15448,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.128240+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:02.128240+0000","last_peered":"2024-09-16T10:30:02.128240+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:02.128240+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24036294,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15170,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1533}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14664","reported_seq":14860,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:01.741270+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:01.741270+0000","last_peered":"2024-09-16T10:30:01.741270+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:01.741270+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23573446,"num_objects":5757,"num_object_clones":0,"num_object_copies":11514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14662,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1524}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14647","reported_seq":18124,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:01.794106+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:01.794106+0000","last_peered":"2024-09-16T10:30:01.794106+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:01.794106+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286150,"num_objects":5931,"num_object_clones":0,"num_object_copies":11862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14645,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14359","reported_seq":14610,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.128271+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:02.128271+0000","last_peered":"2024-09-16T10:30:02.128271+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:02.128271+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23245283,"num_objects":5676,"num_object_clones":0,"num_object_copies":11352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14358,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14987","reported_seq":15263,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.140703+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:02.140703+0000","last_peered":"2024-09-16T10:30:02.140703+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:02.140703+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24278441,"num_objects":5930,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14984,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14425","reported_seq":14657,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.062412+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:02.062412+0000","last_peered":"2024-09-16T10:30:02.062412+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:02.062412+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409606,"num_objects":5717,"num_object_clones":0,"num_object_copies":11434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14423,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15054","reported_seq":15294,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.099452+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:02.099452+0000","last_peered":"2024-09-16T10:30:02.099452+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:02.099452+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23930764,"num_objects":5846,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15051,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1524}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":380891662,"num_objects":93008,"num_object_clones":0,"num_object_copies":186016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213297,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10776,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":973484032,"data_stored":973400015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018971,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626520,"kb_used_data":324172,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93745320,"statfs":{"total":96636764160,"available":95995207680,"internally_reserved":0,"allocated":331952128,"data_stored":331683977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.37},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.633}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.3610000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607404,"kb_used_data":282916,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93764436,"statfs":{"total":96636764160,"available":96014782464,"internally_reserved":0,"allocated":289705984,"data_stored":289372799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":332263082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.7910000000000004},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":5.758}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":5.8339999999999996},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":6.2789999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215198,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735380,"kb_used_data":346956,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93636460,"statfs":{"total":96636764160,"available":95883735040,"internally_reserved":0,"allocated":355282944,"data_stored":355022479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397733544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.0700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.85699999999999998},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.98799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354680832,"data_stored":354677219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":287973376,"data_stored":287900657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330829824,"data_stored":330822139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:03.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:04.129 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:04.132 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:04.153 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-16T10:30:02.704420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":380891681,"num_objects":93010,"num_object_clones":0,"num_object_copies":186020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213299,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10777,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160935,"ondisk_log_size":160935,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1969304,"kb_used_data":954044,"kb_used_omap":12,"kb_used_meta":1015219,"kb_avail":281146216,"statfs":{"total":289910292480,"available":287893725184,"internally_reserved":0,"allocated":976941056,"data_stored":976079255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1039584258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,5,3,11,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26640384,"num_objects":-6504,"num_object_clones":0,"num_object_copies":-13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-395,"num_objects_misplaced":-74,"num_objects_unfound":0,"num_objects_dirty":-6504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6504,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":303104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005173"},"pg_stats":[{"pgid":"2.f","version":"46'14619","reported_seq":23084,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:02.088648+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:02.088648+0000","last_peered":"2024-09-16T10:30:02.088648+0000","last_clean":"2024-09-16T10:30:02.088648+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:02.088648+0000","last_undegraded":"2024-09-16T10:30:02.088648+0000","last_fullsized":"2024-09-16T10:30:02.088648+0000","mapping_epoch":45,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23957504,"num_objects":5849,"num_object_clones":0,"num_object_copies":11698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14619,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14688","reported_seq":14846,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:01.831758+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:01.831758+0000","last_peered":"2024-09-16T10:30:01.831758+0000","last_clean":"2024-09-16T10:30:01.831758+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:01.831758+0000","last_undegraded":"2024-09-16T10:30:01.831758+0000","last_fullsized":"2024-09-16T10:30:01.831758+0000","mapping_epoch":34,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":5774,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10901,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14847","reported_seq":15100,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:01.811562+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:01.811562+0000","last_peered":"2024-09-16T10:30:01.811562+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:01.811562+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23822360,"num_objects":5817,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10904,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1514}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14716","reported_seq":14776,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:01.755078+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:01.755078+0000","last_peered":"2024-09-16T10:30:01.755078+0000","last_clean":"2024-09-16T10:30:01.755078+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:01.755078+0000","last_undegraded":"2024-09-16T10:30:01.755078+0000","last_fullsized":"2024-09-16T10:30:01.755078+0000","mapping_epoch":34,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23535616,"num_objects":5746,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14716,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14688","reported_seq":14848,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:02.140439+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:02.140439+0000","last_peered":"2024-09-16T10:30:02.140439+0000","last_clean":"2024-09-16T10:30:02.140439+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:02.140439+0000","last_undegraded":"2024-09-16T10:30:02.140439+0000","last_fullsized":"2024-09-16T10:30:02.140439+0000","mapping_epoch":33,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23826432,"num_objects":5817,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11330,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14589","reported_seq":14922,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.095491+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:02.095491+0000","last_peered":"2024-09-16T10:30:02.095491+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:02.095491+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23683072,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15063","reported_seq":15472,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.102214+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:02.102214+0000","last_peered":"2024-09-16T10:30:02.102214+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:02.102214+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23990272,"num_objects":5857,"num_object_clones":0,"num_object_copies":11714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11305,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14616","reported_seq":14794,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:02.127444+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:02.127444+0000","last_peered":"2024-09-16T10:30:02.127444+0000","last_clean":"2024-09-16T10:30:02.127444+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:02.127444+0000","last_undegraded":"2024-09-16T10:30:02.127444+0000","last_fullsized":"2024-09-16T10:30:02.127444+0000","mapping_epoch":33,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":5774,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14598","reported_seq":23104,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:00.784515+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:00.784515+0000","last_peered":"2024-09-16T10:30:00.784515+0000","last_clean":"2024-09-16T10:30:00.784515+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:00.784515+0000","last_undegraded":"2024-09-16T10:30:00.784515+0000","last_fullsized":"2024-09-16T10:30:00.784515+0000","mapping_epoch":43,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24015814,"num_objects":5865,"num_object_clones":0,"num_object_copies":11730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14596,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15172","reported_seq":15448,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.128240+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:02.128240+0000","last_peered":"2024-09-16T10:30:02.128240+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:02.128240+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24036294,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15170,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1533}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14664","reported_seq":14860,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:01.741270+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:01.741270+0000","last_peered":"2024-09-16T10:30:01.741270+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:01.741270+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23573446,"num_objects":5757,"num_object_clones":0,"num_object_copies":11514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14662,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1524}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14647","reported_seq":18124,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:01.794106+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:01.794106+0000","last_peered":"2024-09-16T10:30:01.794106+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:01.794106+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286150,"num_objects":5931,"num_object_clones":0,"num_object_copies":11862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14645,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14359","reported_seq":14610,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.128271+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:02.128271+0000","last_peered":"2024-09-16T10:30:02.128271+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:02.128271+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23245283,"num_objects":5676,"num_object_clones":0,"num_object_copies":11352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14358,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14987","reported_seq":15263,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.140703+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:02.140703+0000","last_peered":"2024-09-16T10:30:02.140703+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:02.140703+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24278441,"num_objects":5930,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14984,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14425","reported_seq":14657,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.062412+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:02.062412+0000","last_peered":"2024-09-16T10:30:02.062412+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:02.062412+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409606,"num_objects":5717,"num_object_clones":0,"num_object_copies":11434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14423,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15054","reported_seq":15294,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.099452+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:02.099452+0000","last_peered":"2024-09-16T10:30:02.099452+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:02.099452+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23930764,"num_objects":5846,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15051,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1524}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":380891662,"num_objects":93008,"num_object_clones":0,"num_object_copies":186016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213297,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10776,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":973484032,"data_stored":973400015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018971,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626520,"kb_used_data":324172,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93745320,"statfs":{"total":96636764160,"available":95995207680,"internally_reserved":0,"allocated":331952128,"data_stored":331683977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.37},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.633}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.3610000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607404,"kb_used_data":282916,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93764436,"statfs":{"total":96636764160,"available":96014782464,"internally_reserved":0,"allocated":289705984,"data_stored":289372799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":332263082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.7910000000000004},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":5.758}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":5.8339999999999996},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":6.2789999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215198,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735380,"kb_used_data":346956,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93636460,"statfs":{"total":96636764160,"available":95883735040,"internally_reserved":0,"allocated":355282944,"data_stored":355022479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397733544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.0700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.85699999999999998},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.98799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354680832,"data_stored":354677219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":287973376,"data_stored":287900657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330829824,"data_stored":330822139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:04.154 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:04.154 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'14847", 'reported_seq': 15100, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:01.811562+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:01.811562+0000', 'last_peered': '2024-09-16T10:30:01.811562+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:01.811562+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'4800", 'ondisk_log_start': "42'4800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23822360, 'num_objects': 5817, 'num_object_clones': 0, 'num_object_copies': 11634, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5817, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10904, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1514}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:04.154 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:04.154 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'14589", 'reported_seq': 14922, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:02.095491+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:02.095491+0000', 'last_peered': '2024-09-16T10:30:02.095491+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:02.095491+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'4500", 'ondisk_log_start': "42'4500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23683072, 'num_objects': 5782, 'num_object_clones': 0, 'num_object_copies': 11564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11214, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1648}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'15063", 'reported_seq': 15472, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:02.102214+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:02.102214+0000', 'last_peered': '2024-09-16T10:30:02.102214+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:02.102214+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'5000", 'ondisk_log_start': "42'5000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23990272, 'num_objects': 5857, 'num_object_clones': 0, 'num_object_copies': 11714, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11305, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1570}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'15172", 'reported_seq': 15448, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:02.128240+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:02.128240+0000', 'last_peered': '2024-09-16T10:30:02.128240+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:02.128240+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'5100", 'ondisk_log_start': "42'5100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24036294, 'num_objects': 5870, 'num_object_clones': 0, 'num_object_copies': 11740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5870, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15170, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1533}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'14664", 'reported_seq': 14860, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:01.741270+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:01.741270+0000', 'last_peered': '2024-09-16T10:30:01.741270+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:01.741270+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'4600", 'ondisk_log_start': "42'4600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23573446, 'num_objects': 5757, 'num_object_clones': 0, 'num_object_copies': 11514, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14662, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1524}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'14647", 'reported_seq': 18124, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:01.794106+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:01.794106+0000', 'last_peered': '2024-09-16T10:30:01.794106+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:01.794106+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'4600", 'ondisk_log_start': "42'4600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24286150, 'num_objects': 5931, 'num_object_clones': 0, 'num_object_copies': 11862, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2501, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14645, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1695, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3430}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'14359", 'reported_seq': 14610, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:02.128271+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:02.128271+0000', 'last_peered': '2024-09-16T10:30:02.128271+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:02.128271+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'4300", 'ondisk_log_start': "42'4300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23245283, 'num_objects': 5676, 'num_object_clones': 0, 'num_object_copies': 11352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14358, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:04.155 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:04.156 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'14987", 'reported_seq': 15263, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:02.140703+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:02.140703+0000', 'last_peered': '2024-09-16T10:30:02.140703+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:02.140703+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'4900", 'ondisk_log_start': "42'4900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24278441, 'num_objects': 5930, 'num_object_clones': 0, 'num_object_copies': 11860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14984, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1663}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:04.156 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:04.156 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'14425", 'reported_seq': 14657, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:02.062412+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:02.062412+0000', 'last_peered': '2024-09-16T10:30:02.062412+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:02.062412+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'4400", 'ondisk_log_start': "42'4400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23409606, 'num_objects': 5717, 'num_object_clones': 0, 'num_object_copies': 11434, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14423, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:04.156 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:04.156 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'15054", 'reported_seq': 15294, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:02.099452+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:02.099452+0000', 'last_peered': '2024-09-16T10:30:02.099452+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:02.099452+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'5000", 'ondisk_log_start': "42'5000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23930764, 'num_objects': 5846, 'num_object_clones': 0, 'num_object_copies': 11692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15051, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1524}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:04.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:04.844 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:04.844 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 40570/186020 objects degraded (21.809%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":493,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":93010,"data_bytes":380891681,"bytes_used":2016567296,"bytes_avail":287893725184,"bytes_total":289910292480,"degraded_objects":40570,"degraded_total":186020,"degraded_ratio":0.21809482851306311,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":315,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":368},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T10:29:36.693808+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:30:04.864 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:04.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:05.386 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:05.389 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:05.415 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-09-16T10:30:04.705191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":380891681,"num_objects":93010,"num_object_clones":0,"num_object_copies":186020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213299,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10777,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160935,"ondisk_log_size":160935,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1969304,"kb_used_data":954044,"kb_used_omap":12,"kb_used_meta":1015219,"kb_avail":281146216,"statfs":{"total":289910292480,"available":287893725184,"internally_reserved":0,"allocated":976941056,"data_stored":976079255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1039584258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,5,3,11,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18112512,"num_objects":-4422,"num_object_clones":0,"num_object_copies":-8844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":315,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004701"},"pg_stats":[{"pgid":"2.f","version":"46'14619","reported_seq":23084,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:02.088648+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:02.088648+0000","last_peered":"2024-09-16T10:30:02.088648+0000","last_clean":"2024-09-16T10:30:02.088648+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:02.088648+0000","last_undegraded":"2024-09-16T10:30:02.088648+0000","last_fullsized":"2024-09-16T10:30:02.088648+0000","mapping_epoch":45,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23957504,"num_objects":5849,"num_object_clones":0,"num_object_copies":11698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14619,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14688","reported_seq":14846,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:01.831758+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:01.831758+0000","last_peered":"2024-09-16T10:30:01.831758+0000","last_clean":"2024-09-16T10:30:01.831758+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:01.831758+0000","last_undegraded":"2024-09-16T10:30:01.831758+0000","last_fullsized":"2024-09-16T10:30:01.831758+0000","mapping_epoch":34,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":5774,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10901,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14847","reported_seq":15100,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:01.811562+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:01.811562+0000","last_peered":"2024-09-16T10:30:01.811562+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:01.811562+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23822360,"num_objects":5817,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10904,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1514}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14716","reported_seq":14776,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:01.755078+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:01.755078+0000","last_peered":"2024-09-16T10:30:01.755078+0000","last_clean":"2024-09-16T10:30:01.755078+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:01.755078+0000","last_undegraded":"2024-09-16T10:30:01.755078+0000","last_fullsized":"2024-09-16T10:30:01.755078+0000","mapping_epoch":34,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23535616,"num_objects":5746,"num_object_clones":0,"num_object_copies":11492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14716,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14688","reported_seq":14848,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:02.140439+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:02.140439+0000","last_peered":"2024-09-16T10:30:02.140439+0000","last_clean":"2024-09-16T10:30:02.140439+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:02.140439+0000","last_undegraded":"2024-09-16T10:30:02.140439+0000","last_fullsized":"2024-09-16T10:30:02.140439+0000","mapping_epoch":33,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23826432,"num_objects":5817,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11330,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14589","reported_seq":14922,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.095491+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:02.095491+0000","last_peered":"2024-09-16T10:30:02.095491+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:02.095491+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23683072,"num_objects":5782,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15063","reported_seq":15472,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.102214+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:02.102214+0000","last_peered":"2024-09-16T10:30:02.102214+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:02.102214+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23990272,"num_objects":5857,"num_object_clones":0,"num_object_copies":11714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11305,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14616","reported_seq":14794,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:02.127444+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:02.127444+0000","last_peered":"2024-09-16T10:30:02.127444+0000","last_clean":"2024-09-16T10:30:02.127444+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:02.127444+0000","last_undegraded":"2024-09-16T10:30:02.127444+0000","last_fullsized":"2024-09-16T10:30:02.127444+0000","mapping_epoch":33,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23650304,"num_objects":5774,"num_object_clones":0,"num_object_copies":11548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10419,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14598","reported_seq":23104,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:00.784515+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:00.784515+0000","last_peered":"2024-09-16T10:30:00.784515+0000","last_clean":"2024-09-16T10:30:00.784515+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:00.784515+0000","last_undegraded":"2024-09-16T10:30:00.784515+0000","last_fullsized":"2024-09-16T10:30:00.784515+0000","mapping_epoch":43,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24015814,"num_objects":5865,"num_object_clones":0,"num_object_copies":11730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14596,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15172","reported_seq":15448,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.128240+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:02.128240+0000","last_peered":"2024-09-16T10:30:02.128240+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:02.128240+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24036294,"num_objects":5870,"num_object_clones":0,"num_object_copies":11740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15170,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1533}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14664","reported_seq":14860,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:01.741270+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:01.741270+0000","last_peered":"2024-09-16T10:30:01.741270+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:01.741270+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23573446,"num_objects":5757,"num_object_clones":0,"num_object_copies":11514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14662,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1524}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14647","reported_seq":18124,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:01.794106+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:01.794106+0000","last_peered":"2024-09-16T10:30:01.794106+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:01.794106+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24286150,"num_objects":5931,"num_object_clones":0,"num_object_copies":11862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14645,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3430}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14359","reported_seq":14610,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.128271+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:02.128271+0000","last_peered":"2024-09-16T10:30:02.128271+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:02.128271+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4300","ondisk_log_start":"42'4300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23245283,"num_objects":5676,"num_object_clones":0,"num_object_copies":11352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14358,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'14987","reported_seq":15263,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.140703+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:02.140703+0000","last_peered":"2024-09-16T10:30:02.140703+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:02.140703+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24278441,"num_objects":5930,"num_object_clones":0,"num_object_copies":11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14984,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1663}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14425","reported_seq":14657,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.062412+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:02.062412+0000","last_peered":"2024-09-16T10:30:02.062412+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:02.062412+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23409606,"num_objects":5717,"num_object_clones":0,"num_object_copies":11434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14423,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15054","reported_seq":15294,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:02.099452+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:02.099452+0000","last_peered":"2024-09-16T10:30:02.099452+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:02.099452+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23930764,"num_objects":5846,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15051,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1524}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":380891662,"num_objects":93008,"num_object_clones":0,"num_object_copies":186016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":213297,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10776,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":973484032,"data_stored":973400015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018971,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626520,"kb_used_data":324172,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93745320,"statfs":{"total":96636764160,"available":95995207680,"internally_reserved":0,"allocated":331952128,"data_stored":331683977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.512,"15min":1.0780000000000001},"min":{"1min":0.19,"5min":0.19,"15min":0.184},"max":{"1min":3.1110000000000002,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.37},{"interface":"front","average":{"1min":1.26,"5min":2.407,"15min":1.4019999999999999},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":3.7090000000000001,"5min":8.9879999999999995,"15min":8.9879999999999995},"last":3.633}]},{"osd":1,"last update":"Mon Sep 16 10:29:01 2024","interfaces":[{"interface":"back","average":{"1min":2.7029999999999998,"5min":2.8279999999999998,"15min":2.8740000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2059999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":3.1960000000000002,"5min":2.7469999999999999,"15min":2.4089999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":17.914000000000001,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.3610000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607404,"kb_used_data":282916,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93764436,"statfs":{"total":96636764160,"available":96014782464,"internally_reserved":0,"allocated":289705984,"data_stored":289372799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":332263082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.7910000000000004},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":5.758}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":5.8339999999999996},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":6.2789999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215198,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735380,"kb_used_data":346956,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93636460,"statfs":{"total":96636764160,"available":95883735040,"internally_reserved":0,"allocated":355282944,"data_stored":355022479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397733544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.0700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":0.85699999999999998},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.98799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":354680832,"data_stored":354677219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":287973376,"data_stored":287900657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":330829824,"data_stored":330822139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:08.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:08.983 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:08.984 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:09.006 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-16T10:30:08.706115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":371577377,"num_objects":90736,"num_object_clones":0,"num_object_copies":181472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40450,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90736,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215573,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10897,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161009,"ondisk_log_size":161009,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1966296,"kb_used_data":934644,"kb_used_omap":12,"kb_used_meta":1031603,"kb_avail":281149224,"statfs":{"total":289910292480,"available":287896805376,"internally_reserved":0,"allocated":957075456,"data_stored":956213655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1056361474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,8,8,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26918912,"num_objects":-6572,"num_object_clones":0,"num_object_copies":-13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6572,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003963"},"pg_stats":[{"pgid":"2.f","version":"46'14771","reported_seq":23235,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:07.123440+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:07.123440+0000","last_peered":"2024-09-16T10:30:07.123440+0000","last_clean":"2024-09-16T10:30:07.123440+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:07.123440+0000","last_undegraded":"2024-09-16T10:30:07.123440+0000","last_fullsized":"2024-09-16T10:30:07.123440+0000","mapping_epoch":45,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23334912,"num_objects":5697,"num_object_clones":0,"num_object_copies":11394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14771,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14843","reported_seq":15003,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:06.782381+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:06.782381+0000","last_peered":"2024-09-16T10:30:06.782381+0000","last_clean":"2024-09-16T10:30:06.782381+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:06.782381+0000","last_undegraded":"2024-09-16T10:30:06.782381+0000","last_fullsized":"2024-09-16T10:30:06.782381+0000","mapping_epoch":34,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015424,"num_objects":5619,"num_object_clones":0,"num_object_copies":11238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11056,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14982","reported_seq":15232,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:06.835515+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:06.835515+0000","last_peered":"2024-09-16T10:30:06.835515+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:06.835515+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23269400,"num_objects":5682,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11039,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14854","reported_seq":14914,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:06.818852+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:06.818852+0000","last_peered":"2024-09-16T10:30:06.818852+0000","last_clean":"2024-09-16T10:30:06.818852+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:06.818852+0000","last_undegraded":"2024-09-16T10:30:06.818852+0000","last_fullsized":"2024-09-16T10:30:06.818852+0000","mapping_epoch":34,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22970368,"num_objects":5608,"num_object_clones":0,"num_object_copies":11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14854,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14821","reported_seq":14981,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:07.194038+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:07.194038+0000","last_peered":"2024-09-16T10:30:07.194038+0000","last_clean":"2024-09-16T10:30:07.194038+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:07.194038+0000","last_undegraded":"2024-09-16T10:30:07.194038+0000","last_fullsized":"2024-09-16T10:30:07.194038+0000","mapping_epoch":33,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23281664,"num_objects":5684,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11463,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14746","reported_seq":15079,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.198051+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:07.198051+0000","last_peered":"2024-09-16T10:30:07.198051+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:07.198051+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23040000,"num_objects":5625,"num_object_clones":0,"num_object_copies":11250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11371,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15199","reported_seq":15608,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.215677+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:07.215677+0000","last_peered":"2024-09-16T10:30:07.215677+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:07.215677+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23433216,"num_objects":5721,"num_object_clones":0,"num_object_copies":11442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11441,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14758","reported_seq":14936,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:07.183826+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:07.183826+0000","last_peered":"2024-09-16T10:30:07.183826+0000","last_clean":"2024-09-16T10:30:07.183826+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:07.183826+0000","last_undegraded":"2024-09-16T10:30:07.183826+0000","last_fullsized":"2024-09-16T10:30:07.183826+0000","mapping_epoch":33,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":5632,"num_object_clones":0,"num_object_copies":11264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14740","reported_seq":23246,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:05.920386+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:05.920386+0000","last_peered":"2024-09-16T10:30:05.920386+0000","last_clean":"2024-09-16T10:30:05.920386+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:05.920386+0000","last_undegraded":"2024-09-16T10:30:05.920386+0000","last_fullsized":"2024-09-16T10:30:05.920386+0000","mapping_epoch":43,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23434182,"num_objects":5723,"num_object_clones":0,"num_object_copies":11446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14738,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15290","reported_seq":15565,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.215190+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:07.215190+0000","last_peered":"2024-09-16T10:30:07.215190+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:07.215190+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23552966,"num_objects":5752,"num_object_clones":0,"num_object_copies":11504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15288,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1415}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14787","reported_seq":14984,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:06.771985+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:06.771985+0000","last_peered":"2024-09-16T10:30:06.771985+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:06.771985+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23069638,"num_objects":5634,"num_object_clones":0,"num_object_copies":11268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14785,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1401}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14787","reported_seq":18503,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:06.847406+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:06.847406+0000","last_peered":"2024-09-16T10:30:06.847406+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:06.847406+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23712710,"num_objects":5791,"num_object_clones":0,"num_object_copies":11582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14785,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3410}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14493","reported_seq":14743,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.242664+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:07.242664+0000","last_peered":"2024-09-16T10:30:07.242664+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:07.242664+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22696419,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14492,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15139","reported_seq":15415,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.228185+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:07.228185+0000","last_peered":"2024-09-16T10:30:07.228185+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:07.228185+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23655849,"num_objects":5778,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15136,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14583","reported_seq":14815,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.179838+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:07.179838+0000","last_peered":"2024-09-16T10:30:07.179838+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:07.179838+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22762438,"num_objects":5559,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14581,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15213","reported_seq":15455,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.167949+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:07.167949+0000","last_peered":"2024-09-16T10:30:07.167949+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:07.167949+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23279500,"num_objects":5687,"num_object_clones":0,"num_object_copies":11374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15210,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":371577358,"num_objects":90734,"num_object_clones":0,"num_object_copies":181468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40450,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215571,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10896,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":954896384,"data_stored":954812367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161006,"ondisk_log_size":161006,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626168,"kb_used_data":319724,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93745672,"statfs":{"total":96636764160,"available":95995568128,"internally_reserved":0,"allocated":327397376,"data_stored":327129225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":313781936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":13.74},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":1.6859999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.4630000000000001},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":9.0429999999999993}]}]},{"osd":1,"up_from":40,"seq":171798691916,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603384,"kb_used_data":274796,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93768456,"statfs":{"total":96636764160,"available":96018898944,"internally_reserved":0,"allocated":281391104,"data_stored":281057919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":336457386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.8620000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.833}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.784},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":3.5110000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215199,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736744,"kb_used_data":340124,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93635096,"statfs":{"total":96636764160,"available":95882338304,"internally_reserved":0,"allocated":348286976,"data_stored":348026511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":406122152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":2.0129999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":1.6140000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.47999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347590656,"data_stored":347587043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280936448,"data_stored":280863729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":326369280,"data_stored":326361595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:09.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:09.565 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:09.567 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:09.590 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-16T10:30:08.706115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":371577377,"num_objects":90736,"num_object_clones":0,"num_object_copies":181472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40450,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90736,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215573,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10897,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161009,"ondisk_log_size":161009,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1966296,"kb_used_data":934644,"kb_used_omap":12,"kb_used_meta":1031603,"kb_avail":281149224,"statfs":{"total":289910292480,"available":287896805376,"internally_reserved":0,"allocated":957075456,"data_stored":956213655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1056361474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,8,8,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26918912,"num_objects":-6572,"num_object_clones":0,"num_object_copies":-13144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-435,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6572,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003963"},"pg_stats":[{"pgid":"2.f","version":"46'14771","reported_seq":23235,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:07.123440+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:07.123440+0000","last_peered":"2024-09-16T10:30:07.123440+0000","last_clean":"2024-09-16T10:30:07.123440+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:07.123440+0000","last_undegraded":"2024-09-16T10:30:07.123440+0000","last_fullsized":"2024-09-16T10:30:07.123440+0000","mapping_epoch":45,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23334912,"num_objects":5697,"num_object_clones":0,"num_object_copies":11394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14771,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14843","reported_seq":15003,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:06.782381+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:06.782381+0000","last_peered":"2024-09-16T10:30:06.782381+0000","last_clean":"2024-09-16T10:30:06.782381+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:06.782381+0000","last_undegraded":"2024-09-16T10:30:06.782381+0000","last_fullsized":"2024-09-16T10:30:06.782381+0000","mapping_epoch":34,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015424,"num_objects":5619,"num_object_clones":0,"num_object_copies":11238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11056,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14982","reported_seq":15232,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:06.835515+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:06.835515+0000","last_peered":"2024-09-16T10:30:06.835515+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:06.835515+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23269400,"num_objects":5682,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11039,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14854","reported_seq":14914,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:06.818852+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:06.818852+0000","last_peered":"2024-09-16T10:30:06.818852+0000","last_clean":"2024-09-16T10:30:06.818852+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:06.818852+0000","last_undegraded":"2024-09-16T10:30:06.818852+0000","last_fullsized":"2024-09-16T10:30:06.818852+0000","mapping_epoch":34,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22970368,"num_objects":5608,"num_object_clones":0,"num_object_copies":11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14854,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14821","reported_seq":14981,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:07.194038+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:07.194038+0000","last_peered":"2024-09-16T10:30:07.194038+0000","last_clean":"2024-09-16T10:30:07.194038+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:07.194038+0000","last_undegraded":"2024-09-16T10:30:07.194038+0000","last_fullsized":"2024-09-16T10:30:07.194038+0000","mapping_epoch":33,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23281664,"num_objects":5684,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11463,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14746","reported_seq":15079,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.198051+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:07.198051+0000","last_peered":"2024-09-16T10:30:07.198051+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:07.198051+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23040000,"num_objects":5625,"num_object_clones":0,"num_object_copies":11250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11371,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15199","reported_seq":15608,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.215677+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:07.215677+0000","last_peered":"2024-09-16T10:30:07.215677+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:07.215677+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23433216,"num_objects":5721,"num_object_clones":0,"num_object_copies":11442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11441,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14758","reported_seq":14936,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:07.183826+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:07.183826+0000","last_peered":"2024-09-16T10:30:07.183826+0000","last_clean":"2024-09-16T10:30:07.183826+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:07.183826+0000","last_undegraded":"2024-09-16T10:30:07.183826+0000","last_fullsized":"2024-09-16T10:30:07.183826+0000","mapping_epoch":33,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":5632,"num_object_clones":0,"num_object_copies":11264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14740","reported_seq":23246,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:05.920386+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:05.920386+0000","last_peered":"2024-09-16T10:30:05.920386+0000","last_clean":"2024-09-16T10:30:05.920386+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:05.920386+0000","last_undegraded":"2024-09-16T10:30:05.920386+0000","last_fullsized":"2024-09-16T10:30:05.920386+0000","mapping_epoch":43,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23434182,"num_objects":5723,"num_object_clones":0,"num_object_copies":11446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14738,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15290","reported_seq":15565,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.215190+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:07.215190+0000","last_peered":"2024-09-16T10:30:07.215190+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:07.215190+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23552966,"num_objects":5752,"num_object_clones":0,"num_object_copies":11504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15288,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1415}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14787","reported_seq":14984,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:06.771985+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:06.771985+0000","last_peered":"2024-09-16T10:30:06.771985+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:06.771985+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23069638,"num_objects":5634,"num_object_clones":0,"num_object_copies":11268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14785,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1401}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14787","reported_seq":18503,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:06.847406+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:06.847406+0000","last_peered":"2024-09-16T10:30:06.847406+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:06.847406+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23712710,"num_objects":5791,"num_object_clones":0,"num_object_copies":11582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14785,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3410}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14493","reported_seq":14743,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.242664+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:07.242664+0000","last_peered":"2024-09-16T10:30:07.242664+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:07.242664+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22696419,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14492,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15139","reported_seq":15415,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.228185+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:07.228185+0000","last_peered":"2024-09-16T10:30:07.228185+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:07.228185+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23655849,"num_objects":5778,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15136,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14583","reported_seq":14815,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.179838+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:07.179838+0000","last_peered":"2024-09-16T10:30:07.179838+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:07.179838+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22762438,"num_objects":5559,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14581,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15213","reported_seq":15455,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.167949+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:07.167949+0000","last_peered":"2024-09-16T10:30:07.167949+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:07.167949+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23279500,"num_objects":5687,"num_object_clones":0,"num_object_copies":11374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15210,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":371577358,"num_objects":90734,"num_object_clones":0,"num_object_copies":181468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40450,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215571,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10896,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":954896384,"data_stored":954812367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161006,"ondisk_log_size":161006,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626168,"kb_used_data":319724,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93745672,"statfs":{"total":96636764160,"available":95995568128,"internally_reserved":0,"allocated":327397376,"data_stored":327129225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":313781936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":13.74},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":1.6859999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.4630000000000001},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":9.0429999999999993}]}]},{"osd":1,"up_from":40,"seq":171798691916,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603384,"kb_used_data":274796,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93768456,"statfs":{"total":96636764160,"available":96018898944,"internally_reserved":0,"allocated":281391104,"data_stored":281057919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":336457386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.8620000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.833}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.784},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":3.5110000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215199,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736744,"kb_used_data":340124,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93635096,"statfs":{"total":96636764160,"available":95882338304,"internally_reserved":0,"allocated":348286976,"data_stored":348026511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":406122152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":2.0129999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":1.6140000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.47999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347590656,"data_stored":347587043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280936448,"data_stored":280863729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":326369280,"data_stored":326361595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:09.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:09.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'14982", 'reported_seq': 15232, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:06.835515+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:06.835515+0000', 'last_peered': '2024-09-16T10:30:06.835515+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:06.835515+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'4900", 'ondisk_log_start': "42'4900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23269400, 'num_objects': 5682, 'num_object_clones': 0, 'num_object_copies': 11364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5682, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11039, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:09.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:09.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'14746", 'reported_seq': 15079, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:07.198051+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:07.198051+0000', 'last_peered': '2024-09-16T10:30:07.198051+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:07.198051+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23040000, 'num_objects': 5625, 'num_object_clones': 0, 'num_object_copies': 11250, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5625, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11371, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1491}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:09.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:09.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'15199", 'reported_seq': 15608, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:07.215677+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:07.215677+0000', 'last_peered': '2024-09-16T10:30:07.215677+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:07.215677+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'5100", 'ondisk_log_start': "42'5100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23433216, 'num_objects': 5721, 'num_object_clones': 0, 'num_object_copies': 11442, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11441, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1434}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'15290", 'reported_seq': 15565, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:07.215190+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:07.215190+0000', 'last_peered': '2024-09-16T10:30:07.215190+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:07.215190+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'5200", 'ondisk_log_start': "42'5200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23552966, 'num_objects': 5752, 'num_object_clones': 0, 'num_object_copies': 11504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15288, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1415}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'14787", 'reported_seq': 14984, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:06.771985+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:06.771985+0000', 'last_peered': '2024-09-16T10:30:06.771985+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:06.771985+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23069638, 'num_objects': 5634, 'num_object_clones': 0, 'num_object_copies': 11268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14785, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1401}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'14787", 'reported_seq': 18503, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:06.847406+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:06.847406+0000', 'last_peered': '2024-09-16T10:30:06.847406+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:06.847406+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23712710, 'num_objects': 5791, 'num_object_clones': 0, 'num_object_copies': 11582, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2381, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14785, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1815, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3410}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'14493", 'reported_seq': 14743, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:07.242664+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:07.242664+0000', 'last_peered': '2024-09-16T10:30:07.242664+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:07.242664+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'4400", 'ondisk_log_start': "42'4400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22696419, 'num_objects': 5542, 'num_object_clones': 0, 'num_object_copies': 11084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14492, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1467}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'15139", 'reported_seq': 15415, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:07.228185+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:07.228185+0000', 'last_peered': '2024-09-16T10:30:07.228185+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:07.228185+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'5100", 'ondisk_log_start': "42'5100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23655849, 'num_objects': 5778, 'num_object_clones': 0, 'num_object_copies': 11556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5778, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15136, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1511}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:09.592 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:09.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'14583", 'reported_seq': 14815, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:07.179838+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:07.179838+0000', 'last_peered': '2024-09-16T10:30:07.179838+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:07.179838+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'4500", 'ondisk_log_start': "42'4500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22762438, 'num_objects': 5559, 'num_object_clones': 0, 'num_object_copies': 11118, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14581, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1466}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:09.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:09.593 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'15213", 'reported_seq': 15455, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:07.167949+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:07.167949+0000', 'last_peered': '2024-09-16T10:30:07.167949+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:07.167949+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'5200", 'ondisk_log_start': "42'5200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23279500, 'num_objects': 5687, 'num_object_clones': 0, 'num_object_copies': 11374, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5687, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15210, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1365}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:09.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:10.294 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:10.294 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 40450/181472 objects degraded (22.290%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":499,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":90736,"data_bytes":371577377,"bytes_used":2013487104,"bytes_avail":287896805376,"bytes_total":289910292480,"degraded_objects":40450,"degraded_total":181472,"degraded_ratio":0.22289940045847292,"recovering_objects_per_sec":36,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":435,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":547},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T10:29:36.693808+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:30:10.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:10.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:10.895 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:10.898 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:10.922 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-16T10:30:10.706600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":371577377,"num_objects":90736,"num_object_clones":0,"num_object_copies":181472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40450,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90736,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215573,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10897,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161009,"ondisk_log_size":161009,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1966296,"kb_used_data":934644,"kb_used_omap":12,"kb_used_meta":1031603,"kb_avail":281149224,"statfs":{"total":289910292480,"available":287896805376,"internally_reserved":0,"allocated":957075456,"data_stored":956213655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1056361474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,8,8,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18509824,"num_objects":-4519,"num_object_clones":0,"num_object_copies":-9038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4519,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003504"},"pg_stats":[{"pgid":"2.f","version":"46'14771","reported_seq":23235,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:07.123440+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:07.123440+0000","last_peered":"2024-09-16T10:30:07.123440+0000","last_clean":"2024-09-16T10:30:07.123440+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:07.123440+0000","last_undegraded":"2024-09-16T10:30:07.123440+0000","last_fullsized":"2024-09-16T10:30:07.123440+0000","mapping_epoch":45,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23334912,"num_objects":5697,"num_object_clones":0,"num_object_copies":11394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14771,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'14843","reported_seq":15003,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:06.782381+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:06.782381+0000","last_peered":"2024-09-16T10:30:06.782381+0000","last_clean":"2024-09-16T10:30:06.782381+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:06.782381+0000","last_undegraded":"2024-09-16T10:30:06.782381+0000","last_fullsized":"2024-09-16T10:30:06.782381+0000","mapping_epoch":34,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23015424,"num_objects":5619,"num_object_clones":0,"num_object_copies":11238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11056,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'14982","reported_seq":15232,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:06.835515+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:06.835515+0000","last_peered":"2024-09-16T10:30:06.835515+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:06.835515+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23269400,"num_objects":5682,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11039,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14854","reported_seq":14914,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:06.818852+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:06.818852+0000","last_peered":"2024-09-16T10:30:06.818852+0000","last_clean":"2024-09-16T10:30:06.818852+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:06.818852+0000","last_undegraded":"2024-09-16T10:30:06.818852+0000","last_fullsized":"2024-09-16T10:30:06.818852+0000","mapping_epoch":34,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22970368,"num_objects":5608,"num_object_clones":0,"num_object_copies":11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14854,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14821","reported_seq":14981,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:07.194038+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:07.194038+0000","last_peered":"2024-09-16T10:30:07.194038+0000","last_clean":"2024-09-16T10:30:07.194038+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:07.194038+0000","last_undegraded":"2024-09-16T10:30:07.194038+0000","last_fullsized":"2024-09-16T10:30:07.194038+0000","mapping_epoch":33,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23281664,"num_objects":5684,"num_object_clones":0,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11463,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14746","reported_seq":15079,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.198051+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:07.198051+0000","last_peered":"2024-09-16T10:30:07.198051+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:07.198051+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23040000,"num_objects":5625,"num_object_clones":0,"num_object_copies":11250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11371,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1491}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15199","reported_seq":15608,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.215677+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:07.215677+0000","last_peered":"2024-09-16T10:30:07.215677+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:07.215677+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23433216,"num_objects":5721,"num_object_clones":0,"num_object_copies":11442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11441,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1434}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14758","reported_seq":14936,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:07.183826+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:07.183826+0000","last_peered":"2024-09-16T10:30:07.183826+0000","last_clean":"2024-09-16T10:30:07.183826+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:07.183826+0000","last_undegraded":"2024-09-16T10:30:07.183826+0000","last_fullsized":"2024-09-16T10:30:07.183826+0000","mapping_epoch":33,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":5632,"num_object_clones":0,"num_object_copies":11264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14740","reported_seq":23246,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:05.920386+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:05.920386+0000","last_peered":"2024-09-16T10:30:05.920386+0000","last_clean":"2024-09-16T10:30:05.920386+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:05.920386+0000","last_undegraded":"2024-09-16T10:30:05.920386+0000","last_fullsized":"2024-09-16T10:30:05.920386+0000","mapping_epoch":43,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23434182,"num_objects":5723,"num_object_clones":0,"num_object_copies":11446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14738,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15290","reported_seq":15565,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.215190+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:07.215190+0000","last_peered":"2024-09-16T10:30:07.215190+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:07.215190+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23552966,"num_objects":5752,"num_object_clones":0,"num_object_copies":11504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15288,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1415}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14787","reported_seq":14984,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:06.771985+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:06.771985+0000","last_peered":"2024-09-16T10:30:06.771985+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:06.771985+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23069638,"num_objects":5634,"num_object_clones":0,"num_object_copies":11268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14785,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1401}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.972968+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:02.972968+0000","last_peered":"2024-09-16T10:29:02.972968+0000","last_clean":"2024-09-16T10:29:02.972968+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:02.972968+0000","last_undegraded":"2024-09-16T10:29:02.972968+0000","last_fullsized":"2024-09-16T10:29:02.972968+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14787","reported_seq":18503,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:06.847406+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:06.847406+0000","last_peered":"2024-09-16T10:30:06.847406+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:06.847406+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23712710,"num_objects":5791,"num_object_clones":0,"num_object_copies":11582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2381,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14785,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3410}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14493","reported_seq":14743,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.242664+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:07.242664+0000","last_peered":"2024-09-16T10:30:07.242664+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:07.242664+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4400","ondisk_log_start":"42'4400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22696419,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14492,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.989435+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:02.989435+0000","last_peered":"2024-09-16T10:29:02.989435+0000","last_clean":"2024-09-16T10:29:02.989435+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:02.989435+0000","last_undegraded":"2024-09-16T10:29:02.989435+0000","last_fullsized":"2024-09-16T10:29:02.989435+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15139","reported_seq":15415,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.228185+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:07.228185+0000","last_peered":"2024-09-16T10:30:07.228185+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:07.228185+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23655849,"num_objects":5778,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15136,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14583","reported_seq":14815,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.179838+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:07.179838+0000","last_peered":"2024-09-16T10:30:07.179838+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:07.179838+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4500","ondisk_log_start":"42'4500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22762438,"num_objects":5559,"num_object_clones":0,"num_object_copies":11118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14581,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1466}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":57,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973127+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:02.973127+0000","last_peered":"2024-09-16T10:29:02.973127+0000","last_clean":"2024-09-16T10:29:02.973127+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:02.973127+0000","last_undegraded":"2024-09-16T10:29:02.973127+0000","last_fullsized":"2024-09-16T10:29:02.973127+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15213","reported_seq":15455,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:07.167949+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:07.167949+0000","last_peered":"2024-09-16T10:30:07.167949+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:07.167949+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23279500,"num_objects":5687,"num_object_clones":0,"num_object_copies":11374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15210,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1365}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":371577358,"num_objects":90734,"num_object_clones":0,"num_object_copies":181468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40450,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215571,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10896,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":954896384,"data_stored":954812367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161006,"ondisk_log_size":161006,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018972,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626168,"kb_used_data":319724,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93745672,"statfs":{"total":96636764160,"available":95995568128,"internally_reserved":0,"allocated":327397376,"data_stored":327129225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":313781936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":13.74},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":1.6859999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.4630000000000001},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":9.0429999999999993}]}]},{"osd":1,"up_from":40,"seq":171798691916,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":603384,"kb_used_data":274796,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93768456,"statfs":{"total":96636764160,"available":96018898944,"internally_reserved":0,"allocated":281391104,"data_stored":281057919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":336457386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.8620000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.833}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.784},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":3.5110000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215199,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736744,"kb_used_data":340124,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93635096,"statfs":{"total":96636764160,"available":95882338304,"internally_reserved":0,"allocated":348286976,"data_stored":348026511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":406122152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9750000000000001,"5min":2.7709999999999999,"15min":2.7970000000000002},"min":{"1min":0.214,"5min":0.184,"15min":0.184},"max":{"1min":13.795999999999999,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.302,"5min":2.2040000000000002,"15min":2.0830000000000002},"min":{"1min":0.246,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.2960000000000003,"5min":12.999000000000001,"15min":12.999000000000001},"last":2.0129999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":2.7549999999999999,"15min":1.3200000000000001},"min":{"1min":0.32300000000000001,"5min":0.21199999999999999,"15min":0.182},"max":{"1min":8.9689999999999994,"5min":18.093,"15min":18.093},"last":1.6140000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":2.4369999999999998,"15min":1.587},"min":{"1min":0.25,"5min":0.214,"15min":0.192},"max":{"1min":13.752000000000001,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.47999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347590656,"data_stored":347587043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280936448,"data_stored":280863729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":326369280,"data_stored":326361595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:13.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:14.463 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:14.464 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:14.483 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-16T10:30:12.707083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":362336801,"num_objects":88480,"num_object_clones":0,"num_object_copies":176960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217829,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11025,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160665,"ondisk_log_size":160665,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1948284,"kb_used_data":916612,"kb_used_omap":12,"kb_used_meta":1031603,"kb_avail":281167236,"statfs":{"total":289910292480,"available":287915249664,"internally_reserved":0,"allocated":938610688,"data_stored":937748887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1056361474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,7,6,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27750400,"num_objects":-6775,"num_object_clones":0,"num_object_copies":-13550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6775,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"2.f","version":"46'14918","reported_seq":23383,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:12.278956+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:12.278956+0000","last_peered":"2024-09-16T10:30:12.278956+0000","last_clean":"2024-09-16T10:30:12.278956+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:12.278956+0000","last_undegraded":"2024-09-16T10:30:12.278956+0000","last_fullsized":"2024-09-16T10:30:12.278956+0000","mapping_epoch":45,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22732800,"num_objects":5550,"num_object_clones":0,"num_object_copies":11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14918,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15001","reported_seq":15161,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:11.762311+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:11.762311+0000","last_peered":"2024-09-16T10:30:11.762311+0000","last_clean":"2024-09-16T10:30:11.762311+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:11.762311+0000","last_undegraded":"2024-09-16T10:30:11.762311+0000","last_fullsized":"2024-09-16T10:30:11.762311+0000","mapping_epoch":34,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22368256,"num_objects":5461,"num_object_clones":0,"num_object_copies":10922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15122","reported_seq":15376,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:11.782911+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:11.782911+0000","last_peered":"2024-09-16T10:30:11.782911+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:11.782911+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22695960,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11179,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14996","reported_seq":15055,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:11.829831+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:11.829831+0000","last_peered":"2024-09-16T10:30:11.829831+0000","last_clean":"2024-09-16T10:30:11.829831+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:11.829831+0000","last_undegraded":"2024-09-16T10:30:11.829831+0000","last_fullsized":"2024-09-16T10:30:11.829831+0000","mapping_epoch":34,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22388736,"num_objects":5466,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14996,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14963","reported_seq":15123,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:12.276672+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:12.276672+0000","last_peered":"2024-09-16T10:30:12.276672+0000","last_clean":"2024-09-16T10:30:12.276672+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:12.276672+0000","last_undegraded":"2024-09-16T10:30:12.276672+0000","last_fullsized":"2024-09-16T10:30:12.276672+0000","mapping_epoch":33,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22700032,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14881","reported_seq":15214,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.167726+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:12.167726+0000","last_peered":"2024-09-16T10:30:12.167726+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:12.167726+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22487040,"num_objects":5490,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15334","reported_seq":15743,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.201115+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:12.201115+0000","last_peered":"2024-09-16T10:30:12.201115+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:12.201115+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22880256,"num_objects":5586,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14903","reported_seq":15079,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:12.267815+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:12.267815+0000","last_peered":"2024-09-16T10:30:12.267815+0000","last_clean":"2024-09-16T10:30:12.267815+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:12.267815+0000","last_undegraded":"2024-09-16T10:30:12.267815+0000","last_fullsized":"2024-09-16T10:30:12.267815+0000","mapping_epoch":33,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22474752,"num_objects":5487,"num_object_clones":0,"num_object_copies":10974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10706,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14874","reported_seq":23380,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:10.944137+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:10.944137+0000","last_peered":"2024-09-16T10:30:10.944137+0000","last_clean":"2024-09-16T10:30:10.944137+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:10.944137+0000","last_undegraded":"2024-09-16T10:30:10.944137+0000","last_fullsized":"2024-09-16T10:30:10.944137+0000","mapping_epoch":43,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22885318,"num_objects":5589,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14872,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15433","reported_seq":15709,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.198940+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:12.198940+0000","last_peered":"2024-09-16T10:30:12.198940+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:12.198940+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22967238,"num_objects":5609,"num_object_clones":0,"num_object_copies":11218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15431,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1272}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14929","reported_seq":15126,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:11.836207+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:11.836207+0000","last_peered":"2024-09-16T10:30:11.836207+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:11.836207+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22488006,"num_objects":5492,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14927,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14926","reported_seq":18899,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:11.854418+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:11.854418+0000","last_peered":"2024-09-16T10:30:11.854418+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:11.854418+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23143366,"num_objects":5652,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14924,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1943,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3399}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14625","reported_seq":14876,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.237430+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:12.237430+0000","last_peered":"2024-09-16T10:30:12.237430+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:12.237430+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22155747,"num_objects":5410,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14624,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15271","reported_seq":15547,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.263315+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:12.263315+0000","last_peered":"2024-09-16T10:30:12.263315+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:12.263315+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23115177,"num_objects":5646,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15268,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14724","reported_seq":14956,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.220683+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:12.220683+0000","last_peered":"2024-09-16T10:30:12.220683+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:12.220683+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22184902,"num_objects":5418,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14722,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15362","reported_seq":15604,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.218965+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:12.218965+0000","last_peered":"2024-09-16T10:30:12.218965+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:12.218965+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22669196,"num_objects":5538,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15359,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":362336782,"num_objects":88478,"num_object_clones":0,"num_object_copies":176956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88478,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217827,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11024,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":936255488,"data_stored":936171471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018973,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625840,"kb_used_data":315288,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93746000,"statfs":{"total":96636764160,"available":95995904000,"internally_reserved":0,"allocated":322854912,"data_stored":322586761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":9.7759999999999998},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.1520000000000001},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":5.7839999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691917,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588416,"kb_used_data":268008,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93783424,"statfs":{"total":96636764160,"available":96034226176,"internally_reserved":0,"allocated":274440192,"data_stored":274107007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":328068778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.4509999999999996},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.4329999999999998},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":11.074}]}]},{"osd":0,"up_from":24,"seq":103079215200,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734028,"kb_used_data":333316,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93637812,"statfs":{"total":96636764160,"available":95885119488,"internally_reserved":0,"allocated":341315584,"data_stored":341055119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":410316456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,0,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.495},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":6.9420000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.22700000000000001},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.56899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":340615168,"data_stored":340611555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273989632,"data_stored":273916913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321650688,"data_stored":321643003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:14.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:15.057 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:15.059 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:15.086 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-16T10:30:14.707586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":362336801,"num_objects":88480,"num_object_clones":0,"num_object_copies":176960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217829,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11025,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160665,"ondisk_log_size":160665,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1948284,"kb_used_data":916612,"kb_used_omap":12,"kb_used_meta":1031603,"kb_avail":281167236,"statfs":{"total":289910292480,"available":287915249664,"internally_reserved":0,"allocated":938610688,"data_stored":937748887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1056361474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,7,6,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18554880,"num_objects":-4530,"num_object_clones":0,"num_object_copies":-9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4530,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"2.f","version":"46'14918","reported_seq":23383,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:12.278956+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:12.278956+0000","last_peered":"2024-09-16T10:30:12.278956+0000","last_clean":"2024-09-16T10:30:12.278956+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:12.278956+0000","last_undegraded":"2024-09-16T10:30:12.278956+0000","last_fullsized":"2024-09-16T10:30:12.278956+0000","mapping_epoch":45,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22732800,"num_objects":5550,"num_object_clones":0,"num_object_copies":11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14918,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15001","reported_seq":15161,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:11.762311+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:11.762311+0000","last_peered":"2024-09-16T10:30:11.762311+0000","last_clean":"2024-09-16T10:30:11.762311+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:11.762311+0000","last_undegraded":"2024-09-16T10:30:11.762311+0000","last_fullsized":"2024-09-16T10:30:11.762311+0000","mapping_epoch":34,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22368256,"num_objects":5461,"num_object_clones":0,"num_object_copies":10922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15122","reported_seq":15376,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:11.782911+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:11.782911+0000","last_peered":"2024-09-16T10:30:11.782911+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:11.782911+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22695960,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11179,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14996","reported_seq":15055,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:11.829831+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:11.829831+0000","last_peered":"2024-09-16T10:30:11.829831+0000","last_clean":"2024-09-16T10:30:11.829831+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:11.829831+0000","last_undegraded":"2024-09-16T10:30:11.829831+0000","last_fullsized":"2024-09-16T10:30:11.829831+0000","mapping_epoch":34,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22388736,"num_objects":5466,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14996,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14963","reported_seq":15123,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:12.276672+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:12.276672+0000","last_peered":"2024-09-16T10:30:12.276672+0000","last_clean":"2024-09-16T10:30:12.276672+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:12.276672+0000","last_undegraded":"2024-09-16T10:30:12.276672+0000","last_fullsized":"2024-09-16T10:30:12.276672+0000","mapping_epoch":33,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22700032,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14881","reported_seq":15214,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.167726+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:12.167726+0000","last_peered":"2024-09-16T10:30:12.167726+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:12.167726+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22487040,"num_objects":5490,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15334","reported_seq":15743,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.201115+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:12.201115+0000","last_peered":"2024-09-16T10:30:12.201115+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:12.201115+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22880256,"num_objects":5586,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14903","reported_seq":15079,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:12.267815+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:12.267815+0000","last_peered":"2024-09-16T10:30:12.267815+0000","last_clean":"2024-09-16T10:30:12.267815+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:12.267815+0000","last_undegraded":"2024-09-16T10:30:12.267815+0000","last_fullsized":"2024-09-16T10:30:12.267815+0000","mapping_epoch":33,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22474752,"num_objects":5487,"num_object_clones":0,"num_object_copies":10974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10706,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14874","reported_seq":23380,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:10.944137+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:10.944137+0000","last_peered":"2024-09-16T10:30:10.944137+0000","last_clean":"2024-09-16T10:30:10.944137+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:10.944137+0000","last_undegraded":"2024-09-16T10:30:10.944137+0000","last_fullsized":"2024-09-16T10:30:10.944137+0000","mapping_epoch":43,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22885318,"num_objects":5589,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14872,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15433","reported_seq":15709,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.198940+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:12.198940+0000","last_peered":"2024-09-16T10:30:12.198940+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:12.198940+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22967238,"num_objects":5609,"num_object_clones":0,"num_object_copies":11218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15431,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1272}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14929","reported_seq":15126,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:11.836207+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:11.836207+0000","last_peered":"2024-09-16T10:30:11.836207+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:11.836207+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22488006,"num_objects":5492,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14927,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14926","reported_seq":18899,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:11.854418+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:11.854418+0000","last_peered":"2024-09-16T10:30:11.854418+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:11.854418+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23143366,"num_objects":5652,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14924,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1943,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3399}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14625","reported_seq":14876,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.237430+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:12.237430+0000","last_peered":"2024-09-16T10:30:12.237430+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:12.237430+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22155747,"num_objects":5410,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14624,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15271","reported_seq":15547,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.263315+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:12.263315+0000","last_peered":"2024-09-16T10:30:12.263315+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:12.263315+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23115177,"num_objects":5646,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15268,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14724","reported_seq":14956,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.220683+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:12.220683+0000","last_peered":"2024-09-16T10:30:12.220683+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:12.220683+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22184902,"num_objects":5418,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14722,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15362","reported_seq":15604,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.218965+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:12.218965+0000","last_peered":"2024-09-16T10:30:12.218965+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:12.218965+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22669196,"num_objects":5538,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15359,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":362336782,"num_objects":88478,"num_object_clones":0,"num_object_copies":176956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88478,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217827,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11024,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":936255488,"data_stored":936171471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018973,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625840,"kb_used_data":315288,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93746000,"statfs":{"total":96636764160,"available":95995904000,"internally_reserved":0,"allocated":322854912,"data_stored":322586761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":9.7759999999999998},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.1520000000000001},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":5.7839999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691917,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588416,"kb_used_data":268008,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93783424,"statfs":{"total":96636764160,"available":96034226176,"internally_reserved":0,"allocated":274440192,"data_stored":274107007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":328068778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.4509999999999996},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.4329999999999998},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":11.074}]}]},{"osd":0,"up_from":24,"seq":103079215200,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734028,"kb_used_data":333316,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93637812,"statfs":{"total":96636764160,"available":95885119488,"internally_reserved":0,"allocated":341315584,"data_stored":341055119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":410316456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,0,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.495},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":6.9420000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.22700000000000001},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.56899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":340615168,"data_stored":340611555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273989632,"data_stored":273916913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321650688,"data_stored":321643003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:15.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:15.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'15122", 'reported_seq': 15376, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:11.782911+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:11.782911+0000', 'last_peered': '2024-09-16T10:30:11.782911+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:11.782911+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'5100", 'ondisk_log_start': "42'5100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22695960, 'num_objects': 5542, 'num_object_clones': 0, 'num_object_copies': 11084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5542, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11179, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1239}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:15.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:15.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'14881", 'reported_seq': 15214, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:12.167726+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:12.167726+0000', 'last_peered': '2024-09-16T10:30:12.167726+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:12.167726+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'4800", 'ondisk_log_start': "42'4800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22487040, 'num_objects': 5490, 'num_object_clones': 0, 'num_object_copies': 10980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11506, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1356}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:15.087 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'15334", 'reported_seq': 15743, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:12.201115+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:12.201115+0000', 'last_peered': '2024-09-16T10:30:12.201115+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:12.201115+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'5300", 'ondisk_log_start': "42'5300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22880256, 'num_objects': 5586, 'num_object_clones': 0, 'num_object_copies': 11172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11576, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'15433", 'reported_seq': 15709, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:12.198940+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:12.198940+0000', 'last_peered': '2024-09-16T10:30:12.198940+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:12.198940+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'5400", 'ondisk_log_start': "42'5400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22967238, 'num_objects': 5609, 'num_object_clones': 0, 'num_object_copies': 11218, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15431, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1272}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'14929", 'reported_seq': 15126, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:11.836207+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:11.836207+0000', 'last_peered': '2024-09-16T10:30:11.836207+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:11.836207+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'4900", 'ondisk_log_start': "42'4900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22488006, 'num_objects': 5492, 'num_object_clones': 0, 'num_object_copies': 10984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14927, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1259}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'14926", 'reported_seq': 18899, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:11.854418+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:11.854418+0000', 'last_peered': '2024-09-16T10:30:11.854418+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:11.854418+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'4900", 'ondisk_log_start': "42'4900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23143366, 'num_objects': 5652, 'num_object_clones': 0, 'num_object_copies': 11304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2253, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14924, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1943, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3399}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'14625", 'reported_seq': 14876, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:12.237430+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:12.237430+0000', 'last_peered': '2024-09-16T10:30:12.237430+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:12.237430+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'4600", 'ondisk_log_start': "42'4600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22155747, 'num_objects': 5410, 'num_object_clones': 0, 'num_object_copies': 10820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14624, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1335}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:15.088 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'15271", 'reported_seq': 15547, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:12.263315+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:12.263315+0000', 'last_peered': '2024-09-16T10:30:12.263315+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:12.263315+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'5200", 'ondisk_log_start': "42'5200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23115177, 'num_objects': 5646, 'num_object_clones': 0, 'num_object_copies': 11292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15268, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:15.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:15.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'14724", 'reported_seq': 14956, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:12.220683+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:12.220683+0000', 'last_peered': '2024-09-16T10:30:12.220683+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:12.220683+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22184902, 'num_objects': 5418, 'num_object_clones': 0, 'num_object_copies': 10836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14722, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1325}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:15.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:15.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'15362", 'reported_seq': 15604, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:12.218965+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:12.218965+0000', 'last_peered': '2024-09-16T10:30:12.218965+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:12.218965+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'5300", 'ondisk_log_start': "42'5300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22669196, 'num_objects': 5538, 'num_object_clones': 0, 'num_object_copies': 11076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15359, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:15.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:15.781 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:15.781 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 40322/176960 objects degraded (22.786%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":504,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":88480,"data_bytes":362336801,"bytes_used":1995042816,"bytes_avail":287915249664,"bytes_total":289910292480,"degraded_objects":40322,"degraded_total":176960,"degraded_ratio":0.22785940325497286,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":377},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T10:29:36.693808+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:30:15.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:15.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:16.375 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:16.376 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:16.404 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-16T10:30:14.707586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":362336801,"num_objects":88480,"num_object_clones":0,"num_object_copies":176960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217829,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11025,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160665,"ondisk_log_size":160665,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1948284,"kb_used_data":916612,"kb_used_omap":12,"kb_used_meta":1031603,"kb_avail":281167236,"statfs":{"total":289910292480,"available":287915249664,"internally_reserved":0,"allocated":938610688,"data_stored":937748887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1056361474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,7,6,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18554880,"num_objects":-4530,"num_object_clones":0,"num_object_copies":-9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4530,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"2.f","version":"46'14918","reported_seq":23383,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:12.278956+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:12.278956+0000","last_peered":"2024-09-16T10:30:12.278956+0000","last_clean":"2024-09-16T10:30:12.278956+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:12.278956+0000","last_undegraded":"2024-09-16T10:30:12.278956+0000","last_fullsized":"2024-09-16T10:30:12.278956+0000","mapping_epoch":45,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22732800,"num_objects":5550,"num_object_clones":0,"num_object_copies":11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14918,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15001","reported_seq":15161,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:11.762311+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:11.762311+0000","last_peered":"2024-09-16T10:30:11.762311+0000","last_clean":"2024-09-16T10:30:11.762311+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:11.762311+0000","last_undegraded":"2024-09-16T10:30:11.762311+0000","last_fullsized":"2024-09-16T10:30:11.762311+0000","mapping_epoch":34,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22368256,"num_objects":5461,"num_object_clones":0,"num_object_copies":10922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11214,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15122","reported_seq":15376,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:11.782911+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:11.782911+0000","last_peered":"2024-09-16T10:30:11.782911+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:11.782911+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22695960,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11179,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'14996","reported_seq":15055,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:11.829831+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:11.829831+0000","last_peered":"2024-09-16T10:30:11.829831+0000","last_clean":"2024-09-16T10:30:11.829831+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:11.829831+0000","last_undegraded":"2024-09-16T10:30:11.829831+0000","last_fullsized":"2024-09-16T10:30:11.829831+0000","mapping_epoch":34,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22388736,"num_objects":5466,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14996,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'14963","reported_seq":15123,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:12.276672+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:12.276672+0000","last_peered":"2024-09-16T10:30:12.276672+0000","last_clean":"2024-09-16T10:30:12.276672+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:12.276672+0000","last_undegraded":"2024-09-16T10:30:12.276672+0000","last_fullsized":"2024-09-16T10:30:12.276672+0000","mapping_epoch":33,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22700032,"num_objects":5542,"num_object_clones":0,"num_object_copies":11084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'14881","reported_seq":15214,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.167726+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:12.167726+0000","last_peered":"2024-09-16T10:30:12.167726+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:12.167726+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22487040,"num_objects":5490,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15334","reported_seq":15743,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.201115+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:12.201115+0000","last_peered":"2024-09-16T10:30:12.201115+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:12.201115+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22880256,"num_objects":5586,"num_object_clones":0,"num_object_copies":11172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1299}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'14903","reported_seq":15079,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:12.267815+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:12.267815+0000","last_peered":"2024-09-16T10:30:12.267815+0000","last_clean":"2024-09-16T10:30:12.267815+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:12.267815+0000","last_undegraded":"2024-09-16T10:30:12.267815+0000","last_fullsized":"2024-09-16T10:30:12.267815+0000","mapping_epoch":33,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22474752,"num_objects":5487,"num_object_clones":0,"num_object_copies":10974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10706,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'14874","reported_seq":23380,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:10.944137+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:10.944137+0000","last_peered":"2024-09-16T10:30:10.944137+0000","last_clean":"2024-09-16T10:30:10.944137+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:10.944137+0000","last_undegraded":"2024-09-16T10:30:10.944137+0000","last_fullsized":"2024-09-16T10:30:10.944137+0000","mapping_epoch":43,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22885318,"num_objects":5589,"num_object_clones":0,"num_object_copies":11178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14872,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15433","reported_seq":15709,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.198940+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:12.198940+0000","last_peered":"2024-09-16T10:30:12.198940+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:12.198940+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22967238,"num_objects":5609,"num_object_clones":0,"num_object_copies":11218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15431,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1272}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'14929","reported_seq":15126,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:11.836207+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:11.836207+0000","last_peered":"2024-09-16T10:30:11.836207+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:11.836207+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22488006,"num_objects":5492,"num_object_clones":0,"num_object_copies":10984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14927,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'14926","reported_seq":18899,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:11.854418+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:11.854418+0000","last_peered":"2024-09-16T10:30:11.854418+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:11.854418+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23143366,"num_objects":5652,"num_object_clones":0,"num_object_copies":11304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14924,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1943,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3399}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14625","reported_seq":14876,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.237430+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:12.237430+0000","last_peered":"2024-09-16T10:30:12.237430+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:12.237430+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4600","ondisk_log_start":"42'4600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22155747,"num_objects":5410,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14624,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1335}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15271","reported_seq":15547,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.263315+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:12.263315+0000","last_peered":"2024-09-16T10:30:12.263315+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:12.263315+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23115177,"num_objects":5646,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15268,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14724","reported_seq":14956,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.220683+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:12.220683+0000","last_peered":"2024-09-16T10:30:12.220683+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:12.220683+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22184902,"num_objects":5418,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14722,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1325}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15362","reported_seq":15604,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:12.218965+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:12.218965+0000","last_peered":"2024-09-16T10:30:12.218965+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:12.218965+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22669196,"num_objects":5538,"num_object_clones":0,"num_object_copies":11076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15359,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":362336782,"num_objects":88478,"num_object_clones":0,"num_object_copies":176956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88478,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217827,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11024,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":936255488,"data_stored":936171471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160662,"ondisk_log_size":160662,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018973,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625840,"kb_used_data":315288,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93746000,"statfs":{"total":96636764160,"available":95995904000,"internally_reserved":0,"allocated":322854912,"data_stored":322586761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":9.7759999999999998},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.1520000000000001},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":5.7839999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691917,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":588416,"kb_used_data":268008,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93783424,"statfs":{"total":96636764160,"available":96034226176,"internally_reserved":0,"allocated":274440192,"data_stored":274107007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":328068778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":2.2330000000000001,"5min":3.2749999999999999,"15min":3.4910000000000001},"min":{"1min":0.224,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.0190000000000001,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.4509999999999996},{"interface":"front","average":{"1min":1.4079999999999999,"5min":3.0030000000000001,"15min":2.605},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":8.0519999999999996,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:29:08 2024","interfaces":[{"interface":"back","average":{"1min":3.5139999999999998,"5min":2.839,"15min":2.492},"min":{"1min":0.28999999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":18.638999999999999,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.4329999999999998},{"interface":"front","average":{"1min":2.3029999999999999,"5min":2.8439999999999999,"15min":2.8759999999999999},"min":{"1min":0.26500000000000001,"5min":0.216,"15min":0.216},"max":{"1min":10.869999999999999,"5min":11.412000000000001,"15min":11.412000000000001},"last":11.074}]}]},{"osd":0,"up_from":24,"seq":103079215200,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734028,"kb_used_data":333316,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93637812,"statfs":{"total":96636764160,"available":95885119488,"internally_reserved":0,"allocated":341315584,"data_stored":341055119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":410316456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,3,0,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.495},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":6.9420000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.22700000000000001},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.56899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":340615168,"data_stored":340611555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":273989632,"data_stored":273916913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":321650688,"data_stored":321643003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:19.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:19.995 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:19.997 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:20.023 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-16T10:30:18.708500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":353071649,"num_objects":86218,"num_object_clones":0,"num_object_copies":172436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220091,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11157,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160927,"ondisk_log_size":160927,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1909860,"kb_used_data":898692,"kb_used_omap":12,"kb_used_meta":1011123,"kb_avail":281205660,"statfs":{"total":289910292480,"available":287954595840,"internally_reserved":0,"allocated":920260608,"data_stored":919398807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1035389954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,4,8,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27238400,"num_objects":-6650,"num_object_clones":0,"num_object_copies":-13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6650,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002877"},"pg_stats":[{"pgid":"2.f","version":"46'15057","reported_seq":23522,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.341081+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:17.341081+0000","last_peered":"2024-09-16T10:30:17.341081+0000","last_clean":"2024-09-16T10:30:17.341081+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:17.341081+0000","last_undegraded":"2024-09-16T10:30:17.341081+0000","last_fullsized":"2024-09-16T10:30:17.341081+0000","mapping_epoch":45,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163456,"num_objects":5411,"num_object_clones":0,"num_object_copies":10822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15057,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15143","reported_seq":15303,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.786711+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:16.786711+0000","last_peered":"2024-09-16T10:30:16.786711+0000","last_clean":"2024-09-16T10:30:16.786711+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:16.786711+0000","last_undegraded":"2024-09-16T10:30:16.786711+0000","last_fullsized":"2024-09-16T10:30:16.786711+0000","mapping_epoch":34,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21786624,"num_objects":5319,"num_object_clones":0,"num_object_copies":10638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11356,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15261","reported_seq":15515,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:16.847130+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:16.847130+0000","last_peered":"2024-09-16T10:30:16.847130+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:16.847130+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22126616,"num_objects":5403,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11318,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15136","reported_seq":15196,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.803823+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:16.803823+0000","last_peered":"2024-09-16T10:30:16.803823+0000","last_clean":"2024-09-16T10:30:16.803823+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:16.803823+0000","last_undegraded":"2024-09-16T10:30:16.803823+0000","last_fullsized":"2024-09-16T10:30:16.803823+0000","mapping_epoch":34,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21815296,"num_objects":5326,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15136,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15090","reported_seq":15250,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.359340+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:17.359340+0000","last_peered":"2024-09-16T10:30:17.359340+0000","last_clean":"2024-09-16T10:30:17.359340+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:17.359340+0000","last_undegraded":"2024-09-16T10:30:17.359340+0000","last_fullsized":"2024-09-16T10:30:17.359340+0000","mapping_epoch":33,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22179840,"num_objects":5415,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11732,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15039","reported_seq":15370,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.372011+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:17.372011+0000","last_peered":"2024-09-16T10:30:17.372011+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:17.372011+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21839872,"num_objects":5332,"num_object_clones":0,"num_object_copies":10664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11664,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15483","reported_seq":15892,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.282845+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:17.282845+0000","last_peered":"2024-09-16T10:30:17.282845+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:17.282845+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22269952,"num_objects":5437,"num_object_clones":0,"num_object_copies":10874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15025","reported_seq":15203,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.181729+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:17.181729+0000","last_peered":"2024-09-16T10:30:17.181729+0000","last_clean":"2024-09-16T10:30:17.181729+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:17.181729+0000","last_undegraded":"2024-09-16T10:30:17.181729+0000","last_fullsized":"2024-09-16T10:30:17.181729+0000","mapping_epoch":33,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21975040,"num_objects":5365,"num_object_clones":0,"num_object_copies":10730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15018","reported_seq":23525,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074566+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:16.074566+0000","last_peered":"2024-09-16T10:30:16.074566+0000","last_clean":"2024-09-16T10:30:16.074566+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:16.074566+0000","last_undegraded":"2024-09-16T10:30:16.074566+0000","last_fullsized":"2024-09-16T10:30:16.074566+0000","mapping_epoch":43,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22295494,"num_objects":5445,"num_object_clones":0,"num_object_copies":10890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15016,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15589","reported_seq":15865,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.369890+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:17.369890+0000","last_peered":"2024-09-16T10:30:17.369890+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:17.369890+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328262,"num_objects":5453,"num_object_clones":0,"num_object_copies":10906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15587,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1116}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15060","reported_seq":15254,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:16.816183+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:16.816183+0000","last_peered":"2024-09-16T10:30:16.816183+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:16.816183+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21951430,"num_objects":5361,"num_object_clones":0,"num_object_copies":10722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15058,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1128}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15066","reported_seq":19303,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:16.852439+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:16.852439+0000","last_peered":"2024-09-16T10:30:16.852439+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:16.852439+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22569926,"num_objects":5512,"num_object_clones":0,"num_object_copies":11024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15064,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2075,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14766","reported_seq":15017,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.338940+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:17.338940+0000","last_peered":"2024-09-16T10:30:17.338940+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:17.338940+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21578211,"num_objects":5269,"num_object_clones":0,"num_object_copies":10538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14765,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15437","reported_seq":15711,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.313136+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:17.313136+0000","last_peered":"2024-09-16T10:30:17.313136+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:17.313136+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22435241,"num_objects":5480,"num_object_clones":0,"num_object_copies":10960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15434,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14864","reported_seq":15096,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.304970+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:17.304970+0000","last_peered":"2024-09-16T10:30:17.304970+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:17.304970+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21611462,"num_objects":5278,"num_object_clones":0,"num_object_copies":10556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14862,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1185}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15490","reported_seq":15732,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.293119+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:17.293119+0000","last_peered":"2024-09-16T10:30:17.293119+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:17.293119+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22144908,"num_objects":5410,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15487,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":353071630,"num_objects":86216,"num_object_clones":0,"num_object_copies":172432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220089,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11156,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":917721088,"data_stored":917637071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160924,"ondisk_log_size":160924,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018974,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609180,"kb_used_data":310932,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93762660,"statfs":{"total":96636764160,"available":96012963840,"internally_reserved":0,"allocated":318394368,"data_stored":318126217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":2.7789999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.0410000000000004},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.23400000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585552,"kb_used_data":261060,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93786288,"statfs":{"total":96636764160,"available":96037158912,"internally_reserved":0,"allocated":267325440,"data_stored":266992255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":332263082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,6,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.9220000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.58599999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215201,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715128,"kb_used_data":326700,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93656712,"statfs":{"total":96636764160,"available":95904473088,"internally_reserved":0,"allocated":334540800,"data_stored":334280335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397733544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.9000000000000004},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.9960000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.68999999999999995},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.8540000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333602816,"data_stored":333599203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266858496,"data_stored":266785777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317259776,"data_stored":317252091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:20.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:20.562 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:20.563 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:20.588 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-16T10:30:18.708500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":353071649,"num_objects":86218,"num_object_clones":0,"num_object_copies":172436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220091,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11157,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160927,"ondisk_log_size":160927,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1909860,"kb_used_data":898692,"kb_used_omap":12,"kb_used_meta":1011123,"kb_avail":281205660,"statfs":{"total":289910292480,"available":287954595840,"internally_reserved":0,"allocated":920260608,"data_stored":919398807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1035389954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,4,8,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27238400,"num_objects":-6650,"num_object_clones":0,"num_object_copies":-13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6650,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":380,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002877"},"pg_stats":[{"pgid":"2.f","version":"46'15057","reported_seq":23522,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.341081+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:17.341081+0000","last_peered":"2024-09-16T10:30:17.341081+0000","last_clean":"2024-09-16T10:30:17.341081+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:17.341081+0000","last_undegraded":"2024-09-16T10:30:17.341081+0000","last_fullsized":"2024-09-16T10:30:17.341081+0000","mapping_epoch":45,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163456,"num_objects":5411,"num_object_clones":0,"num_object_copies":10822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15057,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15143","reported_seq":15303,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.786711+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:16.786711+0000","last_peered":"2024-09-16T10:30:16.786711+0000","last_clean":"2024-09-16T10:30:16.786711+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:16.786711+0000","last_undegraded":"2024-09-16T10:30:16.786711+0000","last_fullsized":"2024-09-16T10:30:16.786711+0000","mapping_epoch":34,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21786624,"num_objects":5319,"num_object_clones":0,"num_object_copies":10638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11356,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15261","reported_seq":15515,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:16.847130+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:16.847130+0000","last_peered":"2024-09-16T10:30:16.847130+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:16.847130+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22126616,"num_objects":5403,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11318,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15136","reported_seq":15196,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.803823+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:16.803823+0000","last_peered":"2024-09-16T10:30:16.803823+0000","last_clean":"2024-09-16T10:30:16.803823+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:16.803823+0000","last_undegraded":"2024-09-16T10:30:16.803823+0000","last_fullsized":"2024-09-16T10:30:16.803823+0000","mapping_epoch":34,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21815296,"num_objects":5326,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15136,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15090","reported_seq":15250,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.359340+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:17.359340+0000","last_peered":"2024-09-16T10:30:17.359340+0000","last_clean":"2024-09-16T10:30:17.359340+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:17.359340+0000","last_undegraded":"2024-09-16T10:30:17.359340+0000","last_fullsized":"2024-09-16T10:30:17.359340+0000","mapping_epoch":33,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22179840,"num_objects":5415,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11732,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15039","reported_seq":15370,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.372011+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:17.372011+0000","last_peered":"2024-09-16T10:30:17.372011+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:17.372011+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21839872,"num_objects":5332,"num_object_clones":0,"num_object_copies":10664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11664,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15483","reported_seq":15892,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.282845+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:17.282845+0000","last_peered":"2024-09-16T10:30:17.282845+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:17.282845+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22269952,"num_objects":5437,"num_object_clones":0,"num_object_copies":10874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15025","reported_seq":15203,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.181729+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:17.181729+0000","last_peered":"2024-09-16T10:30:17.181729+0000","last_clean":"2024-09-16T10:30:17.181729+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:17.181729+0000","last_undegraded":"2024-09-16T10:30:17.181729+0000","last_fullsized":"2024-09-16T10:30:17.181729+0000","mapping_epoch":33,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21975040,"num_objects":5365,"num_object_clones":0,"num_object_copies":10730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15018","reported_seq":23525,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074566+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:16.074566+0000","last_peered":"2024-09-16T10:30:16.074566+0000","last_clean":"2024-09-16T10:30:16.074566+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:16.074566+0000","last_undegraded":"2024-09-16T10:30:16.074566+0000","last_fullsized":"2024-09-16T10:30:16.074566+0000","mapping_epoch":43,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22295494,"num_objects":5445,"num_object_clones":0,"num_object_copies":10890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15016,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15589","reported_seq":15865,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.369890+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:17.369890+0000","last_peered":"2024-09-16T10:30:17.369890+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:17.369890+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328262,"num_objects":5453,"num_object_clones":0,"num_object_copies":10906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15587,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1116}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15060","reported_seq":15254,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:16.816183+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:16.816183+0000","last_peered":"2024-09-16T10:30:16.816183+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:16.816183+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21951430,"num_objects":5361,"num_object_clones":0,"num_object_copies":10722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15058,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1128}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15066","reported_seq":19303,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:16.852439+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:16.852439+0000","last_peered":"2024-09-16T10:30:16.852439+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:16.852439+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22569926,"num_objects":5512,"num_object_clones":0,"num_object_copies":11024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15064,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2075,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14766","reported_seq":15017,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.338940+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:17.338940+0000","last_peered":"2024-09-16T10:30:17.338940+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:17.338940+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21578211,"num_objects":5269,"num_object_clones":0,"num_object_copies":10538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14765,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15437","reported_seq":15711,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.313136+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:17.313136+0000","last_peered":"2024-09-16T10:30:17.313136+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:17.313136+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22435241,"num_objects":5480,"num_object_clones":0,"num_object_copies":10960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15434,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14864","reported_seq":15096,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.304970+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:17.304970+0000","last_peered":"2024-09-16T10:30:17.304970+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:17.304970+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21611462,"num_objects":5278,"num_object_clones":0,"num_object_copies":10556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14862,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1185}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15490","reported_seq":15732,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.293119+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:17.293119+0000","last_peered":"2024-09-16T10:30:17.293119+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:17.293119+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22144908,"num_objects":5410,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15487,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":353071630,"num_objects":86216,"num_object_clones":0,"num_object_copies":172432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220089,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11156,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":917721088,"data_stored":917637071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160924,"ondisk_log_size":160924,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018974,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609180,"kb_used_data":310932,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93762660,"statfs":{"total":96636764160,"available":96012963840,"internally_reserved":0,"allocated":318394368,"data_stored":318126217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":2.7789999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.0410000000000004},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.23400000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585552,"kb_used_data":261060,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93786288,"statfs":{"total":96636764160,"available":96037158912,"internally_reserved":0,"allocated":267325440,"data_stored":266992255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":332263082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,6,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.9220000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.58599999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215201,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715128,"kb_used_data":326700,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93656712,"statfs":{"total":96636764160,"available":95904473088,"internally_reserved":0,"allocated":334540800,"data_stored":334280335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397733544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.9000000000000004},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.9960000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.68999999999999995},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.8540000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333602816,"data_stored":333599203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266858496,"data_stored":266785777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317259776,"data_stored":317252091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:20.589 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:20.589 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'15261", 'reported_seq': 15515, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:16.847130+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:16.847130+0000', 'last_peered': '2024-09-16T10:30:16.847130+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:16.847130+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'5200", 'ondisk_log_start': "42'5200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22126616, 'num_objects': 5403, 'num_object_clones': 0, 'num_object_copies': 10806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5403, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11318, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1100}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'15039", 'reported_seq': 15370, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:17.372011+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:17.372011+0000', 'last_peered': '2024-09-16T10:30:17.372011+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:17.372011+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'5000", 'ondisk_log_start': "42'5000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21839872, 'num_objects': 5332, 'num_object_clones': 0, 'num_object_copies': 10664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11664, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'15483", 'reported_seq': 15892, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:17.282845+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:17.282845+0000', 'last_peered': '2024-09-16T10:30:17.282845+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:17.282845+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'5400", 'ondisk_log_start': "42'5400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22269952, 'num_objects': 5437, 'num_object_clones': 0, 'num_object_copies': 10874, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11725, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1150}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'15589", 'reported_seq': 15865, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:17.369890+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:17.369890+0000', 'last_peered': '2024-09-16T10:30:17.369890+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:17.369890+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'5500", 'ondisk_log_start': "42'5500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22328262, 'num_objects': 5453, 'num_object_clones': 0, 'num_object_copies': 10906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15587, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1116}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'15060", 'reported_seq': 15254, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:16.816183+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:16.816183+0000', 'last_peered': '2024-09-16T10:30:16.816183+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:16.816183+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'5000", 'ondisk_log_start': "42'5000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21951430, 'num_objects': 5361, 'num_object_clones': 0, 'num_object_copies': 10722, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15058, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 1128}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'15066", 'reported_seq': 19303, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:16.852439+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:16.852439+0000', 'last_peered': '2024-09-16T10:30:16.852439+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:16.852439+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'5000", 'ondisk_log_start': "42'5000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22569926, 'num_objects': 5512, 'num_object_clones': 0, 'num_object_copies': 11024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2121, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15064, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2075, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3391}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:20.590 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:20.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'14766", 'reported_seq': 15017, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:17.338940+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:17.338940+0000', 'last_peered': '2024-09-16T10:30:17.338940+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:17.338940+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'4700", 'ondisk_log_start': "42'4700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21578211, 'num_objects': 5269, 'num_object_clones': 0, 'num_object_copies': 10538, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14765, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:20.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:20.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'15437", 'reported_seq': 15711, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:17.313136+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:17.313136+0000', 'last_peered': '2024-09-16T10:30:17.313136+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:17.313136+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'5400", 'ondisk_log_start': "42'5400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22435241, 'num_objects': 5480, 'num_object_clones': 0, 'num_object_copies': 10960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5480, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15434, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1213}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:20.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:20.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'14864", 'reported_seq': 15096, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:17.304970+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:17.304970+0000', 'last_peered': '2024-09-16T10:30:17.304970+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:17.304970+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'4800", 'ondisk_log_start': "42'4800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21611462, 'num_objects': 5278, 'num_object_clones': 0, 'num_object_copies': 10556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14862, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1185}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:20.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:20.591 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'15490", 'reported_seq': 15732, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:17.293119+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:17.293119+0000', 'last_peered': '2024-09-16T10:30:17.293119+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:17.293119+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'5400", 'ondisk_log_start': "42'5400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22144908, 'num_objects': 5410, 'num_object_clones': 0, 'num_object_copies': 10820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15487, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:20.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:21.301 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:21.301 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 40190/172436 objects degraded (23.307%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":510,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":86218,"data_bytes":353071649,"bytes_used":1955696640,"bytes_avail":287954595840,"bytes_total":289910292480,"degraded_objects":40190,"degraded_total":172436,"degraded_ratio":0.23307198032893364,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":376},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T10:29:36.693808+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:30:21.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:21.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:21.861 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:21.861 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:21.894 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-16T10:30:20.710178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":353071649,"num_objects":86218,"num_object_clones":0,"num_object_copies":172436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220091,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11157,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160927,"ondisk_log_size":160927,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1909860,"kb_used_data":898692,"kb_used_omap":12,"kb_used_meta":1011123,"kb_avail":281205660,"statfs":{"total":289910292480,"available":287954595840,"internally_reserved":0,"allocated":920260608,"data_stored":919398807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1035389954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,4,8,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18505728,"num_objects":-4518,"num_object_clones":0,"num_object_copies":-9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4518,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004062"},"pg_stats":[{"pgid":"2.f","version":"46'15057","reported_seq":23522,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.341081+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:17.341081+0000","last_peered":"2024-09-16T10:30:17.341081+0000","last_clean":"2024-09-16T10:30:17.341081+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:17.341081+0000","last_undegraded":"2024-09-16T10:30:17.341081+0000","last_fullsized":"2024-09-16T10:30:17.341081+0000","mapping_epoch":45,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22163456,"num_objects":5411,"num_object_clones":0,"num_object_copies":10822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15057,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15143","reported_seq":15303,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.786711+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:16.786711+0000","last_peered":"2024-09-16T10:30:16.786711+0000","last_clean":"2024-09-16T10:30:16.786711+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:16.786711+0000","last_undegraded":"2024-09-16T10:30:16.786711+0000","last_fullsized":"2024-09-16T10:30:16.786711+0000","mapping_epoch":34,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21786624,"num_objects":5319,"num_object_clones":0,"num_object_copies":10638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11356,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15261","reported_seq":15515,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:16.847130+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:16.847130+0000","last_peered":"2024-09-16T10:30:16.847130+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:16.847130+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22126616,"num_objects":5403,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11318,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1100}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15136","reported_seq":15196,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.803823+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:16.803823+0000","last_peered":"2024-09-16T10:30:16.803823+0000","last_clean":"2024-09-16T10:30:16.803823+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:16.803823+0000","last_undegraded":"2024-09-16T10:30:16.803823+0000","last_fullsized":"2024-09-16T10:30:16.803823+0000","mapping_epoch":34,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21815296,"num_objects":5326,"num_object_clones":0,"num_object_copies":10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15136,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15090","reported_seq":15250,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.359340+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:17.359340+0000","last_peered":"2024-09-16T10:30:17.359340+0000","last_clean":"2024-09-16T10:30:17.359340+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:17.359340+0000","last_undegraded":"2024-09-16T10:30:17.359340+0000","last_fullsized":"2024-09-16T10:30:17.359340+0000","mapping_epoch":33,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22179840,"num_objects":5415,"num_object_clones":0,"num_object_copies":10830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11732,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15039","reported_seq":15370,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.372011+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:17.372011+0000","last_peered":"2024-09-16T10:30:17.372011+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:17.372011+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21839872,"num_objects":5332,"num_object_clones":0,"num_object_copies":10664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11664,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15483","reported_seq":15892,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.282845+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:17.282845+0000","last_peered":"2024-09-16T10:30:17.282845+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:17.282845+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22269952,"num_objects":5437,"num_object_clones":0,"num_object_copies":10874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15025","reported_seq":15203,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:17.181729+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:17.181729+0000","last_peered":"2024-09-16T10:30:17.181729+0000","last_clean":"2024-09-16T10:30:17.181729+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:17.181729+0000","last_undegraded":"2024-09-16T10:30:17.181729+0000","last_fullsized":"2024-09-16T10:30:17.181729+0000","mapping_epoch":33,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21975040,"num_objects":5365,"num_object_clones":0,"num_object_copies":10730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10828,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15018","reported_seq":23525,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:16.074566+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:16.074566+0000","last_peered":"2024-09-16T10:30:16.074566+0000","last_clean":"2024-09-16T10:30:16.074566+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:16.074566+0000","last_undegraded":"2024-09-16T10:30:16.074566+0000","last_fullsized":"2024-09-16T10:30:16.074566+0000","mapping_epoch":43,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22295494,"num_objects":5445,"num_object_clones":0,"num_object_copies":10890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15016,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15589","reported_seq":15865,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.369890+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:17.369890+0000","last_peered":"2024-09-16T10:30:17.369890+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:17.369890+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22328262,"num_objects":5453,"num_object_clones":0,"num_object_copies":10906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15587,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1116}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15060","reported_seq":15254,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:16.816183+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:16.816183+0000","last_peered":"2024-09-16T10:30:16.816183+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:16.816183+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21951430,"num_objects":5361,"num_object_clones":0,"num_object_copies":10722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15058,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":1128}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15066","reported_seq":19303,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:16.852439+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:16.852439+0000","last_peered":"2024-09-16T10:30:16.852439+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:16.852439+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22569926,"num_objects":5512,"num_object_clones":0,"num_object_copies":11024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15064,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2075,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14766","reported_seq":15017,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.338940+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:17.338940+0000","last_peered":"2024-09-16T10:30:17.338940+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:17.338940+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4700","ondisk_log_start":"42'4700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21578211,"num_objects":5269,"num_object_clones":0,"num_object_copies":10538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14765,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15437","reported_seq":15711,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.313136+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:17.313136+0000","last_peered":"2024-09-16T10:30:17.313136+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:17.313136+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22435241,"num_objects":5480,"num_object_clones":0,"num_object_copies":10960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15434,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'14864","reported_seq":15096,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.304970+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:17.304970+0000","last_peered":"2024-09-16T10:30:17.304970+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:17.304970+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'4800","ondisk_log_start":"42'4800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21611462,"num_objects":5278,"num_object_clones":0,"num_object_copies":10556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14862,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1185}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15490","reported_seq":15732,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:17.293119+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:17.293119+0000","last_peered":"2024-09-16T10:30:17.293119+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:17.293119+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22144908,"num_objects":5410,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15487,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":353071630,"num_objects":86216,"num_object_clones":0,"num_object_copies":172432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40190,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":220089,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11156,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":917721088,"data_stored":917637071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160924,"ondisk_log_size":160924,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018974,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609180,"kb_used_data":310932,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93762660,"statfs":{"total":96636764160,"available":96012963840,"internally_reserved":0,"allocated":318394368,"data_stored":318126217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":2.7789999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.0410000000000004},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.23400000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691918,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585552,"kb_used_data":261060,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93786288,"statfs":{"total":96636764160,"available":96037158912,"internally_reserved":0,"allocated":267325440,"data_stored":266992255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":332263082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,6,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.9220000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.58599999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215201,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715128,"kb_used_data":326700,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93656712,"statfs":{"total":96636764160,"available":95904473088,"internally_reserved":0,"allocated":334540800,"data_stored":334280335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397733544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.9000000000000004},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.9960000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.68999999999999995},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.8540000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333602816,"data_stored":333599203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266858496,"data_stored":266785777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":317259776,"data_stored":317252091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:24.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:25.453 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:25.455 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:25.476 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-16T10:30:24.711123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":343843361,"num_objects":83965,"num_object_clones":0,"num_object_copies":167930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83965,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222344,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11292,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160780,"ondisk_log_size":160780,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1895988,"kb_used_data":880708,"kb_used_omap":12,"kb_used_meta":1015219,"kb_avail":281219532,"statfs":{"total":289910292480,"available":287968800768,"internally_reserved":0,"allocated":901844992,"data_stored":900983191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1039584258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,2,13,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18493440,"num_objects":-4515,"num_object_clones":0,"num_object_copies":-9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4515,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004040"},"pg_stats":[{"pgid":"2.f","version":"46'15200","reported_seq":23665,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:22.395889+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:22.395889+0000","last_peered":"2024-09-16T10:30:22.395889+0000","last_clean":"2024-09-16T10:30:22.395889+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:22.395889+0000","last_undegraded":"2024-09-16T10:30:22.395889+0000","last_fullsized":"2024-09-16T10:30:22.395889+0000","mapping_epoch":45,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21577728,"num_objects":5268,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15200,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15285","reported_seq":15445,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:21.790309+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:21.790309+0000","last_peered":"2024-09-16T10:30:21.790309+0000","last_clean":"2024-09-16T10:30:21.790309+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:21.790309+0000","last_undegraded":"2024-09-16T10:30:21.790309+0000","last_fullsized":"2024-09-16T10:30:21.790309+0000","mapping_epoch":34,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21204992,"num_objects":5177,"num_object_clones":0,"num_object_copies":10354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11498,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15412","reported_seq":15666,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:21.852924+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:21.852924+0000","last_peered":"2024-09-16T10:30:21.852924+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:21.852924+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21508120,"num_objects":5252,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11469,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15292","reported_seq":15351,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:21.727117+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:21.727117+0000","last_peered":"2024-09-16T10:30:21.727117+0000","last_clean":"2024-09-16T10:30:21.727117+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:21.727117+0000","last_undegraded":"2024-09-16T10:30:21.727117+0000","last_fullsized":"2024-09-16T10:30:21.727117+0000","mapping_epoch":34,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21176320,"num_objects":5170,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15292,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15231","reported_seq":15391,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:22.269161+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:22.269161+0000","last_peered":"2024-09-16T10:30:22.269161+0000","last_clean":"2024-09-16T10:30:22.269161+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:22.269161+0000","last_undegraded":"2024-09-16T10:30:22.269161+0000","last_fullsized":"2024-09-16T10:30:22.269161+0000","mapping_epoch":33,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21602304,"num_objects":5274,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11873,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15161","reported_seq":15494,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.243528+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:22.243528+0000","last_peered":"2024-09-16T10:30:22.243528+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:22.243528+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21340160,"num_objects":5210,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11786,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15625","reported_seq":16034,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.394938+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:22.394938+0000","last_peered":"2024-09-16T10:30:22.394938+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:22.394938+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21688320,"num_objects":5295,"num_object_clones":0,"num_object_copies":10590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11867,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1008}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15175","reported_seq":15353,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:22.364198+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:22.364198+0000","last_peered":"2024-09-16T10:30:22.364198+0000","last_clean":"2024-09-16T10:30:22.364198+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:22.364198+0000","last_undegraded":"2024-09-16T10:30:22.364198+0000","last_fullsized":"2024-09-16T10:30:22.364198+0000","mapping_epoch":33,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21360640,"num_objects":5215,"num_object_clones":0,"num_object_copies":10430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10978,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15149","reported_seq":23656,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:21.011009+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:21.011009+0000","last_peered":"2024-09-16T10:30:21.011009+0000","last_clean":"2024-09-16T10:30:21.011009+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:21.011009+0000","last_undegraded":"2024-09-16T10:30:21.011009+0000","last_fullsized":"2024-09-16T10:30:21.011009+0000","mapping_epoch":43,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21758918,"num_objects":5314,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15147,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15714","reported_seq":15990,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.270827+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:22.270827+0000","last_peered":"2024-09-16T10:30:22.270827+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:22.270827+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21816262,"num_objects":5328,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15712,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":991}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15191","reported_seq":15384,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:21.885016+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:21.885016+0000","last_peered":"2024-09-16T10:30:21.885016+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:21.885016+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21414854,"num_objects":5230,"num_object_clones":0,"num_object_copies":10460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15189,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":997}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15201","reported_seq":19708,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:21.901523+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:21.901523+0000","last_peered":"2024-09-16T10:30:21.901523+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:21.901523+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22016966,"num_objects":5377,"num_object_clones":0,"num_object_copies":10754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15199,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2210,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14917","reported_seq":15168,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.402811+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:22.402811+0000","last_peered":"2024-09-16T10:30:22.402811+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:22.402811+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20959715,"num_objects":5118,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14916,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1043}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15581","reported_seq":15858,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.389918+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:22.389918+0000","last_peered":"2024-09-16T10:30:22.389918+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:22.389918+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21845417,"num_objects":5336,"num_object_clones":0,"num_object_copies":10672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15578,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1069}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15011","reported_seq":15243,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.264758+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:22.264758+0000","last_peered":"2024-09-16T10:30:22.264758+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:22.264758+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21009350,"num_objects":5131,"num_object_clones":0,"num_object_copies":10262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15009,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15632","reported_seq":15874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.371552+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:22.371552+0000","last_peered":"2024-09-16T10:30:22.371552+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:22.371552+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21563276,"num_objects":5268,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15629,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":343843342,"num_objects":83963,"num_object_clones":0,"num_object_copies":167926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83963,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222342,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11291,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":899108864,"data_stored":899024847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018975,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608720,"kb_used_data":306376,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93763120,"statfs":{"total":96636764160,"available":96013434880,"internally_reserved":0,"allocated":313729024,"data_stored":313460873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.024},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.2290000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691919,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570696,"kb_used_data":254396,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93801144,"statfs":{"total":96636764160,"available":96052371456,"internally_reserved":0,"allocated":260501504,"data_stored":260168319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":323874474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.0779999999999998},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":5.8099999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.726},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.23899999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215202,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716572,"kb_used_data":319936,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93655268,"statfs":{"total":96636764160,"available":95902994432,"internally_reserved":0,"allocated":327614464,"data_stored":327353999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":406122152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.3439999999999999},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.0710000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.8840000000000003},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.3769999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326582272,"data_stored":326578659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259915776,"data_stored":259843057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312610816,"data_stored":312603131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:25.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:26.026 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:26.027 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:26.050 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-16T10:30:24.711123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":343843361,"num_objects":83965,"num_object_clones":0,"num_object_copies":167930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83965,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222344,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11292,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160780,"ondisk_log_size":160780,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1895988,"kb_used_data":880708,"kb_used_omap":12,"kb_used_meta":1015219,"kb_avail":281219532,"statfs":{"total":289910292480,"available":287968800768,"internally_reserved":0,"allocated":901844992,"data_stored":900983191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1039584258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,2,13,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18493440,"num_objects":-4515,"num_object_clones":0,"num_object_copies":-9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4515,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004040"},"pg_stats":[{"pgid":"2.f","version":"46'15200","reported_seq":23665,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:22.395889+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:22.395889+0000","last_peered":"2024-09-16T10:30:22.395889+0000","last_clean":"2024-09-16T10:30:22.395889+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:22.395889+0000","last_undegraded":"2024-09-16T10:30:22.395889+0000","last_fullsized":"2024-09-16T10:30:22.395889+0000","mapping_epoch":45,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21577728,"num_objects":5268,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15200,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15285","reported_seq":15445,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:21.790309+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:21.790309+0000","last_peered":"2024-09-16T10:30:21.790309+0000","last_clean":"2024-09-16T10:30:21.790309+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:21.790309+0000","last_undegraded":"2024-09-16T10:30:21.790309+0000","last_fullsized":"2024-09-16T10:30:21.790309+0000","mapping_epoch":34,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21204992,"num_objects":5177,"num_object_clones":0,"num_object_copies":10354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11498,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15412","reported_seq":15666,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:21.852924+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:21.852924+0000","last_peered":"2024-09-16T10:30:21.852924+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:21.852924+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21508120,"num_objects":5252,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11469,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15292","reported_seq":15351,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:21.727117+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:21.727117+0000","last_peered":"2024-09-16T10:30:21.727117+0000","last_clean":"2024-09-16T10:30:21.727117+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:21.727117+0000","last_undegraded":"2024-09-16T10:30:21.727117+0000","last_fullsized":"2024-09-16T10:30:21.727117+0000","mapping_epoch":34,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21176320,"num_objects":5170,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15292,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15231","reported_seq":15391,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:22.269161+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:22.269161+0000","last_peered":"2024-09-16T10:30:22.269161+0000","last_clean":"2024-09-16T10:30:22.269161+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:22.269161+0000","last_undegraded":"2024-09-16T10:30:22.269161+0000","last_fullsized":"2024-09-16T10:30:22.269161+0000","mapping_epoch":33,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21602304,"num_objects":5274,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11873,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15161","reported_seq":15494,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.243528+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:22.243528+0000","last_peered":"2024-09-16T10:30:22.243528+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:22.243528+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21340160,"num_objects":5210,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11786,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15625","reported_seq":16034,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.394938+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:22.394938+0000","last_peered":"2024-09-16T10:30:22.394938+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:22.394938+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21688320,"num_objects":5295,"num_object_clones":0,"num_object_copies":10590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11867,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1008}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15175","reported_seq":15353,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:22.364198+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:22.364198+0000","last_peered":"2024-09-16T10:30:22.364198+0000","last_clean":"2024-09-16T10:30:22.364198+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:22.364198+0000","last_undegraded":"2024-09-16T10:30:22.364198+0000","last_fullsized":"2024-09-16T10:30:22.364198+0000","mapping_epoch":33,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21360640,"num_objects":5215,"num_object_clones":0,"num_object_copies":10430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10978,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15149","reported_seq":23656,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:21.011009+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:21.011009+0000","last_peered":"2024-09-16T10:30:21.011009+0000","last_clean":"2024-09-16T10:30:21.011009+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:21.011009+0000","last_undegraded":"2024-09-16T10:30:21.011009+0000","last_fullsized":"2024-09-16T10:30:21.011009+0000","mapping_epoch":43,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21758918,"num_objects":5314,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15147,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15714","reported_seq":15990,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.270827+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:22.270827+0000","last_peered":"2024-09-16T10:30:22.270827+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:22.270827+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21816262,"num_objects":5328,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15712,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":991}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15191","reported_seq":15384,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:21.885016+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:21.885016+0000","last_peered":"2024-09-16T10:30:21.885016+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:21.885016+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21414854,"num_objects":5230,"num_object_clones":0,"num_object_copies":10460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15189,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":997}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15201","reported_seq":19708,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:21.901523+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:21.901523+0000","last_peered":"2024-09-16T10:30:21.901523+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:21.901523+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22016966,"num_objects":5377,"num_object_clones":0,"num_object_copies":10754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15199,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2210,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14917","reported_seq":15168,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.402811+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:22.402811+0000","last_peered":"2024-09-16T10:30:22.402811+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:22.402811+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20959715,"num_objects":5118,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14916,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1043}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15581","reported_seq":15858,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.389918+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:22.389918+0000","last_peered":"2024-09-16T10:30:22.389918+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:22.389918+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21845417,"num_objects":5336,"num_object_clones":0,"num_object_copies":10672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15578,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1069}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15011","reported_seq":15243,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.264758+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:22.264758+0000","last_peered":"2024-09-16T10:30:22.264758+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:22.264758+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21009350,"num_objects":5131,"num_object_clones":0,"num_object_copies":10262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15009,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":66,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.973168+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:02.973168+0000","last_peered":"2024-09-16T10:29:02.973168+0000","last_clean":"2024-09-16T10:29:02.973168+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:02.973168+0000","last_undegraded":"2024-09-16T10:29:02.973168+0000","last_fullsized":"2024-09-16T10:29:02.973168+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15632","reported_seq":15874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.371552+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:22.371552+0000","last_peered":"2024-09-16T10:30:22.371552+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:22.371552+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21563276,"num_objects":5268,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15629,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":343843342,"num_objects":83963,"num_object_clones":0,"num_object_copies":167926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83963,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222342,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11291,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":899108864,"data_stored":899024847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160777,"ondisk_log_size":160777,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018975,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608720,"kb_used_data":306376,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93763120,"statfs":{"total":96636764160,"available":96013434880,"internally_reserved":0,"allocated":313729024,"data_stored":313460873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.024},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.2290000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691919,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570696,"kb_used_data":254396,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93801144,"statfs":{"total":96636764160,"available":96052371456,"internally_reserved":0,"allocated":260501504,"data_stored":260168319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":323874474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.0779999999999998},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":5.8099999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.726},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.23899999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215202,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716572,"kb_used_data":319936,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93655268,"statfs":{"total":96636764160,"available":95902994432,"internally_reserved":0,"allocated":327614464,"data_stored":327353999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":406122152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.3439999999999999},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.0710000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.8840000000000003},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.3769999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326582272,"data_stored":326578659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259915776,"data_stored":259843057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312610816,"data_stored":312603131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:26.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:26.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'15412", 'reported_seq': 15666, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:21.852924+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:21.852924+0000', 'last_peered': '2024-09-16T10:30:21.852924+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:21.852924+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'5400", 'ondisk_log_start': "42'5400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21508120, 'num_objects': 5252, 'num_object_clones': 0, 'num_object_copies': 10504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5252, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11469, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 949}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:26.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:26.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'15161", 'reported_seq': 15494, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:22.243528+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:22.243528+0000', 'last_peered': '2024-09-16T10:30:22.243528+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:22.243528+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'5100", 'ondisk_log_start': "42'5100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21340160, 'num_objects': 5210, 'num_object_clones': 0, 'num_object_copies': 10420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11786, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1076}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:26.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:26.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'15625", 'reported_seq': 16034, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:22.394938+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:22.394938+0000', 'last_peered': '2024-09-16T10:30:22.394938+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:22.394938+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'5600", 'ondisk_log_start': "42'5600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21688320, 'num_objects': 5295, 'num_object_clones': 0, 'num_object_copies': 10590, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11867, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1008}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:26.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:26.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'15714", 'reported_seq': 15990, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:22.270827+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:22.270827+0000', 'last_peered': '2024-09-16T10:30:22.270827+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:22.270827+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'5700", 'ondisk_log_start': "42'5700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21816262, 'num_objects': 5328, 'num_object_clones': 0, 'num_object_copies': 10656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15712, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 991}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:26.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:26.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'15191", 'reported_seq': 15384, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:21.885016+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:21.885016+0000', 'last_peered': '2024-09-16T10:30:21.885016+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:21.885016+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'5100", 'ondisk_log_start': "42'5100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21414854, 'num_objects': 5230, 'num_object_clones': 0, 'num_object_copies': 10460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15189, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 997}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:26.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:26.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'15201", 'reported_seq': 19708, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:21.901523+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:21.901523+0000', 'last_peered': '2024-09-16T10:30:21.901523+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:21.901523+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'5200", 'ondisk_log_start': "42'5200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22016966, 'num_objects': 5377, 'num_object_clones': 0, 'num_object_copies': 10754, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15199, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2210, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3391}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:26.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:26.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'14917", 'reported_seq': 15168, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:22.402811+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:22.402811+0000', 'last_peered': '2024-09-16T10:30:22.402811+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:22.402811+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'4900", 'ondisk_log_start': "42'4900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20959715, 'num_objects': 5118, 'num_object_clones': 0, 'num_object_copies': 10236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14916, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1043}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:26.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:26.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'15581", 'reported_seq': 15858, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:22.389918+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:22.389918+0000', 'last_peered': '2024-09-16T10:30:22.389918+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:22.389918+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'5500", 'ondisk_log_start': "42'5500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21845417, 'num_objects': 5336, 'num_object_clones': 0, 'num_object_copies': 10672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5336, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15578, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1069}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:26.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:26.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'15011", 'reported_seq': 15243, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:22.264758+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:22.264758+0000', 'last_peered': '2024-09-16T10:30:22.264758+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:22.264758+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'5000", 'ondisk_log_start': "42'5000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21009350, 'num_objects': 5131, 'num_object_clones': 0, 'num_object_copies': 10262, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15009, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 1038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:26.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:26.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'15632", 'reported_seq': 15874, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:22.371552+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:22.371552+0000', 'last_peered': '2024-09-16T10:30:22.371552+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:22.371552+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'5600", 'ondisk_log_start': "42'5600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21563276, 'num_objects': 5268, 'num_object_clones': 0, 'num_object_copies': 10536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5268, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15629, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 946}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:26.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:26.743 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:26.743 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 40055/167930 objects degraded (23.852%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":515,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":83965,"data_bytes":343843361,"bytes_used":1941491712,"bytes_avail":287968800768,"bytes_total":289910292480,"degraded_objects":40055,"degraded_total":167930,"degraded_ratio":0.23852200321562556,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":267,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":376},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-16T10:29:36.693808+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-16T10:30:26.768 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:26.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:27.309 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:27.309 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:27.331 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-16T10:30:26.711563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":343183905,"num_objects":83804,"num_object_clones":0,"num_object_copies":167608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83804,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222505,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11292,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1893092,"kb_used_data":873720,"kb_used_omap":12,"kb_used_meta":1019315,"kb_avail":281222428,"statfs":{"total":289910292480,"available":287971766272,"internally_reserved":0,"allocated":894689280,"data_stored":893827479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1043778562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,3,11,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19152896,"num_objects":-4676,"num_object_clones":0,"num_object_copies":-9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4676,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003976"},"pg_stats":[{"pgid":"2.f","version":"46'15200","reported_seq":23665,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:22.395889+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:22.395889+0000","last_peered":"2024-09-16T10:30:22.395889+0000","last_clean":"2024-09-16T10:30:22.395889+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:22.395889+0000","last_undegraded":"2024-09-16T10:30:22.395889+0000","last_fullsized":"2024-09-16T10:30:22.395889+0000","mapping_epoch":45,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21577728,"num_objects":5268,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15200,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15285","reported_seq":15445,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:21.790309+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:21.790309+0000","last_peered":"2024-09-16T10:30:21.790309+0000","last_clean":"2024-09-16T10:30:21.790309+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:21.790309+0000","last_undegraded":"2024-09-16T10:30:21.790309+0000","last_fullsized":"2024-09-16T10:30:21.790309+0000","mapping_epoch":34,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21204992,"num_objects":5177,"num_object_clones":0,"num_object_copies":10354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11498,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15412","reported_seq":15666,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:21.852924+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:21.852924+0000","last_peered":"2024-09-16T10:30:21.852924+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:21.852924+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21508120,"num_objects":5252,"num_object_clones":0,"num_object_copies":10504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11469,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15292","reported_seq":15351,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:21.727117+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:21.727117+0000","last_peered":"2024-09-16T10:30:21.727117+0000","last_clean":"2024-09-16T10:30:21.727117+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:21.727117+0000","last_undegraded":"2024-09-16T10:30:21.727117+0000","last_fullsized":"2024-09-16T10:30:21.727117+0000","mapping_epoch":34,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21176320,"num_objects":5170,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15292,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15231","reported_seq":15391,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:22.269161+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:22.269161+0000","last_peered":"2024-09-16T10:30:22.269161+0000","last_clean":"2024-09-16T10:30:22.269161+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:22.269161+0000","last_undegraded":"2024-09-16T10:30:22.269161+0000","last_fullsized":"2024-09-16T10:30:22.269161+0000","mapping_epoch":33,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21602304,"num_objects":5274,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11873,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15161","reported_seq":15494,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.243528+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:22.243528+0000","last_peered":"2024-09-16T10:30:22.243528+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:22.243528+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21340160,"num_objects":5210,"num_object_clones":0,"num_object_copies":10420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11786,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15625","reported_seq":16034,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.394938+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:22.394938+0000","last_peered":"2024-09-16T10:30:22.394938+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:22.394938+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21688320,"num_objects":5295,"num_object_clones":0,"num_object_copies":10590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11867,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1008}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15175","reported_seq":15353,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:22.364198+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:22.364198+0000","last_peered":"2024-09-16T10:30:22.364198+0000","last_clean":"2024-09-16T10:30:22.364198+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:22.364198+0000","last_undegraded":"2024-09-16T10:30:22.364198+0000","last_fullsized":"2024-09-16T10:30:22.364198+0000","mapping_epoch":33,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21360640,"num_objects":5215,"num_object_clones":0,"num_object_copies":10430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10978,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15310","reported_seq":23817,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:26.292881+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:26.292881+0000","last_peered":"2024-09-16T10:30:26.292881+0000","last_clean":"2024-09-16T10:30:26.292881+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:26.292881+0000","last_undegraded":"2024-09-16T10:30:26.292881+0000","last_fullsized":"2024-09-16T10:30:26.292881+0000","mapping_epoch":43,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21099462,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15308,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15714","reported_seq":15990,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.270827+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:22.270827+0000","last_peered":"2024-09-16T10:30:22.270827+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:22.270827+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21816262,"num_objects":5328,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15712,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":991}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15191","reported_seq":15384,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:21.885016+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:21.885016+0000","last_peered":"2024-09-16T10:30:21.885016+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:21.885016+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21414854,"num_objects":5230,"num_object_clones":0,"num_object_copies":10460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15189,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":997}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15201","reported_seq":19708,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:21.901523+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:21.901523+0000","last_peered":"2024-09-16T10:30:21.901523+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:21.901523+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22016966,"num_objects":5377,"num_object_clones":0,"num_object_copies":10754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15199,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2210,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'14917","reported_seq":15168,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.402811+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:22.402811+0000","last_peered":"2024-09-16T10:30:22.402811+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:22.402811+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'4900","ondisk_log_start":"42'4900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20959715,"num_objects":5118,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14916,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1043}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15581","reported_seq":15858,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.389918+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:22.389918+0000","last_peered":"2024-09-16T10:30:22.389918+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:22.389918+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21845417,"num_objects":5336,"num_object_clones":0,"num_object_copies":10672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15578,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1069}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15011","reported_seq":15243,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.264758+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:22.264758+0000","last_peered":"2024-09-16T10:30:22.264758+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:22.264758+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21009350,"num_objects":5131,"num_object_clones":0,"num_object_copies":10262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15009,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15632","reported_seq":15874,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:22.371552+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:22.371552+0000","last_peered":"2024-09-16T10:30:22.371552+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:22.371552+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21563276,"num_objects":5268,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15629,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":946}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":343183886,"num_objects":83802,"num_object_clones":0,"num_object_copies":167604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83802,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":222503,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11291,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":891801600,"data_stored":891717583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160738,"ondisk_log_size":160738,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018975,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608720,"kb_used_data":306376,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93763120,"statfs":{"total":96636764160,"available":96013434880,"internally_reserved":0,"allocated":313729024,"data_stored":313460873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.024},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.2290000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691920,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567800,"kb_used_data":247408,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93804040,"statfs":{"total":96636764160,"available":96055336960,"internally_reserved":0,"allocated":253345792,"data_stored":253012607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":328068778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.36799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.958},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.8720000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215202,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716572,"kb_used_data":319936,"kb_used_omap":4,"kb_used_meta":396603,"kb_avail":93655268,"statfs":{"total":96636764160,"available":95902994432,"internally_reserved":0,"allocated":327614464,"data_stored":327353999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":406122152},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.3439999999999999},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":3.0710000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.8840000000000003},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.3769999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326582272,"data_stored":326578659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252608512,"data_stored":252535793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":312610816,"data_stored":312603131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:30.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:30.884 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:30.887 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:30.910 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-16T10:30:30.712544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334504481,"num_objects":81685,"num_object_clones":0,"num_object_copies":163370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224624,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11429,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1890020,"kb_used_data":862464,"kb_used_omap":12,"kb_used_meta":1027507,"kb_avail":281225500,"statfs":{"total":289910292480,"available":287974912000,"internally_reserved":0,"allocated":883163136,"data_stored":882301335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1052167170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,2,7,8,6,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18567168,"num_objects":-4533,"num_object_clones":0,"num_object_copies":-9066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004044"},"pg_stats":[{"pgid":"2.f","version":"46'15351","reported_seq":23816,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:27.400132+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:27.400132+0000","last_peered":"2024-09-16T10:30:27.400132+0000","last_clean":"2024-09-16T10:30:27.400132+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:27.400132+0000","last_undegraded":"2024-09-16T10:30:27.400132+0000","last_fullsized":"2024-09-16T10:30:27.400132+0000","mapping_epoch":45,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20959232,"num_objects":5117,"num_object_clones":0,"num_object_copies":10234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15351,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15413","reported_seq":15573,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:26.855179+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:26.855179+0000","last_peered":"2024-09-16T10:30:26.855179+0000","last_clean":"2024-09-16T10:30:26.855179+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:26.855179+0000","last_undegraded":"2024-09-16T10:30:26.855179+0000","last_fullsized":"2024-09-16T10:30:26.855179+0000","mapping_epoch":34,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20680704,"num_objects":5049,"num_object_clones":0,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15556","reported_seq":15810,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:26.843444+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:26.843444+0000","last_peered":"2024-09-16T10:30:26.843444+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:26.843444+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20918296,"num_objects":5108,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11613,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15415","reported_seq":15475,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:26.886027+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:26.886027+0000","last_peered":"2024-09-16T10:30:26.886027+0000","last_clean":"2024-09-16T10:30:26.886027+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:26.886027+0000","last_undegraded":"2024-09-16T10:30:26.886027+0000","last_fullsized":"2024-09-16T10:30:26.886027+0000","mapping_epoch":34,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20672512,"num_objects":5047,"num_object_clones":0,"num_object_copies":10094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15415,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15365","reported_seq":15525,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:27.410869+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:27.410869+0000","last_peered":"2024-09-16T10:30:27.410869+0000","last_clean":"2024-09-16T10:30:27.410869+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:27.410869+0000","last_undegraded":"2024-09-16T10:30:27.410869+0000","last_fullsized":"2024-09-16T10:30:27.410869+0000","mapping_epoch":33,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21053440,"num_objects":5140,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12007,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15285","reported_seq":15618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.347201+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:27.347201+0000","last_peered":"2024-09-16T10:30:27.347201+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:27.347201+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20832256,"num_objects":5086,"num_object_clones":0,"num_object_copies":10172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15761","reported_seq":16170,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.380711+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:27.380711+0000","last_peered":"2024-09-16T10:30:27.380711+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:27.380711+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21131264,"num_objects":5159,"num_object_clones":0,"num_object_copies":10318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12003,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":872}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15323","reported_seq":15499,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:27.381298+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:27.381298+0000","last_peered":"2024-09-16T10:30:27.381298+0000","last_clean":"2024-09-16T10:30:27.381298+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:27.381298+0000","last_undegraded":"2024-09-16T10:30:27.381298+0000","last_fullsized":"2024-09-16T10:30:27.381298+0000","mapping_epoch":33,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20754432,"num_objects":5067,"num_object_clones":0,"num_object_copies":10134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11126,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15310","reported_seq":23817,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:26.292881+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:26.292881+0000","last_peered":"2024-09-16T10:30:26.292881+0000","last_clean":"2024-09-16T10:30:26.292881+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:26.292881+0000","last_undegraded":"2024-09-16T10:30:26.292881+0000","last_fullsized":"2024-09-16T10:30:26.292881+0000","mapping_epoch":43,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21099462,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15308,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15879","reported_seq":16154,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.396688+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:27.396688+0000","last_peered":"2024-09-16T10:30:27.396688+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:27.396688+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21140422,"num_objects":5163,"num_object_clones":0,"num_object_copies":10326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15877,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":826}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15329","reported_seq":15526,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:26.796398+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:26.796398+0000","last_peered":"2024-09-16T10:30:26.796398+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:26.796398+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20849606,"num_objects":5092,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15327,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15344","reported_seq":20125,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:26.899337+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:26.899337+0000","last_peered":"2024-09-16T10:30:26.899337+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:26.899337+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21431238,"num_objects":5234,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15342,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2347,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15041","reported_seq":15292,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.400553+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:27.400553+0000","last_peered":"2024-09-16T10:30:27.400553+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:27.400553+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20451811,"num_objects":4994,"num_object_clones":0,"num_object_copies":9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15040,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15722","reported_seq":15999,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.306056+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:27.306056+0000","last_peered":"2024-09-16T10:30:27.306056+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:27.306056+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21267881,"num_objects":5195,"num_object_clones":0,"num_object_copies":10390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15719,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15175","reported_seq":15407,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.435021+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:27.435021+0000","last_peered":"2024-09-16T10:30:27.435021+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:27.435021+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337606,"num_objects":4967,"num_object_clones":0,"num_object_copies":9934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15173,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":874}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15788","reported_seq":16030,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.272219+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:27.272219+0000","last_peered":"2024-09-16T10:30:27.272219+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:27.272219+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20924300,"num_objects":5112,"num_object_clones":0,"num_object_copies":10224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15785,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":334504462,"num_objects":81683,"num_object_clones":0,"num_object_copies":163366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224622,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11428,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":880295936,"data_stored":880211919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160757,"ondisk_log_size":160757,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018976,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608400,"kb_used_data":301960,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93763440,"statfs":{"total":96636764160,"available":96013762560,"internally_reserved":0,"allocated":309207040,"data_stored":308938889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":313781936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.0800000000000001},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":8.6869999999999994},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":8.8889999999999993}]}]},{"osd":1,"up_from":40,"seq":171798691920,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567800,"kb_used_data":247408,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93804040,"statfs":{"total":96636764160,"available":96055336960,"internally_reserved":0,"allocated":253345792,"data_stored":253012607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":328068778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.36799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.958},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.8720000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215203,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713820,"kb_used_data":313096,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93658020,"statfs":{"total":96636764160,"available":95905812480,"internally_reserved":0,"allocated":320610304,"data_stored":320349839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":410316456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,4,1,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.9779999999999998},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":7.0780000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.65000000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.8680000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319676416,"data_stored":319672803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252608512,"data_stored":252535793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308011008,"data_stored":308003323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:30.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:31.470 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:31.470 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:31.492 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-16T10:30:30.712544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":334504481,"num_objects":81685,"num_object_clones":0,"num_object_copies":163370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224624,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11429,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160760,"ondisk_log_size":160760,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1890020,"kb_used_data":862464,"kb_used_omap":12,"kb_used_meta":1027507,"kb_avail":281225500,"statfs":{"total":289910292480,"available":287974912000,"internally_reserved":0,"allocated":883163136,"data_stored":882301335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1052167170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,2,7,8,6,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18567168,"num_objects":-4533,"num_object_clones":0,"num_object_copies":-9066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4533,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004044"},"pg_stats":[{"pgid":"2.f","version":"46'15351","reported_seq":23816,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:27.400132+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:27.400132+0000","last_peered":"2024-09-16T10:30:27.400132+0000","last_clean":"2024-09-16T10:30:27.400132+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:27.400132+0000","last_undegraded":"2024-09-16T10:30:27.400132+0000","last_fullsized":"2024-09-16T10:30:27.400132+0000","mapping_epoch":45,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20959232,"num_objects":5117,"num_object_clones":0,"num_object_copies":10234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15351,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15413","reported_seq":15573,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:26.855179+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:26.855179+0000","last_peered":"2024-09-16T10:30:26.855179+0000","last_clean":"2024-09-16T10:30:26.855179+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:26.855179+0000","last_undegraded":"2024-09-16T10:30:26.855179+0000","last_fullsized":"2024-09-16T10:30:26.855179+0000","mapping_epoch":34,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20680704,"num_objects":5049,"num_object_clones":0,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15556","reported_seq":15810,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:26.843444+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:26.843444+0000","last_peered":"2024-09-16T10:30:26.843444+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:26.843444+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20918296,"num_objects":5108,"num_object_clones":0,"num_object_copies":10216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11613,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":805}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15415","reported_seq":15475,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:26.886027+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:26.886027+0000","last_peered":"2024-09-16T10:30:26.886027+0000","last_clean":"2024-09-16T10:30:26.886027+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:26.886027+0000","last_undegraded":"2024-09-16T10:30:26.886027+0000","last_fullsized":"2024-09-16T10:30:26.886027+0000","mapping_epoch":34,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20672512,"num_objects":5047,"num_object_clones":0,"num_object_copies":10094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15415,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15365","reported_seq":15525,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:27.410869+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:27.410869+0000","last_peered":"2024-09-16T10:30:27.410869+0000","last_clean":"2024-09-16T10:30:27.410869+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:27.410869+0000","last_undegraded":"2024-09-16T10:30:27.410869+0000","last_fullsized":"2024-09-16T10:30:27.410869+0000","mapping_epoch":33,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21053440,"num_objects":5140,"num_object_clones":0,"num_object_copies":10280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12007,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15285","reported_seq":15618,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.347201+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:27.347201+0000","last_peered":"2024-09-16T10:30:27.347201+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:27.347201+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20832256,"num_objects":5086,"num_object_clones":0,"num_object_copies":10172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11910,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":952}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15761","reported_seq":16170,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.380711+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:27.380711+0000","last_peered":"2024-09-16T10:30:27.380711+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:27.380711+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21131264,"num_objects":5159,"num_object_clones":0,"num_object_copies":10318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12003,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":872}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15323","reported_seq":15499,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:27.381298+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:27.381298+0000","last_peered":"2024-09-16T10:30:27.381298+0000","last_clean":"2024-09-16T10:30:27.381298+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:27.381298+0000","last_undegraded":"2024-09-16T10:30:27.381298+0000","last_fullsized":"2024-09-16T10:30:27.381298+0000","mapping_epoch":33,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20754432,"num_objects":5067,"num_object_clones":0,"num_object_copies":10134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11126,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15310","reported_seq":23817,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:26.292881+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:26.292881+0000","last_peered":"2024-09-16T10:30:26.292881+0000","last_clean":"2024-09-16T10:30:26.292881+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:26.292881+0000","last_undegraded":"2024-09-16T10:30:26.292881+0000","last_fullsized":"2024-09-16T10:30:26.292881+0000","mapping_epoch":43,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21099462,"num_objects":5153,"num_object_clones":0,"num_object_copies":10306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15308,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'15879","reported_seq":16154,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.396688+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:27.396688+0000","last_peered":"2024-09-16T10:30:27.396688+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:27.396688+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21140422,"num_objects":5163,"num_object_clones":0,"num_object_copies":10326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15877,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":826}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15329","reported_seq":15526,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:26.796398+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:26.796398+0000","last_peered":"2024-09-16T10:30:26.796398+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:26.796398+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20849606,"num_objects":5092,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15327,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15344","reported_seq":20125,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:26.899337+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:26.899337+0000","last_peered":"2024-09-16T10:30:26.899337+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:26.899337+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21431238,"num_objects":5234,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15342,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2347,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3385}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15041","reported_seq":15292,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.400553+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:27.400553+0000","last_peered":"2024-09-16T10:30:27.400553+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:27.400553+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5000","ondisk_log_start":"42'5000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20451811,"num_objects":4994,"num_object_clones":0,"num_object_copies":9988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15040,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15722","reported_seq":15999,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.306056+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:27.306056+0000","last_peered":"2024-09-16T10:30:27.306056+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:27.306056+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21267881,"num_objects":5195,"num_object_clones":0,"num_object_copies":10390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15719,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15175","reported_seq":15407,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.435021+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:27.435021+0000","last_peered":"2024-09-16T10:30:27.435021+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:27.435021+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20337606,"num_objects":4967,"num_object_clones":0,"num_object_copies":9934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15173,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":874}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15788","reported_seq":16030,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:27.272219+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:27.272219+0000","last_peered":"2024-09-16T10:30:27.272219+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:27.272219+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20924300,"num_objects":5112,"num_object_clones":0,"num_object_copies":10224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15785,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":334504462,"num_objects":81683,"num_object_clones":0,"num_object_copies":163366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224622,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11428,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":880295936,"data_stored":880211919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160757,"ondisk_log_size":160757,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018976,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":608400,"kb_used_data":301960,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93763440,"statfs":{"total":96636764160,"available":96013762560,"internally_reserved":0,"allocated":309207040,"data_stored":308938889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":313781936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":3.0800000000000001},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":8.6869999999999994},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":8.8889999999999993}]}]},{"osd":1,"up_from":40,"seq":171798691920,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567800,"kb_used_data":247408,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93804040,"statfs":{"total":96636764160,"available":96055336960,"internally_reserved":0,"allocated":253345792,"data_stored":253012607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":328068778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.36799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.958},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.8720000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215203,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713820,"kb_used_data":313096,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93658020,"statfs":{"total":96636764160,"available":95905812480,"internally_reserved":0,"allocated":320610304,"data_stored":320349839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":410316456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,4,1,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.9779999999999998},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":7.0780000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.65000000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.8680000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319676416,"data_stored":319672803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":252608512,"data_stored":252535793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308011008,"data_stored":308003323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:31.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:31.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'15556", 'reported_seq': 15810, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:26.843444+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:26.843444+0000', 'last_peered': '2024-09-16T10:30:26.843444+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:26.843444+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'5500", 'ondisk_log_start': "42'5500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20918296, 'num_objects': 5108, 'num_object_clones': 0, 'num_object_copies': 10216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5108, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11613, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 805}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:31.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:31.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'15285", 'reported_seq': 15618, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:27.347201+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:27.347201+0000', 'last_peered': '2024-09-16T10:30:27.347201+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:27.347201+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'5200", 'ondisk_log_start': "42'5200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20832256, 'num_objects': 5086, 'num_object_clones': 0, 'num_object_copies': 10172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11910, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 952}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:31.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:31.494 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'15761", 'reported_seq': 16170, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:27.380711+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:27.380711+0000', 'last_peered': '2024-09-16T10:30:27.380711+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:27.380711+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'5700", 'ondisk_log_start': "42'5700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21131264, 'num_objects': 5159, 'num_object_clones': 0, 'num_object_copies': 10318, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12003, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 872}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'15879", 'reported_seq': 16154, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:27.396688+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:27.396688+0000', 'last_peered': '2024-09-16T10:30:27.396688+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:27.396688+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'5800", 'ondisk_log_start': "42'5800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21140422, 'num_objects': 5163, 'num_object_clones': 0, 'num_object_copies': 10326, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15877, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 826}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'15329", 'reported_seq': 15526, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:26.796398+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:26.796398+0000', 'last_peered': '2024-09-16T10:30:26.796398+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:26.796398+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'5300", 'ondisk_log_start': "42'5300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20849606, 'num_objects': 5092, 'num_object_clones': 0, 'num_object_copies': 10184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15327, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 859}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'15344", 'reported_seq': 20125, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:26.899337+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:26.899337+0000', 'last_peered': '2024-09-16T10:30:26.899337+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:26.899337+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'5300", 'ondisk_log_start': "42'5300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21431238, 'num_objects': 5234, 'num_object_clones': 0, 'num_object_copies': 10468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1849, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15342, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2347, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3385}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'15041", 'reported_seq': 15292, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:27.400553+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:27.400553+0000', 'last_peered': '2024-09-16T10:30:27.400553+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:27.400553+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'5000", 'ondisk_log_start': "42'5000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20451811, 'num_objects': 4994, 'num_object_clones': 0, 'num_object_copies': 9988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15040, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 919}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'15722", 'reported_seq': 15999, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:27.306056+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:27.306056+0000', 'last_peered': '2024-09-16T10:30:27.306056+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:27.306056+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'5700", 'ondisk_log_start': "42'5700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21267881, 'num_objects': 5195, 'num_object_clones': 0, 'num_object_copies': 10390, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15719, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 928}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:31.495 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'15175", 'reported_seq': 15407, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:27.435021+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:27.435021+0000', 'last_peered': '2024-09-16T10:30:27.435021+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:27.435021+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'5100", 'ondisk_log_start': "42'5100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20337606, 'num_objects': 4967, 'num_object_clones': 0, 'num_object_copies': 9934, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15173, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 874}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:31.496 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:31.496 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'15788", 'reported_seq': 16030, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:27.272219+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:27.272219+0000', 'last_peered': '2024-09-16T10:30:27.272219+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:27.272219+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'5700", 'ondisk_log_start': "42'5700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20924300, 'num_objects': 5112, 'num_object_clones': 0, 'num_object_copies': 10224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15785, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 790}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:31.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:32.185 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:32.185 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39918/163370 objects degraded (24.434%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":520,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":81685,"data_bytes":334504481,"bytes_used":1935380480,"bytes_avail":287974912000,"bytes_total":289910292480,"degraded_objects":39918,"degraded_total":163370,"degraded_ratio":0.24434106629124075,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":272,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":377},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:30:32.213 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:32.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:32.784 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:32.785 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:32.808 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-16T10:30:32.712992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":325128737,"num_objects":79396,"num_object_clones":0,"num_object_copies":158792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226913,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11562,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160949,"ondisk_log_size":160949,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1871668,"kb_used_data":844108,"kb_used_omap":12,"kb_used_meta":1027507,"kb_avail":281243852,"statfs":{"total":289910292480,"available":287993704448,"internally_reserved":0,"allocated":864366592,"data_stored":863504791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1052167170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,0,10,6,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27942912,"num_objects":-6822,"num_object_clones":0,"num_object_copies":-13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6822,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002814"},"pg_stats":[{"pgid":"2.f","version":"46'15511","reported_seq":23976,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.426522+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:32.426522+0000","last_peered":"2024-09-16T10:30:32.426522+0000","last_clean":"2024-09-16T10:30:32.426522+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:32.426522+0000","last_undegraded":"2024-09-16T10:30:32.426522+0000","last_fullsized":"2024-09-16T10:30:32.426522+0000","mapping_epoch":45,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20303872,"num_objects":4957,"num_object_clones":0,"num_object_copies":9914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15511,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15544","reported_seq":15705,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:31.941342+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:31.941342+0000","last_peered":"2024-09-16T10:30:31.941342+0000","last_clean":"2024-09-16T10:30:31.941342+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:31.941342+0000","last_undegraded":"2024-09-16T10:30:31.941342+0000","last_fullsized":"2024-09-16T10:30:31.941342+0000","mapping_epoch":34,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20144128,"num_objects":4918,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15698","reported_seq":15952,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.915469+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:31.915469+0000","last_peered":"2024-09-16T10:30:31.915469+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:31.915469+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20336664,"num_objects":4966,"num_object_clones":0,"num_object_copies":9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11755,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15547","reported_seq":15607,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:31.736627+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:31.736627+0000","last_peered":"2024-09-16T10:30:31.736627+0000","last_clean":"2024-09-16T10:30:31.736627+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:31.736627+0000","last_undegraded":"2024-09-16T10:30:31.736627+0000","last_fullsized":"2024-09-16T10:30:31.736627+0000","mapping_epoch":34,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20131840,"num_objects":4915,"num_object_clones":0,"num_object_copies":9830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15547,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15542","reported_seq":15702,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.439883+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:32.439883+0000","last_peered":"2024-09-16T10:30:32.439883+0000","last_clean":"2024-09-16T10:30:32.439883+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:32.439883+0000","last_undegraded":"2024-09-16T10:30:32.439883+0000","last_fullsized":"2024-09-16T10:30:32.439883+0000","mapping_epoch":33,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20328448,"num_objects":4963,"num_object_clones":0,"num_object_copies":9926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15447","reported_seq":15780,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.419421+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:32.419421+0000","last_peered":"2024-09-16T10:30:32.419421+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:32.419421+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20168704,"num_objects":4924,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15900","reported_seq":16309,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.365871+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:32.365871+0000","last_peered":"2024-09-16T10:30:32.365871+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:32.365871+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20561920,"num_objects":5020,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12142,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":733}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15470","reported_seq":15649,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.408669+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:32.408669+0000","last_peered":"2024-09-16T10:30:32.408669+0000","last_clean":"2024-09-16T10:30:32.408669+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:32.408669+0000","last_undegraded":"2024-09-16T10:30:32.408669+0000","last_fullsized":"2024-09-16T10:30:32.408669+0000","mapping_epoch":33,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20152320,"num_objects":4920,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11273,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15429","reported_seq":23936,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:31.220241+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:31.220241+0000","last_peered":"2024-09-16T10:30:31.220241+0000","last_clean":"2024-09-16T10:30:31.220241+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:31.220241+0000","last_undegraded":"2024-09-16T10:30:31.220241+0000","last_fullsized":"2024-09-16T10:30:31.220241+0000","mapping_epoch":43,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20612038,"num_objects":5034,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15427,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16023","reported_seq":16299,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.323060+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:32.323060+0000","last_peered":"2024-09-16T10:30:32.323060+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:32.323060+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20550598,"num_objects":5019,"num_object_clones":0,"num_object_copies":10038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16021,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15457","reported_seq":15654,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.891071+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:31.891071+0000","last_peered":"2024-09-16T10:30:31.891071+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:31.891071+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20325318,"num_objects":4964,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15455,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":731}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15496","reported_seq":20543,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.909163+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:31.909163+0000","last_peered":"2024-09-16T10:30:31.909163+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:31.909163+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20808646,"num_objects":5082,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15494,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3366}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15171","reported_seq":15422,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.425490+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:32.425490+0000","last_peered":"2024-09-16T10:30:32.425490+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:32.425490+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19919331,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15170,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":789}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15878","reported_seq":16155,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.327823+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:32.327823+0000","last_peered":"2024-09-16T10:30:32.327823+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:32.327823+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20628905,"num_objects":5039,"num_object_clones":0,"num_object_copies":10078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15875,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15300","reported_seq":15532,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.325976+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:32.325976+0000","last_peered":"2024-09-16T10:30:32.325976+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:32.325976+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19825606,"num_objects":4842,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15298,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15933","reported_seq":16174,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.390375+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:32.390375+0000","last_peered":"2024-09-16T10:30:32.390375+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:32.390375+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20330380,"num_objects":4967,"num_object_clones":0,"num_object_copies":9934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15930,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":645}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":325128718,"num_objects":79394,"num_object_clones":0,"num_object_copies":158788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226911,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11561,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":861470720,"data_stored":861386703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160946,"ondisk_log_size":160946,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018977,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607988,"kb_used_data":297444,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93763852,"statfs":{"total":96636764160,"available":96014184448,"internally_reserved":0,"allocated":304582656,"data_stored":304314505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.6459999999999999},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.252},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":6.8280000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691921,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569128,"kb_used_data":240536,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93802712,"statfs":{"total":96636764160,"available":96053977088,"internally_reserved":0,"allocated":246308864,"data_stored":245975679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":336457386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.2359999999999998},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.92800000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.879}]}]},{"osd":0,"up_from":24,"seq":103079215204,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694552,"kb_used_data":306128,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93677288,"statfs":{"total":96636764160,"available":95925542912,"internally_reserved":0,"allocated":313475072,"data_stored":313214607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397733544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.849},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.8080000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":2.0910000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312532992,"data_stored":312529379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245555200,"data_stored":245482481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303382528,"data_stored":303374843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:35.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:36.344 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:36.344 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:36.375 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-09-16T10:30:34.713480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":325128737,"num_objects":79396,"num_object_clones":0,"num_object_copies":158792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79396,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226913,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11562,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160949,"ondisk_log_size":160949,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1871668,"kb_used_data":844108,"kb_used_omap":12,"kb_used_meta":1027507,"kb_avail":281243852,"statfs":{"total":289910292480,"available":287993704448,"internally_reserved":0,"allocated":864366592,"data_stored":863504791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1052167170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,0,10,6,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18714624,"num_objects":-4569,"num_object_clones":0,"num_object_copies":-9138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4569,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002852"},"pg_stats":[{"pgid":"2.f","version":"46'15511","reported_seq":23976,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.426522+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:32.426522+0000","last_peered":"2024-09-16T10:30:32.426522+0000","last_clean":"2024-09-16T10:30:32.426522+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:32.426522+0000","last_undegraded":"2024-09-16T10:30:32.426522+0000","last_fullsized":"2024-09-16T10:30:32.426522+0000","mapping_epoch":45,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20303872,"num_objects":4957,"num_object_clones":0,"num_object_copies":9914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15511,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15544","reported_seq":15705,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:31.941342+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:31.941342+0000","last_peered":"2024-09-16T10:30:31.941342+0000","last_clean":"2024-09-16T10:30:31.941342+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:31.941342+0000","last_undegraded":"2024-09-16T10:30:31.941342+0000","last_fullsized":"2024-09-16T10:30:31.941342+0000","mapping_epoch":34,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20144128,"num_objects":4918,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15698","reported_seq":15952,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.915469+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:31.915469+0000","last_peered":"2024-09-16T10:30:31.915469+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:31.915469+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20336664,"num_objects":4966,"num_object_clones":0,"num_object_copies":9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11755,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15547","reported_seq":15607,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:31.736627+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:31.736627+0000","last_peered":"2024-09-16T10:30:31.736627+0000","last_clean":"2024-09-16T10:30:31.736627+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:31.736627+0000","last_undegraded":"2024-09-16T10:30:31.736627+0000","last_fullsized":"2024-09-16T10:30:31.736627+0000","mapping_epoch":34,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20131840,"num_objects":4915,"num_object_clones":0,"num_object_copies":9830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15547,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15542","reported_seq":15702,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.439883+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:32.439883+0000","last_peered":"2024-09-16T10:30:32.439883+0000","last_clean":"2024-09-16T10:30:32.439883+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:32.439883+0000","last_undegraded":"2024-09-16T10:30:32.439883+0000","last_fullsized":"2024-09-16T10:30:32.439883+0000","mapping_epoch":33,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20328448,"num_objects":4963,"num_object_clones":0,"num_object_copies":9926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15447","reported_seq":15780,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.419421+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:32.419421+0000","last_peered":"2024-09-16T10:30:32.419421+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:32.419421+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20168704,"num_objects":4924,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15900","reported_seq":16309,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.365871+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:32.365871+0000","last_peered":"2024-09-16T10:30:32.365871+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:32.365871+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20561920,"num_objects":5020,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12142,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":733}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15470","reported_seq":15649,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.408669+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:32.408669+0000","last_peered":"2024-09-16T10:30:32.408669+0000","last_clean":"2024-09-16T10:30:32.408669+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:32.408669+0000","last_undegraded":"2024-09-16T10:30:32.408669+0000","last_fullsized":"2024-09-16T10:30:32.408669+0000","mapping_epoch":33,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20152320,"num_objects":4920,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11273,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15429","reported_seq":23936,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:31.220241+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:31.220241+0000","last_peered":"2024-09-16T10:30:31.220241+0000","last_clean":"2024-09-16T10:30:31.220241+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:31.220241+0000","last_undegraded":"2024-09-16T10:30:31.220241+0000","last_fullsized":"2024-09-16T10:30:31.220241+0000","mapping_epoch":43,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20612038,"num_objects":5034,"num_object_clones":0,"num_object_copies":10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15427,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16023","reported_seq":16299,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.323060+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:32.323060+0000","last_peered":"2024-09-16T10:30:32.323060+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:32.323060+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20550598,"num_objects":5019,"num_object_clones":0,"num_object_copies":10038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16021,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15457","reported_seq":15654,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.891071+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:31.891071+0000","last_peered":"2024-09-16T10:30:31.891071+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:31.891071+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20325318,"num_objects":4964,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15455,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":731}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15496","reported_seq":20543,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.909163+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:31.909163+0000","last_peered":"2024-09-16T10:30:31.909163+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:31.909163+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20808646,"num_objects":5082,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15494,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3366}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15171","reported_seq":15422,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.425490+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:32.425490+0000","last_peered":"2024-09-16T10:30:32.425490+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:32.425490+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19919331,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15170,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":789}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15878","reported_seq":16155,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.327823+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:32.327823+0000","last_peered":"2024-09-16T10:30:32.327823+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:32.327823+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20628905,"num_objects":5039,"num_object_clones":0,"num_object_copies":10078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15875,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15300","reported_seq":15532,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.325976+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:32.325976+0000","last_peered":"2024-09-16T10:30:32.325976+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:32.325976+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19825606,"num_objects":4842,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15298,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15933","reported_seq":16174,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.390375+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:32.390375+0000","last_peered":"2024-09-16T10:30:32.390375+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:32.390375+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20330380,"num_objects":4967,"num_object_clones":0,"num_object_copies":9934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15930,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":645}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":325128718,"num_objects":79394,"num_object_clones":0,"num_object_copies":158788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79394,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226911,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11561,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":861470720,"data_stored":861386703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160946,"ondisk_log_size":160946,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018977,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607988,"kb_used_data":297444,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93763852,"statfs":{"total":96636764160,"available":96014184448,"internally_reserved":0,"allocated":304582656,"data_stored":304314505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.6459999999999999},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.252},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":6.8280000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691921,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569128,"kb_used_data":240536,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93802712,"statfs":{"total":96636764160,"available":96053977088,"internally_reserved":0,"allocated":246308864,"data_stored":245975679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":336457386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":5.2359999999999998},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":0.92800000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.879}]}]},{"osd":0,"up_from":24,"seq":103079215204,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694552,"kb_used_data":306128,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93677288,"statfs":{"total":96636764160,"available":95925542912,"internally_reserved":0,"allocated":313475072,"data_stored":313214607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397733544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.849},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.8080000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":2.0910000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312532992,"data_stored":312529379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":245555200,"data_stored":245482481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303382528,"data_stored":303374843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:36.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:36.957 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:36.981 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-16T10:30:36.713922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":324547105,"num_objects":79254,"num_object_clones":0,"num_object_copies":158508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227055,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11562,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1852588,"kb_used_data":837324,"kb_used_omap":12,"kb_used_meta":1015219,"kb_avail":281262932,"statfs":{"total":289910292480,"available":288013242368,"internally_reserved":0,"allocated":857419776,"data_stored":856557975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1039584258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,0,3,9,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19296256,"num_objects":-4711,"num_object_clones":0,"num_object_copies":-9422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4711,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002798"},"pg_stats":[{"pgid":"2.f","version":"46'15511","reported_seq":23976,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.426522+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:32.426522+0000","last_peered":"2024-09-16T10:30:32.426522+0000","last_clean":"2024-09-16T10:30:32.426522+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:32.426522+0000","last_undegraded":"2024-09-16T10:30:32.426522+0000","last_fullsized":"2024-09-16T10:30:32.426522+0000","mapping_epoch":45,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20303872,"num_objects":4957,"num_object_clones":0,"num_object_copies":9914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15511,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15544","reported_seq":15705,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:31.941342+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:31.941342+0000","last_peered":"2024-09-16T10:30:31.941342+0000","last_clean":"2024-09-16T10:30:31.941342+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:31.941342+0000","last_undegraded":"2024-09-16T10:30:31.941342+0000","last_fullsized":"2024-09-16T10:30:31.941342+0000","mapping_epoch":34,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20144128,"num_objects":4918,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15698","reported_seq":15952,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.915469+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:31.915469+0000","last_peered":"2024-09-16T10:30:31.915469+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:31.915469+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20336664,"num_objects":4966,"num_object_clones":0,"num_object_copies":9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11755,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15547","reported_seq":15607,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:31.736627+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:31.736627+0000","last_peered":"2024-09-16T10:30:31.736627+0000","last_clean":"2024-09-16T10:30:31.736627+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:31.736627+0000","last_undegraded":"2024-09-16T10:30:31.736627+0000","last_fullsized":"2024-09-16T10:30:31.736627+0000","mapping_epoch":34,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20131840,"num_objects":4915,"num_object_clones":0,"num_object_copies":9830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15547,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15542","reported_seq":15702,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.439883+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:32.439883+0000","last_peered":"2024-09-16T10:30:32.439883+0000","last_clean":"2024-09-16T10:30:32.439883+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:32.439883+0000","last_undegraded":"2024-09-16T10:30:32.439883+0000","last_fullsized":"2024-09-16T10:30:32.439883+0000","mapping_epoch":33,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20328448,"num_objects":4963,"num_object_clones":0,"num_object_copies":9926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15447","reported_seq":15780,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.419421+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:32.419421+0000","last_peered":"2024-09-16T10:30:32.419421+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:32.419421+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20168704,"num_objects":4924,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15900","reported_seq":16309,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.365871+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:32.365871+0000","last_peered":"2024-09-16T10:30:32.365871+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:32.365871+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20561920,"num_objects":5020,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12142,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":733}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15470","reported_seq":15649,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.408669+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:32.408669+0000","last_peered":"2024-09-16T10:30:32.408669+0000","last_clean":"2024-09-16T10:30:32.408669+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:32.408669+0000","last_undegraded":"2024-09-16T10:30:32.408669+0000","last_fullsized":"2024-09-16T10:30:32.408669+0000","mapping_epoch":33,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20152320,"num_objects":4920,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11273,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15571","reported_seq":24077,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:36.333706+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:36.333706+0000","last_peered":"2024-09-16T10:30:36.333706+0000","last_clean":"2024-09-16T10:30:36.333706+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:36.333706+0000","last_undegraded":"2024-09-16T10:30:36.333706+0000","last_fullsized":"2024-09-16T10:30:36.333706+0000","mapping_epoch":43,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20030406,"num_objects":4892,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15569,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16023","reported_seq":16299,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.323060+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:32.323060+0000","last_peered":"2024-09-16T10:30:32.323060+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:32.323060+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20550598,"num_objects":5019,"num_object_clones":0,"num_object_copies":10038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16021,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15457","reported_seq":15654,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.891071+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:31.891071+0000","last_peered":"2024-09-16T10:30:31.891071+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:31.891071+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20325318,"num_objects":4964,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15455,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":731}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15496","reported_seq":20543,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.909163+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:31.909163+0000","last_peered":"2024-09-16T10:30:31.909163+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:31.909163+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20808646,"num_objects":5082,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15494,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3366}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15171","reported_seq":15422,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.425490+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:32.425490+0000","last_peered":"2024-09-16T10:30:32.425490+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:32.425490+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19919331,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15170,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":789}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15878","reported_seq":16155,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.327823+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:32.327823+0000","last_peered":"2024-09-16T10:30:32.327823+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:32.327823+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20628905,"num_objects":5039,"num_object_clones":0,"num_object_copies":10078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15875,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15300","reported_seq":15532,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.325976+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:32.325976+0000","last_peered":"2024-09-16T10:30:32.325976+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:32.325976+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19825606,"num_objects":4842,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15298,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15933","reported_seq":16174,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.390375+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:32.390375+0000","last_peered":"2024-09-16T10:30:32.390375+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:32.390375+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20330380,"num_objects":4967,"num_object_clones":0,"num_object_copies":9934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15930,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":645}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":324547086,"num_objects":79252,"num_object_clones":0,"num_object_copies":158504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227053,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11561,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":854360064,"data_stored":854276047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160988,"ondisk_log_size":160988,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018977,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607988,"kb_used_data":297444,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93763852,"statfs":{"total":96636764160,"available":96014184448,"internally_reserved":0,"allocated":304582656,"data_stored":304314505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.6459999999999999},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.252},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":6.8280000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691922,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550048,"kb_used_data":233752,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93821792,"statfs":{"total":96636764160,"available":96073515008,"internally_reserved":0,"allocated":239362048,"data_stored":239028863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":323874474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.974}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.0619999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.27700000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215204,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694552,"kb_used_data":306128,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93677288,"statfs":{"total":96636764160,"available":95925542912,"internally_reserved":0,"allocated":313475072,"data_stored":313214607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397733544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.849},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.8080000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":2.0910000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312532992,"data_stored":312529379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238444544,"data_stored":238371825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303382528,"data_stored":303374843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:36.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:36.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'15698", 'reported_seq': 15952, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:31.915469+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:31.915469+0000', 'last_peered': '2024-09-16T10:30:31.915469+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:31.915469+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'5600", 'ondisk_log_start': "42'5600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20336664, 'num_objects': 4966, 'num_object_clones': 0, 'num_object_copies': 9932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4966, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11755, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 663}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:36.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:36.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'15447", 'reported_seq': 15780, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:32.419421+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:32.419421+0000', 'last_peered': '2024-09-16T10:30:32.419421+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:32.419421+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'5400", 'ondisk_log_start': "42'5400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20168704, 'num_objects': 4924, 'num_object_clones': 0, 'num_object_copies': 9848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12072, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 790}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:36.983 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'15900", 'reported_seq': 16309, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:32.365871+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:32.365871+0000', 'last_peered': '2024-09-16T10:30:32.365871+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:32.365871+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'5800", 'ondisk_log_start': "42'5800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20561920, 'num_objects': 5020, 'num_object_clones': 0, 'num_object_copies': 10040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12142, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 733}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'16023", 'reported_seq': 16299, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:32.323060+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:32.323060+0000', 'last_peered': '2024-09-16T10:30:32.323060+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:32.323060+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'6000", 'ondisk_log_start': "42'6000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20550598, 'num_objects': 5019, 'num_object_clones': 0, 'num_object_copies': 10038, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16021, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 682}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'15457", 'reported_seq': 15654, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:31.891071+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:31.891071+0000', 'last_peered': '2024-09-16T10:30:31.891071+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:31.891071+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'5400", 'ondisk_log_start': "42'5400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20325318, 'num_objects': 4964, 'num_object_clones': 0, 'num_object_copies': 9928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15455, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 731}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'15496", 'reported_seq': 20543, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:31.909163+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:31.909163+0000', 'last_peered': '2024-09-16T10:30:31.909163+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:31.909163+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'5400", 'ondisk_log_start': "42'5400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20808646, 'num_objects': 5082, 'num_object_clones': 0, 'num_object_copies': 10164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1716, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5082, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15494, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2480, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3366}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'15171", 'reported_seq': 15422, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:32.425490+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:32.425490+0000', 'last_peered': '2024-09-16T10:30:32.425490+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:32.425490+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'5100", 'ondisk_log_start': "42'5100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19919331, 'num_objects': 4864, 'num_object_clones': 0, 'num_object_copies': 9728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15170, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 789}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:36.984 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'15878", 'reported_seq': 16155, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:32.327823+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:32.327823+0000', 'last_peered': '2024-09-16T10:30:32.327823+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:32.327823+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'5800", 'ondisk_log_start': "42'5800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20628905, 'num_objects': 5039, 'num_object_clones': 0, 'num_object_copies': 10078, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15875, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 772}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:36.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:36.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'15300", 'reported_seq': 15532, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:32.325976+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:32.325976+0000', 'last_peered': '2024-09-16T10:30:32.325976+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:32.325976+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'5200", 'ondisk_log_start': "42'5200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19825606, 'num_objects': 4842, 'num_object_clones': 0, 'num_object_copies': 9684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15298, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:36.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:36.985 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'15933", 'reported_seq': 16174, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:32.390375+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:32.390375+0000', 'last_peered': '2024-09-16T10:30:32.390375+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:32.390375+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'5900", 'ondisk_log_start': "42'5900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20330380, 'num_objects': 4967, 'num_object_clones': 0, 'num_object_copies': 9934, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15930, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 645}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:36.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:37.699 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:37.699 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39785/158508 objects degraded (25.100%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":526,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":79254,"data_bytes":324547105,"bytes_used":1897050112,"bytes_avail":288013242368,"bytes_total":289910292480,"degraded_objects":39785,"degraded_total":158508,"degraded_ratio":0.25099679511444217,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":392},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:30:37.723 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:37.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:38.271 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:38.271 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:38.292 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-16T10:30:36.713922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":324547105,"num_objects":79254,"num_object_clones":0,"num_object_copies":158508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227055,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11562,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1852588,"kb_used_data":837324,"kb_used_omap":12,"kb_used_meta":1015219,"kb_avail":281262932,"statfs":{"total":289910292480,"available":288013242368,"internally_reserved":0,"allocated":857419776,"data_stored":856557975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1039584258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,0,3,9,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19296256,"num_objects":-4711,"num_object_clones":0,"num_object_copies":-9422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4711,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002798"},"pg_stats":[{"pgid":"2.f","version":"46'15511","reported_seq":23976,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.426522+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:32.426522+0000","last_peered":"2024-09-16T10:30:32.426522+0000","last_clean":"2024-09-16T10:30:32.426522+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:32.426522+0000","last_undegraded":"2024-09-16T10:30:32.426522+0000","last_fullsized":"2024-09-16T10:30:32.426522+0000","mapping_epoch":45,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20303872,"num_objects":4957,"num_object_clones":0,"num_object_copies":9914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15511,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15544","reported_seq":15705,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:31.941342+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:31.941342+0000","last_peered":"2024-09-16T10:30:31.941342+0000","last_clean":"2024-09-16T10:30:31.941342+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:31.941342+0000","last_undegraded":"2024-09-16T10:30:31.941342+0000","last_fullsized":"2024-09-16T10:30:31.941342+0000","mapping_epoch":34,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20144128,"num_objects":4918,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11757,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15698","reported_seq":15952,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.915469+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:31.915469+0000","last_peered":"2024-09-16T10:30:31.915469+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:31.915469+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20336664,"num_objects":4966,"num_object_clones":0,"num_object_copies":9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11755,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15547","reported_seq":15607,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:31.736627+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:31.736627+0000","last_peered":"2024-09-16T10:30:31.736627+0000","last_clean":"2024-09-16T10:30:31.736627+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:31.736627+0000","last_undegraded":"2024-09-16T10:30:31.736627+0000","last_fullsized":"2024-09-16T10:30:31.736627+0000","mapping_epoch":34,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20131840,"num_objects":4915,"num_object_clones":0,"num_object_copies":9830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15547,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15542","reported_seq":15702,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.439883+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:32.439883+0000","last_peered":"2024-09-16T10:30:32.439883+0000","last_clean":"2024-09-16T10:30:32.439883+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:32.439883+0000","last_undegraded":"2024-09-16T10:30:32.439883+0000","last_fullsized":"2024-09-16T10:30:32.439883+0000","mapping_epoch":33,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20328448,"num_objects":4963,"num_object_clones":0,"num_object_copies":9926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12184,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15447","reported_seq":15780,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.419421+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:32.419421+0000","last_peered":"2024-09-16T10:30:32.419421+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:32.419421+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20168704,"num_objects":4924,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'15900","reported_seq":16309,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.365871+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:32.365871+0000","last_peered":"2024-09-16T10:30:32.365871+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:32.365871+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20561920,"num_objects":5020,"num_object_clones":0,"num_object_copies":10040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12142,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":733}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15470","reported_seq":15649,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:32.408669+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:32.408669+0000","last_peered":"2024-09-16T10:30:32.408669+0000","last_clean":"2024-09-16T10:30:32.408669+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:32.408669+0000","last_undegraded":"2024-09-16T10:30:32.408669+0000","last_fullsized":"2024-09-16T10:30:32.408669+0000","mapping_epoch":33,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20152320,"num_objects":4920,"num_object_clones":0,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11273,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15571","reported_seq":24077,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:36.333706+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:36.333706+0000","last_peered":"2024-09-16T10:30:36.333706+0000","last_clean":"2024-09-16T10:30:36.333706+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:36.333706+0000","last_undegraded":"2024-09-16T10:30:36.333706+0000","last_fullsized":"2024-09-16T10:30:36.333706+0000","mapping_epoch":43,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20030406,"num_objects":4892,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15569,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16023","reported_seq":16299,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.323060+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:32.323060+0000","last_peered":"2024-09-16T10:30:32.323060+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:32.323060+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20550598,"num_objects":5019,"num_object_clones":0,"num_object_copies":10038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16021,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":682}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15457","reported_seq":15654,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.891071+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:31.891071+0000","last_peered":"2024-09-16T10:30:31.891071+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:31.891071+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20325318,"num_objects":4964,"num_object_clones":0,"num_object_copies":9928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15455,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":731}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15496","reported_seq":20543,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:31.909163+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:31.909163+0000","last_peered":"2024-09-16T10:30:31.909163+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:31.909163+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20808646,"num_objects":5082,"num_object_clones":0,"num_object_copies":10164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15494,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3366}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15171","reported_seq":15422,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.425490+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:32.425490+0000","last_peered":"2024-09-16T10:30:32.425490+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:32.425490+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5100","ondisk_log_start":"42'5100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19919331,"num_objects":4864,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15170,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":789}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'15878","reported_seq":16155,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.327823+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:32.327823+0000","last_peered":"2024-09-16T10:30:32.327823+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:32.327823+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20628905,"num_objects":5039,"num_object_clones":0,"num_object_copies":10078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15875,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15300","reported_seq":15532,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.325976+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:32.325976+0000","last_peered":"2024-09-16T10:30:32.325976+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:32.325976+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5200","ondisk_log_start":"42'5200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19825606,"num_objects":4842,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15298,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'15933","reported_seq":16174,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:32.390375+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:32.390375+0000","last_peered":"2024-09-16T10:30:32.390375+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:32.390375+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20330380,"num_objects":4967,"num_object_clones":0,"num_object_copies":9934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15930,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":645}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":324547086,"num_objects":79252,"num_object_clones":0,"num_object_copies":158504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79252,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227053,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11561,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":854360064,"data_stored":854276047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160988,"ondisk_log_size":160988,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018977,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607988,"kb_used_data":297444,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93763852,"statfs":{"total":96636764160,"available":96014184448,"internally_reserved":0,"allocated":304582656,"data_stored":304314505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.6459999999999999},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.252},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":6.8280000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691922,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550048,"kb_used_data":233752,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93821792,"statfs":{"total":96636764160,"available":96073515008,"internally_reserved":0,"allocated":239362048,"data_stored":239028863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":323874474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.974}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.0619999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.27700000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215204,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694552,"kb_used_data":306128,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93677288,"statfs":{"total":96636764160,"available":95925542912,"internally_reserved":0,"allocated":313475072,"data_stored":313214607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":397733544},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":1.849},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":1.8080000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":2.0910000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":312532992,"data_stored":312529379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238444544,"data_stored":238371825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303382528,"data_stored":303374843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:41.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:41.851 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:41.852 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:41.873 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-16T10:30:40.714836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":315785761,"num_objects":77115,"num_object_clones":0,"num_object_copies":154230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229194,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11689,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161030,"ondisk_log_size":161030,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1832284,"kb_used_data":825220,"kb_used_omap":12,"kb_used_meta":1007027,"kb_avail":281283236,"statfs":{"total":289910292480,"available":288034033664,"internally_reserved":0,"allocated":845025280,"data_stored":844163479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1031195650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,4,18,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18718720,"num_objects":-4570,"num_object_clones":0,"num_object_copies":-9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4570,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002789"},"pg_stats":[{"pgid":"2.f","version":"46'15657","reported_seq":24120,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:37.506589+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:37.506589+0000","last_peered":"2024-09-16T10:30:37.506589+0000","last_clean":"2024-09-16T10:30:37.506589+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:37.506589+0000","last_undegraded":"2024-09-16T10:30:37.506589+0000","last_fullsized":"2024-09-16T10:30:37.506589+0000","mapping_epoch":45,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19705856,"num_objects":4811,"num_object_clones":0,"num_object_copies":9622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15657,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15687","reported_seq":15848,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:36.913288+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:36.913288+0000","last_peered":"2024-09-16T10:30:36.913288+0000","last_clean":"2024-09-16T10:30:36.913288+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:36.913288+0000","last_undegraded":"2024-09-16T10:30:36.913288+0000","last_fullsized":"2024-09-16T10:30:36.913288+0000","mapping_epoch":34,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19558400,"num_objects":4775,"num_object_clones":0,"num_object_copies":9550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11900,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15835","reported_seq":16089,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:36.917820+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:36.917820+0000","last_peered":"2024-09-16T10:30:36.917820+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:36.917820+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19775512,"num_objects":4829,"num_object_clones":0,"num_object_copies":9658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11892,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":526}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15673","reported_seq":15731,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:36.947595+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:36.947595+0000","last_peered":"2024-09-16T10:30:36.947595+0000","last_clean":"2024-09-16T10:30:36.947595+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:36.947595+0000","last_undegraded":"2024-09-16T10:30:36.947595+0000","last_fullsized":"2024-09-16T10:30:36.947595+0000","mapping_epoch":34,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19615744,"num_objects":4789,"num_object_clones":0,"num_object_copies":9578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15673,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15692","reported_seq":15852,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:37.434784+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:37.434784+0000","last_peered":"2024-09-16T10:30:37.434784+0000","last_clean":"2024-09-16T10:30:37.434784+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:37.434784+0000","last_undegraded":"2024-09-16T10:30:37.434784+0000","last_fullsized":"2024-09-16T10:30:37.434784+0000","mapping_epoch":33,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19714048,"num_objects":4813,"num_object_clones":0,"num_object_copies":9626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12334,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15583","reported_seq":15916,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.412627+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:37.412627+0000","last_peered":"2024-09-16T10:30:37.412627+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:37.412627+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19611648,"num_objects":4788,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12208,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16068","reported_seq":16477,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.402307+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:37.402307+0000","last_peered":"2024-09-16T10:30:37.402307+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:37.402307+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19873792,"num_objects":4852,"num_object_clones":0,"num_object_copies":9704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12310,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":565}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15597","reported_seq":15776,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:37.407008+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:37.407008+0000","last_peered":"2024-09-16T10:30:37.407008+0000","last_clean":"2024-09-16T10:30:37.407008+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:37.407008+0000","last_undegraded":"2024-09-16T10:30:37.407008+0000","last_fullsized":"2024-09-16T10:30:37.407008+0000","mapping_epoch":33,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19632128,"num_objects":4793,"num_object_clones":0,"num_object_copies":9586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11400,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15571","reported_seq":24077,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:36.333706+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:36.333706+0000","last_peered":"2024-09-16T10:30:36.333706+0000","last_clean":"2024-09-16T10:30:36.333706+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:36.333706+0000","last_undegraded":"2024-09-16T10:30:36.333706+0000","last_fullsized":"2024-09-16T10:30:36.333706+0000","mapping_epoch":43,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20030406,"num_objects":4892,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15569,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16167","reported_seq":16441,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.458524+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:37.458524+0000","last_peered":"2024-09-16T10:30:37.458524+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:37.458524+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19960774,"num_objects":4875,"num_object_clones":0,"num_object_copies":9750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16165,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":538}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15598","reported_seq":15795,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:36.885839+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:36.885839+0000","last_peered":"2024-09-16T10:30:36.885839+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:36.885839+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19747782,"num_objects":4823,"num_object_clones":0,"num_object_copies":9646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15596,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15675","reported_seq":20976,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:36.949647+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:36.949647+0000","last_peered":"2024-09-16T10:30:36.949647+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:36.949647+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20075462,"num_objects":4903,"num_object_clones":0,"num_object_copies":9806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15673,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2607,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3314}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15307","reported_seq":15558,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.389483+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:37.389483+0000","last_peered":"2024-09-16T10:30:37.389483+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:37.389483+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19362275,"num_objects":4728,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15306,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16007","reported_seq":16284,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.460427+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:37.460427+0000","last_peered":"2024-09-16T10:30:37.460427+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:37.460427+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20100521,"num_objects":4910,"num_object_clones":0,"num_object_copies":9820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16004,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15428","reported_seq":15660,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.451972+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:37.451972+0000","last_peered":"2024-09-16T10:30:37.451972+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:37.451972+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19301318,"num_objects":4714,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15426,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16082","reported_seq":16324,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.331101+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:37.331101+0000","last_peered":"2024-09-16T10:30:37.331101+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:37.331101+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19720076,"num_objects":4818,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16079,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":315785742,"num_objects":77113,"num_object_clones":0,"num_object_copies":154226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229192,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11688,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":842723328,"data_stored":842639311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161027,"ondisk_log_size":161027,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018978,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590304,"kb_used_data":292056,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93781536,"statfs":{"total":96636764160,"available":96032292864,"internally_reserved":0,"allocated":299065344,"data_stored":298797193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.9450000000000003},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.318}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.2060000000000004},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.36199999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691922,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550048,"kb_used_data":233752,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93821792,"statfs":{"total":96636764160,"available":96073515008,"internally_reserved":0,"allocated":239362048,"data_stored":239028863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":323874474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.974}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.0619999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.27700000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215205,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691932,"kb_used_data":299412,"kb_used_omap":4,"kb_used_meta":392507,"kb_avail":93679908,"statfs":{"total":96636764160,"available":95928225792,"internally_reserved":0,"allocated":306597888,"data_stored":306337423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":401927848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,8,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.18099999999999999},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":5.226}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":2.0230000000000001},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305623040,"data_stored":305619427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238444544,"data_stored":238371825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298655744,"data_stored":298648059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:41.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:42.418 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:42.418 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:42.449 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-16T10:30:40.714836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":315785761,"num_objects":77115,"num_object_clones":0,"num_object_copies":154230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77115,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229194,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11689,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161030,"ondisk_log_size":161030,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1832284,"kb_used_data":825220,"kb_used_omap":12,"kb_used_meta":1007027,"kb_avail":281283236,"statfs":{"total":289910292480,"available":288034033664,"internally_reserved":0,"allocated":845025280,"data_stored":844163479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1031195650},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,4,18,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18718720,"num_objects":-4570,"num_object_clones":0,"num_object_copies":-9140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4570,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002789"},"pg_stats":[{"pgid":"2.f","version":"46'15657","reported_seq":24120,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:37.506589+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:37.506589+0000","last_peered":"2024-09-16T10:30:37.506589+0000","last_clean":"2024-09-16T10:30:37.506589+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:37.506589+0000","last_undegraded":"2024-09-16T10:30:37.506589+0000","last_fullsized":"2024-09-16T10:30:37.506589+0000","mapping_epoch":45,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19705856,"num_objects":4811,"num_object_clones":0,"num_object_copies":9622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15657,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15687","reported_seq":15848,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:36.913288+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:36.913288+0000","last_peered":"2024-09-16T10:30:36.913288+0000","last_clean":"2024-09-16T10:30:36.913288+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:36.913288+0000","last_undegraded":"2024-09-16T10:30:36.913288+0000","last_fullsized":"2024-09-16T10:30:36.913288+0000","mapping_epoch":34,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19558400,"num_objects":4775,"num_object_clones":0,"num_object_copies":9550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11900,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15835","reported_seq":16089,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:36.917820+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:36.917820+0000","last_peered":"2024-09-16T10:30:36.917820+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:36.917820+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19775512,"num_objects":4829,"num_object_clones":0,"num_object_copies":9658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11892,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":526}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15673","reported_seq":15731,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:36.947595+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:36.947595+0000","last_peered":"2024-09-16T10:30:36.947595+0000","last_clean":"2024-09-16T10:30:36.947595+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:36.947595+0000","last_undegraded":"2024-09-16T10:30:36.947595+0000","last_fullsized":"2024-09-16T10:30:36.947595+0000","mapping_epoch":34,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19615744,"num_objects":4789,"num_object_clones":0,"num_object_copies":9578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15673,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15692","reported_seq":15852,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:37.434784+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:37.434784+0000","last_peered":"2024-09-16T10:30:37.434784+0000","last_clean":"2024-09-16T10:30:37.434784+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:37.434784+0000","last_undegraded":"2024-09-16T10:30:37.434784+0000","last_fullsized":"2024-09-16T10:30:37.434784+0000","mapping_epoch":33,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19714048,"num_objects":4813,"num_object_clones":0,"num_object_copies":9626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12334,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15583","reported_seq":15916,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.412627+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:37.412627+0000","last_peered":"2024-09-16T10:30:37.412627+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:37.412627+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19611648,"num_objects":4788,"num_object_clones":0,"num_object_copies":9576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12208,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16068","reported_seq":16477,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.402307+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:37.402307+0000","last_peered":"2024-09-16T10:30:37.402307+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:37.402307+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19873792,"num_objects":4852,"num_object_clones":0,"num_object_copies":9704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12310,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":565}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15597","reported_seq":15776,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:37.407008+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:37.407008+0000","last_peered":"2024-09-16T10:30:37.407008+0000","last_clean":"2024-09-16T10:30:37.407008+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:37.407008+0000","last_undegraded":"2024-09-16T10:30:37.407008+0000","last_fullsized":"2024-09-16T10:30:37.407008+0000","mapping_epoch":33,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19632128,"num_objects":4793,"num_object_clones":0,"num_object_copies":9586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11400,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15571","reported_seq":24077,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:36.333706+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:36.333706+0000","last_peered":"2024-09-16T10:30:36.333706+0000","last_clean":"2024-09-16T10:30:36.333706+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:36.333706+0000","last_undegraded":"2024-09-16T10:30:36.333706+0000","last_fullsized":"2024-09-16T10:30:36.333706+0000","mapping_epoch":43,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20030406,"num_objects":4892,"num_object_clones":0,"num_object_copies":9784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15569,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16167","reported_seq":16441,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.458524+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:37.458524+0000","last_peered":"2024-09-16T10:30:37.458524+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:37.458524+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19960774,"num_objects":4875,"num_object_clones":0,"num_object_copies":9750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16165,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":538}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15598","reported_seq":15795,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:36.885839+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:36.885839+0000","last_peered":"2024-09-16T10:30:36.885839+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:36.885839+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19747782,"num_objects":4823,"num_object_clones":0,"num_object_copies":9646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15596,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15675","reported_seq":20976,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:36.949647+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:36.949647+0000","last_peered":"2024-09-16T10:30:36.949647+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:36.949647+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5600","ondisk_log_start":"42'5600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20075462,"num_objects":4903,"num_object_clones":0,"num_object_copies":9806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15673,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2607,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3314}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15307","reported_seq":15558,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.389483+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:37.389483+0000","last_peered":"2024-09-16T10:30:37.389483+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:37.389483+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5300","ondisk_log_start":"42'5300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19362275,"num_objects":4728,"num_object_clones":0,"num_object_copies":9456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15306,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":653}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16007","reported_seq":16284,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.460427+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:37.460427+0000","last_peered":"2024-09-16T10:30:37.460427+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:37.460427+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20100521,"num_objects":4910,"num_object_clones":0,"num_object_copies":9820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16004,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":643}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15428","reported_seq":15660,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.451972+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:37.451972+0000","last_peered":"2024-09-16T10:30:37.451972+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:37.451972+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19301318,"num_objects":4714,"num_object_clones":0,"num_object_copies":9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15426,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":621}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16082","reported_seq":16324,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:37.331101+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:37.331101+0000","last_peered":"2024-09-16T10:30:37.331101+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:37.331101+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19720076,"num_objects":4818,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16079,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":315785742,"num_objects":77113,"num_object_clones":0,"num_object_copies":154226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39658,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":229192,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11688,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":842723328,"data_stored":842639311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161027,"ondisk_log_size":161027,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018978,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590304,"kb_used_data":292056,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93781536,"statfs":{"total":96636764160,"available":96032292864,"internally_reserved":0,"allocated":299065344,"data_stored":298797193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.9450000000000003},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.318}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.2060000000000004},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.36199999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691922,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550048,"kb_used_data":233752,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93821792,"statfs":{"total":96636764160,"available":96073515008,"internally_reserved":0,"allocated":239362048,"data_stored":239028863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":323874474},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.974}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.0619999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.27700000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215205,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691932,"kb_used_data":299412,"kb_used_omap":4,"kb_used_meta":392507,"kb_avail":93679908,"statfs":{"total":96636764160,"available":95928225792,"internally_reserved":0,"allocated":306597888,"data_stored":306337423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":401927848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,8,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.18099999999999999},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":5.226}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":2.0230000000000001},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":305623040,"data_stored":305619427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":238444544,"data_stored":238371825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298655744,"data_stored":298648059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:42.450 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:42.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'15835", 'reported_seq': 16089, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:36.917820+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:36.917820+0000', 'last_peered': '2024-09-16T10:30:36.917820+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:36.917820+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'5800", 'ondisk_log_start': "42'5800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19775512, 'num_objects': 4829, 'num_object_clones': 0, 'num_object_copies': 9658, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4829, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11892, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 526}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:42.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:42.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'15583", 'reported_seq': 15916, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:37.412627+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:37.412627+0000', 'last_peered': '2024-09-16T10:30:37.412627+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:37.412627+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'5500", 'ondisk_log_start': "42'5500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19611648, 'num_objects': 4788, 'num_object_clones': 0, 'num_object_copies': 9576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12208, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 654}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:42.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:42.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'16068", 'reported_seq': 16477, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:37.402307+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:37.402307+0000', 'last_peered': '2024-09-16T10:30:37.402307+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:37.402307+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'6000", 'ondisk_log_start': "42'6000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19873792, 'num_objects': 4852, 'num_object_clones': 0, 'num_object_copies': 9704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12310, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 565}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:42.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:42.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'16167", 'reported_seq': 16441, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:37.458524+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:37.458524+0000', 'last_peered': '2024-09-16T10:30:37.458524+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:37.458524+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'6100", 'ondisk_log_start': "42'6100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19960774, 'num_objects': 4875, 'num_object_clones': 0, 'num_object_copies': 9750, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16165, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 538}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:42.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:42.451 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'15598", 'reported_seq': 15795, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:36.885839+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:36.885839+0000', 'last_peered': '2024-09-16T10:30:36.885839+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:36.885839+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'5500", 'ondisk_log_start': "42'5500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19747782, 'num_objects': 4823, 'num_object_clones': 0, 'num_object_copies': 9646, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15596, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 590}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:42.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:42.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'15675", 'reported_seq': 20976, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:36.949647+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:36.949647+0000', 'last_peered': '2024-09-16T10:30:36.949647+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:36.949647+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'5600", 'ondisk_log_start': "42'5600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20075462, 'num_objects': 4903, 'num_object_clones': 0, 'num_object_copies': 9806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1589, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15673, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2607, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3314}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:42.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:42.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'15307", 'reported_seq': 15558, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:37.389483+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:37.389483+0000', 'last_peered': '2024-09-16T10:30:37.389483+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:37.389483+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'5300", 'ondisk_log_start': "42'5300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19362275, 'num_objects': 4728, 'num_object_clones': 0, 'num_object_copies': 9456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15306, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 653}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:42.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:42.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'16007", 'reported_seq': 16284, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:37.460427+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:37.460427+0000', 'last_peered': '2024-09-16T10:30:37.460427+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:37.460427+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'6000", 'ondisk_log_start': "42'6000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20100521, 'num_objects': 4910, 'num_object_clones': 0, 'num_object_copies': 9820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16004, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 643}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:42.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:42.452 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'15428", 'reported_seq': 15660, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:37.451972+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:37.451972+0000', 'last_peered': '2024-09-16T10:30:37.451972+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:37.451972+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'5400", 'ondisk_log_start': "42'5400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19301318, 'num_objects': 4714, 'num_object_clones': 0, 'num_object_copies': 9428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4714, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15426, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 621}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:42.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:42.453 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'16082", 'reported_seq': 16324, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:37.331101+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:37.331101+0000', 'last_peered': '2024-09-16T10:30:37.331101+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:37.331101+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'6000", 'ondisk_log_start': "42'6000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19720076, 'num_objects': 4818, 'num_object_clones': 0, 'num_object_copies': 9636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16079, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 496}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:42.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:43.150 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:43.150 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39518/149546 objects degraded (26.425%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":531,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":74773,"data_bytes":306192929,"bytes_used":1897762816,"bytes_avail":288012529664,"bytes_total":289910292480,"degraded_objects":39518,"degraded_total":149546,"degraded_ratio":0.26425313950222673,"recovering_objects_per_sec":33,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":400,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":575},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:30:43.173 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:43.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:43.728 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:43.730 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:43.757 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-16T10:30:42.715265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":306192929,"num_objects":74773,"num_object_clones":0,"num_object_copies":149546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74773,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231536,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11829,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160572,"ondisk_log_size":160572,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1853284,"kb_used_data":807672,"kb_used_omap":12,"kb_used_meta":1045555,"kb_avail":281262236,"statfs":{"total":289910292480,"available":288012529664,"internally_reserved":0,"allocated":827056128,"data_stored":826194327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1070648322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,9,3,6,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28311552,"num_objects":-6912,"num_object_clones":0,"num_object_copies":-13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-400,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6912,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002720"},"pg_stats":[{"pgid":"2.f","version":"46'15813","reported_seq":24278,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:42.648415+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:42.648415+0000","last_peered":"2024-09-16T10:30:42.648415+0000","last_clean":"2024-09-16T10:30:42.648415+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:42.648415+0000","last_undegraded":"2024-09-16T10:30:42.648415+0000","last_fullsized":"2024-09-16T10:30:42.648415+0000","mapping_epoch":45,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19066880,"num_objects":4655,"num_object_clones":0,"num_object_copies":9310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15813,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15821","reported_seq":15982,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:41.908117+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:41.908117+0000","last_peered":"2024-09-16T10:30:41.908117+0000","last_clean":"2024-09-16T10:30:41.908117+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:41.908117+0000","last_undegraded":"2024-09-16T10:30:41.908117+0000","last_fullsized":"2024-09-16T10:30:41.908117+0000","mapping_epoch":34,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19009536,"num_objects":4641,"num_object_clones":0,"num_object_copies":9282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12034,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15977","reported_seq":16231,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:41.940191+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:41.940191+0000","last_peered":"2024-09-16T10:30:41.940191+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:41.940191+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19193880,"num_objects":4687,"num_object_clones":0,"num_object_copies":9374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12034,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":384}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15829","reported_seq":15889,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:41.915046+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:41.915046+0000","last_peered":"2024-09-16T10:30:41.915046+0000","last_clean":"2024-09-16T10:30:41.915046+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:41.915046+0000","last_undegraded":"2024-09-16T10:30:41.915046+0000","last_fullsized":"2024-09-16T10:30:41.915046+0000","mapping_epoch":34,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18976768,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15829,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15856","reported_seq":16016,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:42.603207+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:42.603207+0000","last_peered":"2024-09-16T10:30:42.603207+0000","last_clean":"2024-09-16T10:30:42.603207+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:42.603207+0000","last_undegraded":"2024-09-16T10:30:42.603207+0000","last_fullsized":"2024-09-16T10:30:42.603207+0000","mapping_epoch":33,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19042304,"num_objects":4649,"num_object_clones":0,"num_object_copies":9298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12498,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15713","reported_seq":16046,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.596383+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:42.596383+0000","last_peered":"2024-09-16T10:30:42.596383+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:42.596383+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19079168,"num_objects":4658,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":524}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16229","reported_seq":16637,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.559343+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:42.559343+0000","last_peered":"2024-09-16T10:30:42.559343+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:42.559343+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19214336,"num_objects":4691,"num_object_clones":0,"num_object_copies":9382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12471,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15746","reported_seq":15924,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:42.638684+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:42.638684+0000","last_peered":"2024-09-16T10:30:42.638684+0000","last_clean":"2024-09-16T10:30:42.638684+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:42.638684+0000","last_undegraded":"2024-09-16T10:30:42.638684+0000","last_fullsized":"2024-09-16T10:30:42.638684+0000","mapping_epoch":33,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19021824,"num_objects":4644,"num_object_clones":0,"num_object_copies":9288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11549,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15711","reported_seq":24218,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:41.339221+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:41.339221+0000","last_peered":"2024-09-16T10:30:41.339221+0000","last_clean":"2024-09-16T10:30:41.339221+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:41.339221+0000","last_undegraded":"2024-09-16T10:30:41.339221+0000","last_fullsized":"2024-09-16T10:30:41.339221+0000","mapping_epoch":43,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19456966,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15709,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16309","reported_seq":16585,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.605739+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:42.605739+0000","last_peered":"2024-09-16T10:30:42.605739+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:42.605739+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379142,"num_objects":4733,"num_object_clones":0,"num_object_copies":9466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16307,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15749","reported_seq":15946,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:41.851904+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:41.851904+0000","last_peered":"2024-09-16T10:30:41.851904+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:41.851904+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19129286,"num_objects":4672,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15747,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":439}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15826","reported_seq":21407,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:41.942464+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:41.942464+0000","last_peered":"2024-09-16T10:30:41.942464+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:41.942464+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19456966,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15824,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3303}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15444","reported_seq":15695,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.589673+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:42.589673+0000","last_peered":"2024-09-16T10:30:42.589673+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:42.589673+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18801123,"num_objects":4591,"num_object_clones":0,"num_object_copies":9182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15443,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16136","reported_seq":16413,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.486795+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:42.486795+0000","last_peered":"2024-09-16T10:30:42.486795+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:42.486795+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19572137,"num_objects":4781,"num_object_clones":0,"num_object_copies":9562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16133,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":514}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15578","reported_seq":15808,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.590156+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:42.590156+0000","last_peered":"2024-09-16T10:30:42.590156+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:42.590156+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18686918,"num_objects":4564,"num_object_clones":0,"num_object_copies":9128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15576,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16232","reported_seq":16474,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.556802+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:42.556802+0000","last_peered":"2024-09-16T10:30:42.556802+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:42.556802+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19105676,"num_objects":4668,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16229,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":306192910,"num_objects":74771,"num_object_clones":0,"num_object_copies":149542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74771,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231534,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11828,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":823697408,"data_stored":823613391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160569,"ondisk_log_size":160569,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018979,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590612,"kb_used_data":288264,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93781228,"statfs":{"total":96636764160,"available":96031977472,"internally_reserved":0,"allocated":295182336,"data_stored":294914185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.895},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.63}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":1.8580000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691923,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569336,"kb_used_data":226796,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93802504,"statfs":{"total":96636764160,"available":96053764096,"internally_reserved":0,"allocated":232239104,"data_stored":231905919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":350744234},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":3.9990000000000001},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.9649999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.32100000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215206,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693336,"kb_used_data":292612,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93678504,"statfs":{"total":96636764160,"available":95926788096,"internally_reserved":0,"allocated":299634688,"data_stored":299374223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":410316456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,2,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.6589999999999998},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.24199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.6959999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298389504,"data_stored":298385891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":231366656,"data_stored":231293937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":293941248,"data_stored":293933563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:46.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:47.311 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:47.314 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:47.335 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-16T10:30:46.716225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305672737,"num_objects":74646,"num_object_clones":0,"num_object_copies":149292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74646,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231663,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11829,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160599,"ondisk_log_size":160599,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1850656,"kb_used_data":800948,"kb_used_omap":12,"kb_used_meta":1049651,"kb_avail":281264864,"statfs":{"total":289910292480,"available":288015220736,"internally_reserved":0,"allocated":820170752,"data_stored":819308951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1074842626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,9,5,7,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19456000,"num_objects":-4750,"num_object_clones":0,"num_object_copies":-9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4750,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002744"},"pg_stats":[{"pgid":"2.f","version":"46'15813","reported_seq":24278,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:42.648415+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:42.648415+0000","last_peered":"2024-09-16T10:30:42.648415+0000","last_clean":"2024-09-16T10:30:42.648415+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:42.648415+0000","last_undegraded":"2024-09-16T10:30:42.648415+0000","last_fullsized":"2024-09-16T10:30:42.648415+0000","mapping_epoch":45,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19066880,"num_objects":4655,"num_object_clones":0,"num_object_copies":9310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15813,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15821","reported_seq":15982,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:41.908117+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:41.908117+0000","last_peered":"2024-09-16T10:30:41.908117+0000","last_clean":"2024-09-16T10:30:41.908117+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:41.908117+0000","last_undegraded":"2024-09-16T10:30:41.908117+0000","last_fullsized":"2024-09-16T10:30:41.908117+0000","mapping_epoch":34,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19009536,"num_objects":4641,"num_object_clones":0,"num_object_copies":9282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12034,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15977","reported_seq":16231,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:41.940191+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:41.940191+0000","last_peered":"2024-09-16T10:30:41.940191+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:41.940191+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19193880,"num_objects":4687,"num_object_clones":0,"num_object_copies":9374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12034,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":384}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15829","reported_seq":15889,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:41.915046+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:41.915046+0000","last_peered":"2024-09-16T10:30:41.915046+0000","last_clean":"2024-09-16T10:30:41.915046+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:41.915046+0000","last_undegraded":"2024-09-16T10:30:41.915046+0000","last_fullsized":"2024-09-16T10:30:41.915046+0000","mapping_epoch":34,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18976768,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15829,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15856","reported_seq":16016,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:42.603207+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:42.603207+0000","last_peered":"2024-09-16T10:30:42.603207+0000","last_clean":"2024-09-16T10:30:42.603207+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:42.603207+0000","last_undegraded":"2024-09-16T10:30:42.603207+0000","last_fullsized":"2024-09-16T10:30:42.603207+0000","mapping_epoch":33,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19042304,"num_objects":4649,"num_object_clones":0,"num_object_copies":9298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12498,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15713","reported_seq":16046,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.596383+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:42.596383+0000","last_peered":"2024-09-16T10:30:42.596383+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:42.596383+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19079168,"num_objects":4658,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":524}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16229","reported_seq":16637,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.559343+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:42.559343+0000","last_peered":"2024-09-16T10:30:42.559343+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:42.559343+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19214336,"num_objects":4691,"num_object_clones":0,"num_object_copies":9382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12471,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15746","reported_seq":15924,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:42.638684+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:42.638684+0000","last_peered":"2024-09-16T10:30:42.638684+0000","last_clean":"2024-09-16T10:30:42.638684+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:42.638684+0000","last_undegraded":"2024-09-16T10:30:42.638684+0000","last_fullsized":"2024-09-16T10:30:42.638684+0000","mapping_epoch":33,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19021824,"num_objects":4644,"num_object_clones":0,"num_object_copies":9288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11549,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15838","reported_seq":24345,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:46.478419+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:46.478419+0000","last_peered":"2024-09-16T10:30:46.478419+0000","last_clean":"2024-09-16T10:30:46.478419+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:46.478419+0000","last_undegraded":"2024-09-16T10:30:46.478419+0000","last_fullsized":"2024-09-16T10:30:46.478419+0000","mapping_epoch":43,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18936774,"num_objects":4625,"num_object_clones":0,"num_object_copies":9250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15836,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16309","reported_seq":16585,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.605739+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:42.605739+0000","last_peered":"2024-09-16T10:30:42.605739+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:42.605739+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379142,"num_objects":4733,"num_object_clones":0,"num_object_copies":9466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16307,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15749","reported_seq":15946,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:41.851904+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:41.851904+0000","last_peered":"2024-09-16T10:30:41.851904+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:41.851904+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19129286,"num_objects":4672,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15747,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":439}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15826","reported_seq":21407,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:41.942464+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:41.942464+0000","last_peered":"2024-09-16T10:30:41.942464+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:41.942464+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19456966,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15824,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3303}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15444","reported_seq":15695,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.589673+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:42.589673+0000","last_peered":"2024-09-16T10:30:42.589673+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:42.589673+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18801123,"num_objects":4591,"num_object_clones":0,"num_object_copies":9182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15443,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16136","reported_seq":16413,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.486795+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:42.486795+0000","last_peered":"2024-09-16T10:30:42.486795+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:42.486795+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19572137,"num_objects":4781,"num_object_clones":0,"num_object_copies":9562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16133,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":514}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15578","reported_seq":15808,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.590156+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:42.590156+0000","last_peered":"2024-09-16T10:30:42.590156+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:42.590156+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18686918,"num_objects":4564,"num_object_clones":0,"num_object_copies":9128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15576,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16232","reported_seq":16474,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.556802+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:42.556802+0000","last_peered":"2024-09-16T10:30:42.556802+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:42.556802+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19105676,"num_objects":4668,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16229,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":305672718,"num_objects":74644,"num_object_clones":0,"num_object_copies":149288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74644,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231661,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11828,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":816857088,"data_stored":816773071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160596,"ondisk_log_size":160596,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018979,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590612,"kb_used_data":288264,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93781228,"statfs":{"total":96636764160,"available":96031977472,"internally_reserved":0,"allocated":295182336,"data_stored":294914185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.895},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.63}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":1.8580000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566708,"kb_used_data":220072,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93805132,"statfs":{"total":96636764160,"available":96056455168,"internally_reserved":0,"allocated":225353728,"data_stored":225020543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":354938538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.2730000000000001},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.2370000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.4889999999999999},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.524}]}]},{"osd":0,"up_from":24,"seq":103079215206,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693336,"kb_used_data":292612,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93678504,"statfs":{"total":96636764160,"available":95926788096,"internally_reserved":0,"allocated":299634688,"data_stored":299374223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":410316456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,2,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.6589999999999998},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.24199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.6959999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298389504,"data_stored":298385891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224526336,"data_stored":224453617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":293941248,"data_stored":293933563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:47.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:47.897 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:47.898 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:47.919 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-16T10:30:46.716225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":305672737,"num_objects":74646,"num_object_clones":0,"num_object_copies":149292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74646,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231663,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11829,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160599,"ondisk_log_size":160599,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1850656,"kb_used_data":800948,"kb_used_omap":12,"kb_used_meta":1049651,"kb_avail":281264864,"statfs":{"total":289910292480,"available":288015220736,"internally_reserved":0,"allocated":820170752,"data_stored":819308951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1074842626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,9,5,7,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19456000,"num_objects":-4750,"num_object_clones":0,"num_object_copies":-9500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-267,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4750,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002744"},"pg_stats":[{"pgid":"2.f","version":"46'15813","reported_seq":24278,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:42.648415+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:42.648415+0000","last_peered":"2024-09-16T10:30:42.648415+0000","last_clean":"2024-09-16T10:30:42.648415+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:42.648415+0000","last_undegraded":"2024-09-16T10:30:42.648415+0000","last_fullsized":"2024-09-16T10:30:42.648415+0000","mapping_epoch":45,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19066880,"num_objects":4655,"num_object_clones":0,"num_object_copies":9310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15813,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15821","reported_seq":15982,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:41.908117+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:41.908117+0000","last_peered":"2024-09-16T10:30:41.908117+0000","last_clean":"2024-09-16T10:30:41.908117+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:41.908117+0000","last_undegraded":"2024-09-16T10:30:41.908117+0000","last_fullsized":"2024-09-16T10:30:41.908117+0000","mapping_epoch":34,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19009536,"num_objects":4641,"num_object_clones":0,"num_object_copies":9282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12034,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'15977","reported_seq":16231,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:41.940191+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:41.940191+0000","last_peered":"2024-09-16T10:30:41.940191+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:41.940191+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19193880,"num_objects":4687,"num_object_clones":0,"num_object_copies":9374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4687,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12034,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":384}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15829","reported_seq":15889,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:41.915046+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:41.915046+0000","last_peered":"2024-09-16T10:30:41.915046+0000","last_clean":"2024-09-16T10:30:41.915046+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:41.915046+0000","last_undegraded":"2024-09-16T10:30:41.915046+0000","last_fullsized":"2024-09-16T10:30:41.915046+0000","mapping_epoch":34,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18976768,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15829,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'15856","reported_seq":16016,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:42.603207+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:42.603207+0000","last_peered":"2024-09-16T10:30:42.603207+0000","last_clean":"2024-09-16T10:30:42.603207+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:42.603207+0000","last_undegraded":"2024-09-16T10:30:42.603207+0000","last_fullsized":"2024-09-16T10:30:42.603207+0000","mapping_epoch":33,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19042304,"num_objects":4649,"num_object_clones":0,"num_object_copies":9298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12498,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15713","reported_seq":16046,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.596383+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:42.596383+0000","last_peered":"2024-09-16T10:30:42.596383+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:42.596383+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19079168,"num_objects":4658,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12338,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":524}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16229","reported_seq":16637,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.559343+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:42.559343+0000","last_peered":"2024-09-16T10:30:42.559343+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:42.559343+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19214336,"num_objects":4691,"num_object_clones":0,"num_object_copies":9382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12471,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15746","reported_seq":15924,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:42.638684+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:42.638684+0000","last_peered":"2024-09-16T10:30:42.638684+0000","last_clean":"2024-09-16T10:30:42.638684+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:42.638684+0000","last_undegraded":"2024-09-16T10:30:42.638684+0000","last_fullsized":"2024-09-16T10:30:42.638684+0000","mapping_epoch":33,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19021824,"num_objects":4644,"num_object_clones":0,"num_object_copies":9288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11549,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15838","reported_seq":24345,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:46.478419+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:46.478419+0000","last_peered":"2024-09-16T10:30:46.478419+0000","last_clean":"2024-09-16T10:30:46.478419+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:46.478419+0000","last_undegraded":"2024-09-16T10:30:46.478419+0000","last_fullsized":"2024-09-16T10:30:46.478419+0000","mapping_epoch":43,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18936774,"num_objects":4625,"num_object_clones":0,"num_object_copies":9250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15836,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16309","reported_seq":16585,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.605739+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:42.605739+0000","last_peered":"2024-09-16T10:30:42.605739+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:42.605739+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19379142,"num_objects":4733,"num_object_clones":0,"num_object_copies":9466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16307,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":396}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15749","reported_seq":15946,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:41.851904+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:41.851904+0000","last_peered":"2024-09-16T10:30:41.851904+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:41.851904+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19129286,"num_objects":4672,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15747,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":439}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15826","reported_seq":21407,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:41.942464+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:41.942464+0000","last_peered":"2024-09-16T10:30:41.942464+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:41.942464+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19456966,"num_objects":4752,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15824,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2747,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3303}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15444","reported_seq":15695,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.589673+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:42.589673+0000","last_peered":"2024-09-16T10:30:42.589673+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:42.589673+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5400","ondisk_log_start":"42'5400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18801123,"num_objects":4591,"num_object_clones":0,"num_object_copies":9182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15443,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16136","reported_seq":16413,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.486795+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:42.486795+0000","last_peered":"2024-09-16T10:30:42.486795+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:42.486795+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19572137,"num_objects":4781,"num_object_clones":0,"num_object_copies":9562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16133,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":514}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15578","reported_seq":15808,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.590156+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:42.590156+0000","last_peered":"2024-09-16T10:30:42.590156+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:42.590156+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18686918,"num_objects":4564,"num_object_clones":0,"num_object_copies":9128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15576,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":471}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16232","reported_seq":16474,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:42.556802+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:42.556802+0000","last_peered":"2024-09-16T10:30:42.556802+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:42.556802+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19105676,"num_objects":4668,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16229,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":346}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":305672718,"num_objects":74644,"num_object_clones":0,"num_object_copies":149288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74644,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":231661,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11828,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":816857088,"data_stored":816773071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160596,"ondisk_log_size":160596,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018979,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590612,"kb_used_data":288264,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93781228,"statfs":{"total":96636764160,"available":96031977472,"internally_reserved":0,"allocated":295182336,"data_stored":294914185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":1.895},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.63}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":1.8580000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566708,"kb_used_data":220072,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93805132,"statfs":{"total":96636764160,"available":96056455168,"internally_reserved":0,"allocated":225353728,"data_stored":225020543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":354938538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.2730000000000001},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.2370000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.4889999999999999},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.524}]}]},{"osd":0,"up_from":24,"seq":103079215206,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693336,"kb_used_data":292612,"kb_used_omap":4,"kb_used_meta":400699,"kb_avail":93678504,"statfs":{"total":96636764160,"available":95926788096,"internally_reserved":0,"allocated":299634688,"data_stored":299374223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":410316456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,2,2,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.6589999999999998},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.24199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.6959999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":298389504,"data_stored":298385891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224526336,"data_stored":224453617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":293941248,"data_stored":293933563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:47.920 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:47.920 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'15977", 'reported_seq': 16231, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:41.940191+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:41.940191+0000', 'last_peered': '2024-09-16T10:30:41.940191+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:41.940191+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'5900", 'ondisk_log_start': "42'5900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19193880, 'num_objects': 4687, 'num_object_clones': 0, 'num_object_copies': 9374, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4687, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12034, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 384}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:47.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:47.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'15713", 'reported_seq': 16046, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:42.596383+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:42.596383+0000', 'last_peered': '2024-09-16T10:30:42.596383+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:42.596383+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'5700", 'ondisk_log_start': "42'5700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19079168, 'num_objects': 4658, 'num_object_clones': 0, 'num_object_copies': 9316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12338, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 524}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:47.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:47.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'16229", 'reported_seq': 16637, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:42.559343+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:42.559343+0000', 'last_peered': '2024-09-16T10:30:42.559343+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:42.559343+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'6200", 'ondisk_log_start': "42'6200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19214336, 'num_objects': 4691, 'num_object_clones': 0, 'num_object_copies': 9382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12471, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:47.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:47.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'16309", 'reported_seq': 16585, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:42.605739+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:42.605739+0000', 'last_peered': '2024-09-16T10:30:42.605739+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:42.605739+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'6300", 'ondisk_log_start': "42'6300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19379142, 'num_objects': 4733, 'num_object_clones': 0, 'num_object_copies': 9466, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16307, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 396}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:47.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:47.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'15749", 'reported_seq': 15946, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:41.851904+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:41.851904+0000', 'last_peered': '2024-09-16T10:30:41.851904+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:41.851904+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'5700", 'ondisk_log_start': "42'5700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19129286, 'num_objects': 4672, 'num_object_clones': 0, 'num_object_copies': 9344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15747, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 439}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:47.921 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:47.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'15826", 'reported_seq': 21407, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:41.942464+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:41.942464+0000', 'last_peered': '2024-09-16T10:30:41.942464+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:41.942464+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'5800", 'ondisk_log_start': "42'5800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19456966, 'num_objects': 4752, 'num_object_clones': 0, 'num_object_copies': 9504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1449, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15824, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2747, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3303}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:47.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:47.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'15444", 'reported_seq': 15695, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:42.589673+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:42.589673+0000', 'last_peered': '2024-09-16T10:30:42.589673+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:42.589673+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'5400", 'ondisk_log_start': "42'5400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18801123, 'num_objects': 4591, 'num_object_clones': 0, 'num_object_copies': 9182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15443, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 516}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:47.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:47.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'16136", 'reported_seq': 16413, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:42.486795+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:42.486795+0000', 'last_peered': '2024-09-16T10:30:42.486795+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:42.486795+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'6100", 'ondisk_log_start': "42'6100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19572137, 'num_objects': 4781, 'num_object_clones': 0, 'num_object_copies': 9562, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16133, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 514}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:47.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:47.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'15578", 'reported_seq': 15808, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:42.590156+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:42.590156+0000', 'last_peered': '2024-09-16T10:30:42.590156+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:42.590156+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'5500", 'ondisk_log_start': "42'5500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18686918, 'num_objects': 4564, 'num_object_clones': 0, 'num_object_copies': 9128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15576, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:47.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:47.922 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'16232", 'reported_seq': 16474, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:42.556802+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:42.556802+0000', 'last_peered': '2024-09-16T10:30:42.556802+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:42.556802+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'6200", 'ondisk_log_start': "42'6200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19105676, 'num_objects': 4668, 'num_object_clones': 0, 'num_object_copies': 9336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4668, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16229, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 346}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:47.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:48.593 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:48.593 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39518/149292 objects degraded (26.470%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":537,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":74646,"data_bytes":305672737,"bytes_used":1895071744,"bytes_avail":288015220736,"bytes_total":289910292480,"degraded_objects":39518,"degraded_total":149292,"degraded_ratio":0.26470273021997159,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":267,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":395},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:30:48.623 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:48.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:49.164 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:49.167 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:49.188 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-16T10:30:48.716706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":297030177,"num_objects":72536,"num_object_clones":0,"num_object_copies":145072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72536,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233773,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11992,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160909,"ondisk_log_size":160909,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1848488,"kb_used_data":788936,"kb_used_omap":12,"kb_used_meta":1059507,"kb_avail":281267032,"statfs":{"total":289910292480,"available":288017440768,"internally_reserved":0,"allocated":807870464,"data_stored":807008663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1084935170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,4,5,11,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27516928,"num_objects":-6718,"num_object_clones":0,"num_object_copies":-13436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6718,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002784"},"pg_stats":[{"pgid":"2.f","version":"46'15953","reported_seq":24418,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.618866+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:47.618866+0000","last_peered":"2024-09-16T10:30:47.618866+0000","last_clean":"2024-09-16T10:30:47.618866+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:47.618866+0000","last_undegraded":"2024-09-16T10:30:47.618866+0000","last_fullsized":"2024-09-16T10:30:47.618866+0000","mapping_epoch":45,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18493440,"num_objects":4515,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15953,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'15958","reported_seq":16119,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:46.894874+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:46.894874+0000","last_peered":"2024-09-16T10:30:46.894874+0000","last_clean":"2024-09-16T10:30:46.894874+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:46.894874+0000","last_undegraded":"2024-09-16T10:30:46.894874+0000","last_fullsized":"2024-09-16T10:30:46.894874+0000","mapping_epoch":34,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18448384,"num_objects":4504,"num_object_clones":0,"num_object_copies":9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12171,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16113","reported_seq":16365,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:46.912191+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:46.912191+0000","last_peered":"2024-09-16T10:30:46.912191+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:46.912191+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18636824,"num_objects":4551,"num_object_clones":0,"num_object_copies":9102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4551,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12170,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":248}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'15965","reported_seq":16025,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:46.910609+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:46.910609+0000","last_peered":"2024-09-16T10:30:46.910609+0000","last_clean":"2024-09-16T10:30:46.910609+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:46.910609+0000","last_undegraded":"2024-09-16T10:30:46.910609+0000","last_fullsized":"2024-09-16T10:30:46.910609+0000","mapping_epoch":34,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18419712,"num_objects":4497,"num_object_clones":0,"num_object_copies":8994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15965,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16004","reported_seq":16164,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.637895+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:47.637895+0000","last_peered":"2024-09-16T10:30:47.637895+0000","last_clean":"2024-09-16T10:30:47.637895+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:47.637895+0000","last_undegraded":"2024-09-16T10:30:47.637895+0000","last_fullsized":"2024-09-16T10:30:47.637895+0000","mapping_epoch":33,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18436096,"num_objects":4501,"num_object_clones":0,"num_object_copies":9002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12646,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15856","reported_seq":16189,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.525475+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:47.525475+0000","last_peered":"2024-09-16T10:30:47.525475+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:47.525475+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18493440,"num_objects":4515,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16377","reported_seq":16786,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.692286+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:47.692286+0000","last_peered":"2024-09-16T10:30:47.692286+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:47.692286+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18608128,"num_objects":4543,"num_object_clones":0,"num_object_copies":9086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12619,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15895","reported_seq":16073,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.714269+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:47.714269+0000","last_peered":"2024-09-16T10:30:47.714269+0000","last_clean":"2024-09-16T10:30:47.714269+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:47.714269+0000","last_undegraded":"2024-09-16T10:30:47.714269+0000","last_fullsized":"2024-09-16T10:30:47.714269+0000","mapping_epoch":33,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18411520,"num_objects":4495,"num_object_clones":0,"num_object_copies":8990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11698,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'15838","reported_seq":24345,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:46.478419+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:46.478419+0000","last_peered":"2024-09-16T10:30:46.478419+0000","last_clean":"2024-09-16T10:30:46.478419+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:46.478419+0000","last_undegraded":"2024-09-16T10:30:46.478419+0000","last_fullsized":"2024-09-16T10:30:46.478419+0000","mapping_epoch":43,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18936774,"num_objects":4625,"num_object_clones":0,"num_object_copies":9250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15836,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16458","reported_seq":16734,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.611405+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:47.611405+0000","last_peered":"2024-09-16T10:30:47.611405+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:47.611405+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18768838,"num_objects":4584,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16456,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":247}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'15884","reported_seq":16081,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:46.896224+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:46.896224+0000","last_peered":"2024-09-16T10:30:46.896224+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:46.896224+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18576326,"num_objects":4537,"num_object_clones":0,"num_object_copies":9074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15882,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":304}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'15954","reported_seq":21861,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:46.990988+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:46.990988+0000","last_peered":"2024-09-16T10:30:46.990988+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:46.990988+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18932678,"num_objects":4624,"num_object_clones":0,"num_object_copies":9248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15952,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2910,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3338}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15585","reported_seq":15836,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.572233+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:47.572233+0000","last_peered":"2024-09-16T10:30:47.572233+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:47.572233+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18223587,"num_objects":4450,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15584,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16284","reported_seq":16561,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.706353+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:47.706353+0000","last_peered":"2024-09-16T10:30:47.706353+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:47.706353+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18965929,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16281,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15725","reported_seq":15954,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.663510+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:47.663510+0000","last_peered":"2024-09-16T10:30:47.663510+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:47.663510+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18084806,"num_objects":4417,"num_object_clones":0,"num_object_copies":8834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15723,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16357","reported_seq":16599,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.654350+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:47.654350+0000","last_peered":"2024-09-16T10:30:47.654350+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:47.654350+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18593676,"num_objects":4543,"num_object_clones":0,"num_object_copies":9086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16354,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":297030158,"num_objects":72534,"num_object_clones":0,"num_object_copies":145068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72534,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233771,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11991,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":805310464,"data_stored":805226447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018980,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589456,"kb_used_data":283012,"kb_used_omap":4,"kb_used_meta":306427,"kb_avail":93782384,"statfs":{"total":96636764160,"available":96033161216,"internally_reserved":0,"allocated":289804288,"data_stored":289536137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":313781936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.3570000000000002},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.29599999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.2269999999999999},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.238}]}]},{"osd":1,"up_from":40,"seq":171798691924,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566708,"kb_used_data":220072,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93805132,"statfs":{"total":96636764160,"available":96056455168,"internally_reserved":0,"allocated":225353728,"data_stored":225020543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":354938538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.2730000000000001},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":2.2370000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.4889999999999999},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.524}]}]},{"osd":0,"up_from":24,"seq":103079215207,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692324,"kb_used_data":285852,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93679516,"statfs":{"total":96636764160,"available":95927824384,"internally_reserved":0,"allocated":292712448,"data_stored":292451983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.0510000000000002},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":5.71}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.343},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.0860000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291385344,"data_stored":291381731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224526336,"data_stored":224453617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289398784,"data_stored":289391099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:52.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:52.739 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:52.739 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:52.763 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-16T10:30:52.717622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293466657,"num_objects":71666,"num_object_clones":0,"num_object_copies":143332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":234643,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12143,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160779,"ondisk_log_size":160779,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1832872,"kb_used_data":777412,"kb_used_omap":12,"kb_used_meta":1055411,"kb_avail":281282648,"statfs":{"total":289910292480,"available":288033431552,"internally_reserved":0,"allocated":796069888,"data_stored":795208087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1080740866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,1,5,3,12,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22319104,"num_objects":-5449,"num_object_clones":0,"num_object_copies":-10898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5449,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002785"},"pg_stats":[{"pgid":"2.f","version":"46'15953","reported_seq":24418,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.618866+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:47.618866+0000","last_peered":"2024-09-16T10:30:47.618866+0000","last_clean":"2024-09-16T10:30:47.618866+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:47.618866+0000","last_undegraded":"2024-09-16T10:30:47.618866+0000","last_fullsized":"2024-09-16T10:30:47.618866+0000","mapping_epoch":45,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18493440,"num_objects":4515,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15953,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16077","reported_seq":16238,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.886979+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:51.886979+0000","last_peered":"2024-09-16T10:30:51.886979+0000","last_clean":"2024-09-16T10:30:51.886979+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:51.886979+0000","last_undegraded":"2024-09-16T10:30:51.886979+0000","last_fullsized":"2024-09-16T10:30:51.886979+0000","mapping_epoch":34,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17960960,"num_objects":4385,"num_object_clones":0,"num_object_copies":8770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16265","reported_seq":16518,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.937361+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:51.937361+0000","last_peered":"2024-09-16T10:30:51.937361+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:51.937361+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18014232,"num_objects":4399,"num_object_clones":0,"num_object_copies":8798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12322,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16113","reported_seq":16172,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.971193+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:51.971193+0000","last_peered":"2024-09-16T10:30:51.971193+0000","last_clean":"2024-09-16T10:30:51.971193+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:51.971193+0000","last_undegraded":"2024-09-16T10:30:51.971193+0000","last_fullsized":"2024-09-16T10:30:51.971193+0000","mapping_epoch":34,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17813504,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16113,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16004","reported_seq":16164,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.637895+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:47.637895+0000","last_peered":"2024-09-16T10:30:47.637895+0000","last_clean":"2024-09-16T10:30:47.637895+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:47.637895+0000","last_undegraded":"2024-09-16T10:30:47.637895+0000","last_fullsized":"2024-09-16T10:30:47.637895+0000","mapping_epoch":33,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18436096,"num_objects":4501,"num_object_clones":0,"num_object_copies":9002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12646,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15856","reported_seq":16189,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.525475+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:47.525475+0000","last_peered":"2024-09-16T10:30:47.525475+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:47.525475+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18493440,"num_objects":4515,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16377","reported_seq":16786,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.692286+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:47.692286+0000","last_peered":"2024-09-16T10:30:47.692286+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:47.692286+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18608128,"num_objects":4543,"num_object_clones":0,"num_object_copies":9086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12619,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15895","reported_seq":16073,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.714269+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:47.714269+0000","last_peered":"2024-09-16T10:30:47.714269+0000","last_clean":"2024-09-16T10:30:47.714269+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:47.714269+0000","last_undegraded":"2024-09-16T10:30:47.714269+0000","last_fullsized":"2024-09-16T10:30:47.714269+0000","mapping_epoch":33,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18411520,"num_objects":4495,"num_object_clones":0,"num_object_copies":8990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11698,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16002","reported_seq":24509,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.562296+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:51.562296+0000","last_peered":"2024-09-16T10:30:51.562296+0000","last_clean":"2024-09-16T10:30:51.562296+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:51.562296+0000","last_undegraded":"2024-09-16T10:30:51.562296+0000","last_fullsized":"2024-09-16T10:30:51.562296+0000","mapping_epoch":43,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18265030,"num_objects":4461,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16000,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16458","reported_seq":16734,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.611405+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:47.611405+0000","last_peered":"2024-09-16T10:30:47.611405+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:47.611405+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18768838,"num_objects":4584,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16456,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":247}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16024","reported_seq":16221,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.976095+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:51.976095+0000","last_peered":"2024-09-16T10:30:51.976095+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:51.976095+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18002886,"num_objects":4397,"num_object_clones":0,"num_object_copies":8794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16022,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16101","reported_seq":22310,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.975609+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:51.975609+0000","last_peered":"2024-09-16T10:30:51.975609+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:51.975609+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330566,"num_objects":4477,"num_object_clones":0,"num_object_copies":8954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16099,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3061,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15585","reported_seq":15836,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.572233+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:47.572233+0000","last_peered":"2024-09-16T10:30:47.572233+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:47.572233+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18223587,"num_objects":4450,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15584,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16284","reported_seq":16561,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.706353+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:47.706353+0000","last_peered":"2024-09-16T10:30:47.706353+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:47.706353+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18965929,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16281,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15725","reported_seq":15954,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.663510+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:47.663510+0000","last_peered":"2024-09-16T10:30:47.663510+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:47.663510+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18084806,"num_objects":4417,"num_object_clones":0,"num_object_copies":8834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15723,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16357","reported_seq":16599,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.654350+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:47.654350+0000","last_peered":"2024-09-16T10:30:47.654350+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:47.654350+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18593676,"num_objects":4543,"num_object_clones":0,"num_object_copies":9086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16354,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":293466638,"num_objects":71664,"num_object_clones":0,"num_object_copies":143328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":234641,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12142,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":793497600,"data_stored":793413583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160776,"ondisk_log_size":160776,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018981,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589044,"kb_used_data":278504,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93782796,"statfs":{"total":96636764160,"available":96033583104,"internally_reserved":0,"allocated":285188096,"data_stored":284919945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.306},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":13.743}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.5490000000000004},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.04}]}]},{"osd":1,"up_from":40,"seq":171798691925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551504,"kb_used_data":213056,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93820336,"statfs":{"total":96636764160,"available":96072024064,"internally_reserved":0,"allocated":218169344,"data_stored":217836159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":346549930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.774},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.5270000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.601},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.6399999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215207,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692324,"kb_used_data":285852,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93679516,"statfs":{"total":96636764160,"available":95927824384,"internally_reserved":0,"allocated":292712448,"data_stored":292451983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.0510000000000002},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":5.71}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.343},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.0860000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291385344,"data_stored":291381731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217358336,"data_stored":217285617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284753920,"data_stored":284746235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:52.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:53.326 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:53.328 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:53.350 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-16T10:30:52.717622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293466657,"num_objects":71666,"num_object_clones":0,"num_object_copies":143332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":234643,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12143,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160779,"ondisk_log_size":160779,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1832872,"kb_used_data":777412,"kb_used_omap":12,"kb_used_meta":1055411,"kb_avail":281282648,"statfs":{"total":289910292480,"available":288033431552,"internally_reserved":0,"allocated":796069888,"data_stored":795208087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1080740866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,1,5,3,12,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22319104,"num_objects":-5449,"num_object_clones":0,"num_object_copies":-10898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5449,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002785"},"pg_stats":[{"pgid":"2.f","version":"46'15953","reported_seq":24418,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.618866+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:47.618866+0000","last_peered":"2024-09-16T10:30:47.618866+0000","last_clean":"2024-09-16T10:30:47.618866+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:47.618866+0000","last_undegraded":"2024-09-16T10:30:47.618866+0000","last_fullsized":"2024-09-16T10:30:47.618866+0000","mapping_epoch":45,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18493440,"num_objects":4515,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15953,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16077","reported_seq":16238,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.886979+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:51.886979+0000","last_peered":"2024-09-16T10:30:51.886979+0000","last_clean":"2024-09-16T10:30:51.886979+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:51.886979+0000","last_undegraded":"2024-09-16T10:30:51.886979+0000","last_fullsized":"2024-09-16T10:30:51.886979+0000","mapping_epoch":34,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17960960,"num_objects":4385,"num_object_clones":0,"num_object_copies":8770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16265","reported_seq":16518,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.937361+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:51.937361+0000","last_peered":"2024-09-16T10:30:51.937361+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:51.937361+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18014232,"num_objects":4399,"num_object_clones":0,"num_object_copies":8798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12322,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16113","reported_seq":16172,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.971193+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:51.971193+0000","last_peered":"2024-09-16T10:30:51.971193+0000","last_clean":"2024-09-16T10:30:51.971193+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:51.971193+0000","last_undegraded":"2024-09-16T10:30:51.971193+0000","last_fullsized":"2024-09-16T10:30:51.971193+0000","mapping_epoch":34,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17813504,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16113,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16004","reported_seq":16164,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.637895+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:47.637895+0000","last_peered":"2024-09-16T10:30:47.637895+0000","last_clean":"2024-09-16T10:30:47.637895+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:47.637895+0000","last_undegraded":"2024-09-16T10:30:47.637895+0000","last_fullsized":"2024-09-16T10:30:47.637895+0000","mapping_epoch":33,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18436096,"num_objects":4501,"num_object_clones":0,"num_object_copies":9002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12646,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15856","reported_seq":16189,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.525475+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:47.525475+0000","last_peered":"2024-09-16T10:30:47.525475+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:47.525475+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18493440,"num_objects":4515,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16377","reported_seq":16786,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.692286+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:47.692286+0000","last_peered":"2024-09-16T10:30:47.692286+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:47.692286+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18608128,"num_objects":4543,"num_object_clones":0,"num_object_copies":9086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12619,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15895","reported_seq":16073,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.714269+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:47.714269+0000","last_peered":"2024-09-16T10:30:47.714269+0000","last_clean":"2024-09-16T10:30:47.714269+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:47.714269+0000","last_undegraded":"2024-09-16T10:30:47.714269+0000","last_fullsized":"2024-09-16T10:30:47.714269+0000","mapping_epoch":33,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18411520,"num_objects":4495,"num_object_clones":0,"num_object_copies":8990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11698,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16002","reported_seq":24509,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.562296+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:51.562296+0000","last_peered":"2024-09-16T10:30:51.562296+0000","last_clean":"2024-09-16T10:30:51.562296+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:51.562296+0000","last_undegraded":"2024-09-16T10:30:51.562296+0000","last_fullsized":"2024-09-16T10:30:51.562296+0000","mapping_epoch":43,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18265030,"num_objects":4461,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16000,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16458","reported_seq":16734,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.611405+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:47.611405+0000","last_peered":"2024-09-16T10:30:47.611405+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:47.611405+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18768838,"num_objects":4584,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16456,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":247}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16024","reported_seq":16221,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.976095+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:51.976095+0000","last_peered":"2024-09-16T10:30:51.976095+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:51.976095+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18002886,"num_objects":4397,"num_object_clones":0,"num_object_copies":8794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16022,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16101","reported_seq":22310,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.975609+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:51.975609+0000","last_peered":"2024-09-16T10:30:51.975609+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:51.975609+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330566,"num_objects":4477,"num_object_clones":0,"num_object_copies":8954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16099,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3061,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15585","reported_seq":15836,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.572233+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:47.572233+0000","last_peered":"2024-09-16T10:30:47.572233+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:47.572233+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18223587,"num_objects":4450,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15584,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16284","reported_seq":16561,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.706353+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:47.706353+0000","last_peered":"2024-09-16T10:30:47.706353+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:47.706353+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18965929,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16281,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15725","reported_seq":15954,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.663510+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:47.663510+0000","last_peered":"2024-09-16T10:30:47.663510+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:47.663510+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18084806,"num_objects":4417,"num_object_clones":0,"num_object_copies":8834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15723,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16357","reported_seq":16599,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.654350+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:47.654350+0000","last_peered":"2024-09-16T10:30:47.654350+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:47.654350+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18593676,"num_objects":4543,"num_object_clones":0,"num_object_copies":9086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16354,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":293466638,"num_objects":71664,"num_object_clones":0,"num_object_copies":143328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":234641,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12142,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":793497600,"data_stored":793413583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160776,"ondisk_log_size":160776,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018981,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589044,"kb_used_data":278504,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93782796,"statfs":{"total":96636764160,"available":96033583104,"internally_reserved":0,"allocated":285188096,"data_stored":284919945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.306},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":13.743}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.5490000000000004},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.04}]}]},{"osd":1,"up_from":40,"seq":171798691925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551504,"kb_used_data":213056,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93820336,"statfs":{"total":96636764160,"available":96072024064,"internally_reserved":0,"allocated":218169344,"data_stored":217836159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":346549930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.774},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.5270000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.601},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.6399999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215207,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692324,"kb_used_data":285852,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93679516,"statfs":{"total":96636764160,"available":95927824384,"internally_reserved":0,"allocated":292712448,"data_stored":292451983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.0510000000000002},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":5.71}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.343},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.0860000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291385344,"data_stored":291381731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217358336,"data_stored":217285617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284753920,"data_stored":284746235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:53.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:53.351 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'16265", 'reported_seq': 16518, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:51.937361+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:51.937361+0000', 'last_peered': '2024-09-16T10:30:51.937361+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:51.937361+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'6200", 'ondisk_log_start': "42'6200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18014232, 'num_objects': 4399, 'num_object_clones': 0, 'num_object_copies': 8798, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4399, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12322, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 96}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:53.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:53.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'15856", 'reported_seq': 16189, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:47.525475+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:47.525475+0000', 'last_peered': '2024-09-16T10:30:47.525475+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:47.525475+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'5800", 'ondisk_log_start': "42'5800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18493440, 'num_objects': 4515, 'num_object_clones': 0, 'num_object_copies': 9030, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12481, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 381}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:53.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:53.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'16377", 'reported_seq': 16786, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:47.692286+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:47.692286+0000', 'last_peered': '2024-09-16T10:30:47.692286+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:47.692286+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'6300", 'ondisk_log_start': "42'6300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18608128, 'num_objects': 4543, 'num_object_clones': 0, 'num_object_copies': 9086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12619, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:53.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:53.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'16458", 'reported_seq': 16734, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:47.611405+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:47.611405+0000', 'last_peered': '2024-09-16T10:30:47.611405+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:47.611405+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'6400", 'ondisk_log_start': "42'6400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18768838, 'num_objects': 4584, 'num_object_clones': 0, 'num_object_copies': 9168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4584, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16456, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 247}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:53.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:53.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'16024", 'reported_seq': 16221, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:51.976095+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:51.976095+0000', 'last_peered': '2024-09-16T10:30:51.976095+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:51.976095+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'6000", 'ondisk_log_start': "42'6000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18002886, 'num_objects': 4397, 'num_object_clones': 0, 'num_object_copies': 8794, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4397, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16022, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 164}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:53.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:53.352 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'16101", 'reported_seq': 22310, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:51.975609+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:51.975609+0000', 'last_peered': '2024-09-16T10:30:51.975609+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:51.975609+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'6100", 'ondisk_log_start': "42'6100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18330566, 'num_objects': 4477, 'num_object_clones': 0, 'num_object_copies': 8954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1135, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4477, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16099, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3061, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3342}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:53.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:53.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'15585", 'reported_seq': 15836, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:47.572233+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:47.572233+0000', 'last_peered': '2024-09-16T10:30:47.572233+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:47.572233+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'5500", 'ondisk_log_start': "42'5500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18223587, 'num_objects': 4450, 'num_object_clones': 0, 'num_object_copies': 8900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15584, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 375}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:53.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:53.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'16284", 'reported_seq': 16561, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:47.706353+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:47.706353+0000', 'last_peered': '2024-09-16T10:30:47.706353+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:47.706353+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'6200", 'ondisk_log_start': "42'6200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18965929, 'num_objects': 4633, 'num_object_clones': 0, 'num_object_copies': 9266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16281, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 366}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:53.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:53.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'15725", 'reported_seq': 15954, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:47.663510+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:47.663510+0000', 'last_peered': '2024-09-16T10:30:47.663510+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:47.663510+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'5700", 'ondisk_log_start': "42'5700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18084806, 'num_objects': 4417, 'num_object_clones': 0, 'num_object_copies': 8834, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15723, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 324}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:53.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:53.353 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'16357", 'reported_seq': 16599, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:47.654350+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:47.654350+0000', 'last_peered': '2024-09-16T10:30:47.654350+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:47.654350+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'6300", 'ondisk_log_start': "42'6300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18593676, 'num_objects': 4543, 'num_object_clones': 0, 'num_object_copies': 9086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4543, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16354, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 221}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:53.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:54.071 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:54.071 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39204/143332 objects degraded (27.352%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":542,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":71666,"data_bytes":293466657,"bytes_used":1876860928,"bytes_avail":288033431552,"bytes_total":289910292480,"degraded_objects":39204,"degraded_total":143332,"degraded_ratio":0.27351882343091566,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":454,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":453},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:30:54.096 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:54.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:54.641 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:54.642 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:54.669 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-16T10:30:52.717622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293466657,"num_objects":71666,"num_object_clones":0,"num_object_copies":143332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71666,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":234643,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12143,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160779,"ondisk_log_size":160779,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1832872,"kb_used_data":777412,"kb_used_omap":12,"kb_used_meta":1055411,"kb_avail":281282648,"statfs":{"total":289910292480,"available":288033431552,"internally_reserved":0,"allocated":796069888,"data_stored":795208087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1080740866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,1,5,3,12,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22319104,"num_objects":-5449,"num_object_clones":0,"num_object_copies":-10898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5449,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":454,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002785"},"pg_stats":[{"pgid":"2.f","version":"46'15953","reported_seq":24418,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.618866+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:47.618866+0000","last_peered":"2024-09-16T10:30:47.618866+0000","last_clean":"2024-09-16T10:30:47.618866+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:47.618866+0000","last_undegraded":"2024-09-16T10:30:47.618866+0000","last_fullsized":"2024-09-16T10:30:47.618866+0000","mapping_epoch":45,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18493440,"num_objects":4515,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15953,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16077","reported_seq":16238,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.886979+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:51.886979+0000","last_peered":"2024-09-16T10:30:51.886979+0000","last_clean":"2024-09-16T10:30:51.886979+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:51.886979+0000","last_undegraded":"2024-09-16T10:30:51.886979+0000","last_fullsized":"2024-09-16T10:30:51.886979+0000","mapping_epoch":34,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17960960,"num_objects":4385,"num_object_clones":0,"num_object_copies":8770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16265","reported_seq":16518,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.937361+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:51.937361+0000","last_peered":"2024-09-16T10:30:51.937361+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:51.937361+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18014232,"num_objects":4399,"num_object_clones":0,"num_object_copies":8798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12322,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16113","reported_seq":16172,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.971193+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:51.971193+0000","last_peered":"2024-09-16T10:30:51.971193+0000","last_clean":"2024-09-16T10:30:51.971193+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:51.971193+0000","last_undegraded":"2024-09-16T10:30:51.971193+0000","last_fullsized":"2024-09-16T10:30:51.971193+0000","mapping_epoch":34,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17813504,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16113,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16004","reported_seq":16164,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.637895+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:47.637895+0000","last_peered":"2024-09-16T10:30:47.637895+0000","last_clean":"2024-09-16T10:30:47.637895+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:47.637895+0000","last_undegraded":"2024-09-16T10:30:47.637895+0000","last_fullsized":"2024-09-16T10:30:47.637895+0000","mapping_epoch":33,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18436096,"num_objects":4501,"num_object_clones":0,"num_object_copies":9002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12646,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15856","reported_seq":16189,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.525475+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:47.525475+0000","last_peered":"2024-09-16T10:30:47.525475+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:47.525475+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18493440,"num_objects":4515,"num_object_clones":0,"num_object_copies":9030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16377","reported_seq":16786,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.692286+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:47.692286+0000","last_peered":"2024-09-16T10:30:47.692286+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:47.692286+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18608128,"num_objects":4543,"num_object_clones":0,"num_object_copies":9086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12619,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'15895","reported_seq":16073,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:47.714269+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:47.714269+0000","last_peered":"2024-09-16T10:30:47.714269+0000","last_clean":"2024-09-16T10:30:47.714269+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:47.714269+0000","last_undegraded":"2024-09-16T10:30:47.714269+0000","last_fullsized":"2024-09-16T10:30:47.714269+0000","mapping_epoch":33,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18411520,"num_objects":4495,"num_object_clones":0,"num_object_copies":8990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11698,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16002","reported_seq":24509,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.562296+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:51.562296+0000","last_peered":"2024-09-16T10:30:51.562296+0000","last_clean":"2024-09-16T10:30:51.562296+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:51.562296+0000","last_undegraded":"2024-09-16T10:30:51.562296+0000","last_fullsized":"2024-09-16T10:30:51.562296+0000","mapping_epoch":43,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18265030,"num_objects":4461,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16000,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16458","reported_seq":16734,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.611405+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:47.611405+0000","last_peered":"2024-09-16T10:30:47.611405+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:47.611405+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18768838,"num_objects":4584,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16456,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":247}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16024","reported_seq":16221,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.976095+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:51.976095+0000","last_peered":"2024-09-16T10:30:51.976095+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:51.976095+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18002886,"num_objects":4397,"num_object_clones":0,"num_object_copies":8794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16022,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16101","reported_seq":22310,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.975609+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:51.975609+0000","last_peered":"2024-09-16T10:30:51.975609+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:51.975609+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330566,"num_objects":4477,"num_object_clones":0,"num_object_copies":8954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16099,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3061,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15585","reported_seq":15836,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.572233+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:47.572233+0000","last_peered":"2024-09-16T10:30:47.572233+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:47.572233+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5500","ondisk_log_start":"42'5500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18223587,"num_objects":4450,"num_object_clones":0,"num_object_copies":8900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15584,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":375}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16284","reported_seq":16561,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.706353+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:47.706353+0000","last_peered":"2024-09-16T10:30:47.706353+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:47.706353+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18965929,"num_objects":4633,"num_object_clones":0,"num_object_copies":9266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16281,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15725","reported_seq":15954,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.663510+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:47.663510+0000","last_peered":"2024-09-16T10:30:47.663510+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:47.663510+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18084806,"num_objects":4417,"num_object_clones":0,"num_object_copies":8834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15723,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16357","reported_seq":16599,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:47.654350+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:47.654350+0000","last_peered":"2024-09-16T10:30:47.654350+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:47.654350+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18593676,"num_objects":4543,"num_object_clones":0,"num_object_copies":9086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16354,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":221}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":293466638,"num_objects":71664,"num_object_clones":0,"num_object_copies":143328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":234641,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12142,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":793497600,"data_stored":793413583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160776,"ondisk_log_size":160776,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018981,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589044,"kb_used_data":278504,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93782796,"statfs":{"total":96636764160,"available":96033583104,"internally_reserved":0,"allocated":285188096,"data_stored":284919945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.306},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":13.743}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.5490000000000004},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.04}]}]},{"osd":1,"up_from":40,"seq":171798691925,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":551504,"kb_used_data":213056,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93820336,"statfs":{"total":96636764160,"available":96072024064,"internally_reserved":0,"allocated":218169344,"data_stored":217836159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":346549930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,0,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.774},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.5270000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.601},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.6399999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215207,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":692324,"kb_used_data":285852,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93679516,"statfs":{"total":96636764160,"available":95927824384,"internally_reserved":0,"allocated":292712448,"data_stored":292451983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":4.0510000000000002},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":5.71}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.343},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.0860000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291385344,"data_stored":291381731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217358336,"data_stored":217285617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284753920,"data_stored":284746235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:57.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:58.219 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:58.220 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:58.241 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-09-16T10:30:56.718630+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":287089185,"num_objects":70109,"num_object_clones":0,"num_object_copies":140218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70109,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236200,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12143,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160636,"ondisk_log_size":160636,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1831472,"kb_used_data":763724,"kb_used_omap":12,"kb_used_meta":1067699,"kb_avail":281284048,"statfs":{"total":289910292480,"available":288034865152,"internally_reserved":0,"allocated":782053376,"data_stored":781191575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1093323778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,2,7,8,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19103744,"num_objects":-4664,"num_object_clones":0,"num_object_copies":-9328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4664,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002854"},"pg_stats":[{"pgid":"2.f","version":"46'16105","reported_seq":24568,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:52.762584+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:52.762584+0000","last_peered":"2024-09-16T10:30:52.762584+0000","last_clean":"2024-09-16T10:30:52.762584+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:52.762584+0000","last_undegraded":"2024-09-16T10:30:52.762584+0000","last_fullsized":"2024-09-16T10:30:52.762584+0000","mapping_epoch":45,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17870848,"num_objects":4363,"num_object_clones":0,"num_object_copies":8726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16105,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16077","reported_seq":16238,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.886979+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:51.886979+0000","last_peered":"2024-09-16T10:30:51.886979+0000","last_clean":"2024-09-16T10:30:51.886979+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:51.886979+0000","last_undegraded":"2024-09-16T10:30:51.886979+0000","last_fullsized":"2024-09-16T10:30:51.886979+0000","mapping_epoch":34,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17960960,"num_objects":4385,"num_object_clones":0,"num_object_copies":8770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16265","reported_seq":16518,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.937361+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:51.937361+0000","last_peered":"2024-09-16T10:30:51.937361+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:51.937361+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18014232,"num_objects":4399,"num_object_clones":0,"num_object_copies":8798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12322,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":96}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16113","reported_seq":16172,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:51.971193+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:51.971193+0000","last_peered":"2024-09-16T10:30:51.971193+0000","last_clean":"2024-09-16T10:30:51.971193+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:51.971193+0000","last_undegraded":"2024-09-16T10:30:51.971193+0000","last_fullsized":"2024-09-16T10:30:51.971193+0000","mapping_epoch":34,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17813504,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16113,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16133","reported_seq":16293,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:52.707666+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:52.707666+0000","last_peered":"2024-09-16T10:30:52.707666+0000","last_clean":"2024-09-16T10:30:52.707666+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:52.707666+0000","last_undegraded":"2024-09-16T10:30:52.707666+0000","last_fullsized":"2024-09-16T10:30:52.707666+0000","mapping_epoch":33,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17907712,"num_objects":4372,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12775,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'15998","reported_seq":16331,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:52.705998+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:52.705998+0000","last_peered":"2024-09-16T10:30:52.705998+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:52.705998+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17911808,"num_objects":4373,"num_object_clones":0,"num_object_copies":8746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12623,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16516","reported_seq":16925,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:52.698566+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:52.698566+0000","last_peered":"2024-09-16T10:30:52.698566+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:52.698566+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18038784,"num_objects":4404,"num_object_clones":0,"num_object_copies":8808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12758,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":117}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16030","reported_seq":16209,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:52.569589+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:52.569589+0000","last_peered":"2024-09-16T10:30:52.569589+0000","last_clean":"2024-09-16T10:30:52.569589+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:52.569589+0000","last_undegraded":"2024-09-16T10:30:52.569589+0000","last_fullsized":"2024-09-16T10:30:52.569589+0000","mapping_epoch":33,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17858560,"num_objects":4360,"num_object_clones":0,"num_object_copies":8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11833,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16132","reported_seq":24639,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:56.511400+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:56.511400+0000","last_peered":"2024-09-16T10:30:56.511400+0000","last_clean":"2024-09-16T10:30:56.511400+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:56.511400+0000","last_undegraded":"2024-09-16T10:30:56.511400+0000","last_fullsized":"2024-09-16T10:30:56.511400+0000","mapping_epoch":43,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17732550,"num_objects":4331,"num_object_clones":0,"num_object_copies":8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16130,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16596","reported_seq":16872,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:52.702999+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:52.702999+0000","last_peered":"2024-09-16T10:30:52.702999+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:52.702999+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18203590,"num_objects":4446,"num_object_clones":0,"num_object_copies":8892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16594,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":109}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":91,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T10:29:02.966920+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:02.966920+0000","last_peered":"2024-09-16T10:29:02.966920+0000","last_clean":"2024-09-16T10:29:02.966920+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:02.966920+0000","last_undegraded":"2024-09-16T10:29:02.966920+0000","last_fullsized":"2024-09-16T10:29:02.966920+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16024","reported_seq":16221,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.976095+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:51.976095+0000","last_peered":"2024-09-16T10:30:51.976095+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:51.976095+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18002886,"num_objects":4397,"num_object_clones":0,"num_object_copies":8794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16022,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":164}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16101","reported_seq":22310,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:51.975609+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:51.975609+0000","last_peered":"2024-09-16T10:30:51.975609+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:51.975609+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18330566,"num_objects":4477,"num_object_clones":0,"num_object_copies":8954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16099,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3061,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15732","reported_seq":15982,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:52.735987+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:52.735987+0000","last_peered":"2024-09-16T10:30:52.735987+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:52.735987+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5700","ondisk_log_start":"42'5700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17621475,"num_objects":4303,"num_object_clones":0,"num_object_copies":8606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15731,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":228}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16438","reported_seq":16714,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:52.758255+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:52.758255+0000","last_peered":"2024-09-16T10:30:52.758255+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:52.758255+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18335145,"num_objects":4479,"num_object_clones":0,"num_object_copies":8958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16435,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15860","reported_seq":16092,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:52.667821+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:52.667821+0000","last_peered":"2024-09-16T10:30:52.667821+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:52.667821+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17531846,"num_objects":4282,"num_object_clones":0,"num_object_copies":8564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15858,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":189}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16513","reported_seq":16755,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:52.735262+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:52.735262+0000","last_peered":"2024-09-16T10:30:52.735262+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:52.735262+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17954700,"num_objects":4387,"num_object_clones":0,"num_object_copies":8774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16510,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":287089166,"num_objects":70107,"num_object_clones":0,"num_object_copies":140214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39204,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70107,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":236198,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12142,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":779550720,"data_stored":779466703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160633,"ondisk_log_size":160633,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018981,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589044,"kb_used_data":278504,"kb_used_omap":4,"kb_used_meta":310523,"kb_avail":93782796,"statfs":{"total":96636764160,"available":96033583104,"internally_reserved":0,"allocated":285188096,"data_stored":284919945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":317976240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,1,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":4.306},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":13.743}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.5490000000000004},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.04}]}]},{"osd":1,"up_from":40,"seq":171798691926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552796,"kb_used_data":206160,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93819044,"statfs":{"total":96636764160,"available":96070701056,"internally_reserved":0,"allocated":211107840,"data_stored":210774655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":354938538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.218}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.33500000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215208,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689632,"kb_used_data":279060,"kb_used_omap":4,"kb_used_meta":410555,"kb_avail":93682208,"statfs":{"total":96636764160,"available":95930580992,"internally_reserved":0,"allocated":285757440,"data_stored":285496975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":420409000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.24099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.53100000000000003},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":4.9950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284471296,"data_stored":284467683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":210325504,"data_stored":210252785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":284753920,"data_stored":284746235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:58.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:30:58.807 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:58.808 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:30:58.832 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-16T10:30:58.719121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":278123041,"num_objects":67920,"num_object_clones":0,"num_object_copies":135840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67920,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238389,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12281,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161025,"ondisk_log_size":161025,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1816040,"kb_used_data":752372,"kb_used_omap":12,"kb_used_meta":1063603,"kb_avail":281299480,"statfs":{"total":289910292480,"available":288050667520,"internally_reserved":0,"allocated":770428928,"data_stored":769567127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1089129474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,7,17,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27549696,"num_objects":-6726,"num_object_clones":0,"num_object_copies":-13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6726,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":452,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002896"},"pg_stats":[{"pgid":"2.f","version":"46'16256","reported_seq":24721,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.674115+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:57.674115+0000","last_peered":"2024-09-16T10:30:57.674115+0000","last_clean":"2024-09-16T10:30:57.674115+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:57.674115+0000","last_undegraded":"2024-09-16T10:30:57.674115+0000","last_fullsized":"2024-09-16T10:30:57.674115+0000","mapping_epoch":45,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17252352,"num_objects":4212,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16256,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16220","reported_seq":16381,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:56.912703+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:56.912703+0000","last_peered":"2024-09-16T10:30:56.912703+0000","last_clean":"2024-09-16T10:30:56.912703+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:56.912703+0000","last_undegraded":"2024-09-16T10:30:56.912703+0000","last_fullsized":"2024-09-16T10:30:56.912703+0000","mapping_epoch":34,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17375232,"num_objects":4242,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12433,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16400","reported_seq":16654,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.051650+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:57.051650+0000","last_peered":"2024-09-16T10:30:57.051650+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:57.051650+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17461272,"num_objects":4264,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12457,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-39}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16240","reported_seq":16300,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.024151+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:57.024151+0000","last_peered":"2024-09-16T10:30:57.024151+0000","last_clean":"2024-09-16T10:30:57.024151+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:57.024151+0000","last_undegraded":"2024-09-16T10:30:57.024151+0000","last_fullsized":"2024-09-16T10:30:57.024151+0000","mapping_epoch":34,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17293312,"num_objects":4222,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16240,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16279","reported_seq":16439,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.766161+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:57.766161+0000","last_peered":"2024-09-16T10:30:57.766161+0000","last_clean":"2024-09-16T10:30:57.766161+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:57.766161+0000","last_undegraded":"2024-09-16T10:30:57.766161+0000","last_fullsized":"2024-09-16T10:30:57.766161+0000","mapping_epoch":33,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17309696,"num_objects":4226,"num_object_clones":0,"num_object_copies":8452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12921,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16134","reported_seq":16464,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.701899+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:57.701899+0000","last_peered":"2024-09-16T10:30:57.701899+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:57.701899+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17354752,"num_objects":4237,"num_object_clones":0,"num_object_copies":8474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12759,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":103}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16686","reported_seq":17095,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.737877+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:57.737877+0000","last_peered":"2024-09-16T10:30:57.737877+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:57.737877+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17342464,"num_objects":4234,"num_object_clones":0,"num_object_copies":8468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12928,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-53}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16196","reported_seq":16375,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.701139+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:57.701139+0000","last_peered":"2024-09-16T10:30:57.701139+0000","last_clean":"2024-09-16T10:30:57.701139+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:57.701139+0000","last_undegraded":"2024-09-16T10:30:57.701139+0000","last_fullsized":"2024-09-16T10:30:57.701139+0000","mapping_epoch":33,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17178624,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11999,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16132","reported_seq":24639,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:56.511400+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:56.511400+0000","last_peered":"2024-09-16T10:30:56.511400+0000","last_clean":"2024-09-16T10:30:56.511400+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:56.511400+0000","last_undegraded":"2024-09-16T10:30:56.511400+0000","last_fullsized":"2024-09-16T10:30:56.511400+0000","mapping_epoch":43,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17732550,"num_objects":4331,"num_object_clones":0,"num_object_copies":8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16130,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16737","reported_seq":17013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.683580+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:57.683580+0000","last_peered":"2024-09-16T10:30:57.683580+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:57.683580+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17626054,"num_objects":4305,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16735,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-32}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16166","reported_seq":16363,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.041078+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:57.041078+0000","last_peered":"2024-09-16T10:30:57.041078+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:57.041078+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17421254,"num_objects":4255,"num_object_clones":0,"num_object_copies":8510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16164,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":22}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16262","reported_seq":22748,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.044579+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:57.044579+0000","last_peered":"2024-09-16T10:30:57.044579+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:57.044579+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17671110,"num_objects":4316,"num_object_clones":0,"num_object_copies":8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16260,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3199,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15893","reported_seq":16144,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.751280+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:57.751280+0000","last_peered":"2024-09-16T10:30:57.751280+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:57.751280+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16962019,"num_objects":4142,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15892,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":67}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16568","reported_seq":16845,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.779698+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:57.779698+0000","last_peered":"2024-09-16T10:30:57.779698+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:57.779698+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17802665,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16565,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":82}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15971","reported_seq":16203,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.593014+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:57.593014+0000","last_peered":"2024-09-16T10:30:57.593014+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:57.593014+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17077190,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15969,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16682","reported_seq":16924,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.768698+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:57.768698+0000","last_peered":"2024-09-16T10:30:57.768698+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:57.768698+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17262476,"num_objects":4218,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16679,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":278123022,"num_objects":67918,"num_object_clones":0,"num_object_copies":135836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67918,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238387,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12280,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":767602688,"data_stored":767518671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161022,"ondisk_log_size":161022,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018982,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572272,"kb_used_data":274020,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93799568,"statfs":{"total":96636764160,"available":96050757632,"internally_reserved":0,"allocated":280596480,"data_stored":280328329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.2090000000000001},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.379}]}]},{"osd":1,"up_from":40,"seq":171798691926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552796,"kb_used_data":206160,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93819044,"statfs":{"total":96636764160,"available":96070701056,"internally_reserved":0,"allocated":211107840,"data_stored":210774655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":354938538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.218}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.33500000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215209,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690972,"kb_used_data":272192,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93680868,"statfs":{"total":96636764160,"available":95929208832,"internally_reserved":0,"allocated":278724608,"data_stored":278464143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":428797608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.5920000000000001},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":14.301}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":3.0910000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.5630000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277233664,"data_stored":277230051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":210325504,"data_stored":210252785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280043520,"data_stored":280035835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:30:58.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:30:58.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'16400", 'reported_seq': 16654, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.051650+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:30:57.051650+0000', 'last_peered': '2024-09-16T10:30:57.051650+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:30:57.051650+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'6300", 'ondisk_log_start': "42'6300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17461272, 'num_objects': 4264, 'num_object_clones': 0, 'num_object_copies': 8528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4264, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12457, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -39}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:58.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:30:58.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'16134", 'reported_seq': 16464, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.701899+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:57.701899+0000', 'last_peered': '2024-09-16T10:30:57.701899+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:57.701899+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'6100", 'ondisk_log_start': "42'6100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17354752, 'num_objects': 4237, 'num_object_clones': 0, 'num_object_copies': 8474, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12759, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:58.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:30:58.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'16686", 'reported_seq': 17095, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.737877+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:57.737877+0000', 'last_peered': '2024-09-16T10:30:57.737877+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:57.737877+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'6600", 'ondisk_log_start': "42'6600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17342464, 'num_objects': 4234, 'num_object_clones': 0, 'num_object_copies': 8468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12928, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -53}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:58.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:30:58.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'16737", 'reported_seq': 17013, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.683580+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:57.683580+0000', 'last_peered': '2024-09-16T10:30:57.683580+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:57.683580+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'6700", 'ondisk_log_start': "42'6700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17626054, 'num_objects': 4305, 'num_object_clones': 0, 'num_object_copies': 8610, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16735, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -32}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:58.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'16166", 'reported_seq': 16363, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.041078+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:30:57.041078+0000', 'last_peered': '2024-09-16T10:30:57.041078+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:30:57.041078+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'6100", 'ondisk_log_start': "42'6100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17421254, 'num_objects': 4255, 'num_object_clones': 0, 'num_object_copies': 8510, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16164, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 22}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'16262", 'reported_seq': 22748, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.044579+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:30:57.044579+0000', 'last_peered': '2024-09-16T10:30:57.044579+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:30:57.044579+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'6200", 'ondisk_log_start': "42'6200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17671110, 'num_objects': 4316, 'num_object_clones': 0, 'num_object_copies': 8632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 997, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16260, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3199, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3319}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'15893", 'reported_seq': 16144, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.751280+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:57.751280+0000', 'last_peered': '2024-09-16T10:30:57.751280+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:57.751280+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'5800", 'ondisk_log_start': "42'5800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16962019, 'num_objects': 4142, 'num_object_clones': 0, 'num_object_copies': 8284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15892, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 67}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'16568", 'reported_seq': 16845, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.779698+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:57.779698+0000', 'last_peered': '2024-09-16T10:30:57.779698+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:57.779698+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'6500", 'ondisk_log_start': "42'6500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17802665, 'num_objects': 4349, 'num_object_clones': 0, 'num_object_copies': 8698, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16565, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 82}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'15971", 'reported_seq': 16203, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.593014+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:57.593014+0000', 'last_peered': '2024-09-16T10:30:57.593014+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:57.593014+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'5900", 'ondisk_log_start': "42'5900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17077190, 'num_objects': 4171, 'num_object_clones': 0, 'num_object_copies': 8342, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15969, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 78}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:30:58.835 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'16682", 'reported_seq': 16924, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.768698+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:57.768698+0000', 'last_peered': '2024-09-16T10:30:57.768698+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:57.768698+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'6600", 'ondisk_log_start': "42'6600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17262476, 'num_objects': 4218, 'num_object_clones': 0, 'num_object_copies': 8436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16679, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:30:58.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:30:59.530 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:30:59.530 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 39066/135840 objects degraded (28.759%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":548,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":67920,"data_bytes":278123041,"bytes_used":1859624960,"bytes_avail":288050667520,"bytes_total":289910292480,"degraded_objects":39066,"degraded_total":135840,"degraded_ratio":0.28758833922261484,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":452,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":560},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:30:59.552 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:30:59.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:00.134 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:00.136 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:00.163 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-16T10:30:58.719121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":278123041,"num_objects":67920,"num_object_clones":0,"num_object_copies":135840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67920,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238389,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12281,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161025,"ondisk_log_size":161025,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1816040,"kb_used_data":752372,"kb_used_omap":12,"kb_used_meta":1063603,"kb_avail":281299480,"statfs":{"total":289910292480,"available":288050667520,"internally_reserved":0,"allocated":770428928,"data_stored":769567127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1089129474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,7,17,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27549696,"num_objects":-6726,"num_object_clones":0,"num_object_copies":-13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-452,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6726,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":452,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002896"},"pg_stats":[{"pgid":"2.f","version":"46'16256","reported_seq":24721,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.674115+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:57.674115+0000","last_peered":"2024-09-16T10:30:57.674115+0000","last_clean":"2024-09-16T10:30:57.674115+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:57.674115+0000","last_undegraded":"2024-09-16T10:30:57.674115+0000","last_fullsized":"2024-09-16T10:30:57.674115+0000","mapping_epoch":45,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17252352,"num_objects":4212,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16256,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16220","reported_seq":16381,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:56.912703+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:30:56.912703+0000","last_peered":"2024-09-16T10:30:56.912703+0000","last_clean":"2024-09-16T10:30:56.912703+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:30:56.912703+0000","last_undegraded":"2024-09-16T10:30:56.912703+0000","last_fullsized":"2024-09-16T10:30:56.912703+0000","mapping_epoch":34,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17375232,"num_objects":4242,"num_object_clones":0,"num_object_copies":8484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12433,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16400","reported_seq":16654,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.051650+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:30:57.051650+0000","last_peered":"2024-09-16T10:30:57.051650+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:30:57.051650+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17461272,"num_objects":4264,"num_object_clones":0,"num_object_copies":8528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4264,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12457,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-39}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16240","reported_seq":16300,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.024151+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:30:57.024151+0000","last_peered":"2024-09-16T10:30:57.024151+0000","last_clean":"2024-09-16T10:30:57.024151+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:30:57.024151+0000","last_undegraded":"2024-09-16T10:30:57.024151+0000","last_fullsized":"2024-09-16T10:30:57.024151+0000","mapping_epoch":34,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17293312,"num_objects":4222,"num_object_clones":0,"num_object_copies":8444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16240,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16279","reported_seq":16439,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.766161+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:57.766161+0000","last_peered":"2024-09-16T10:30:57.766161+0000","last_clean":"2024-09-16T10:30:57.766161+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:57.766161+0000","last_undegraded":"2024-09-16T10:30:57.766161+0000","last_fullsized":"2024-09-16T10:30:57.766161+0000","mapping_epoch":33,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17309696,"num_objects":4226,"num_object_clones":0,"num_object_copies":8452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12921,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16134","reported_seq":16464,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.701899+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:57.701899+0000","last_peered":"2024-09-16T10:30:57.701899+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:57.701899+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17354752,"num_objects":4237,"num_object_clones":0,"num_object_copies":8474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12759,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":103}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16686","reported_seq":17095,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.737877+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:57.737877+0000","last_peered":"2024-09-16T10:30:57.737877+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:57.737877+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17342464,"num_objects":4234,"num_object_clones":0,"num_object_copies":8468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12928,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-53}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16196","reported_seq":16375,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.701139+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:57.701139+0000","last_peered":"2024-09-16T10:30:57.701139+0000","last_clean":"2024-09-16T10:30:57.701139+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:57.701139+0000","last_undegraded":"2024-09-16T10:30:57.701139+0000","last_fullsized":"2024-09-16T10:30:57.701139+0000","mapping_epoch":33,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17178624,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11999,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16132","reported_seq":24639,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:56.511400+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:30:56.511400+0000","last_peered":"2024-09-16T10:30:56.511400+0000","last_clean":"2024-09-16T10:30:56.511400+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:30:56.511400+0000","last_undegraded":"2024-09-16T10:30:56.511400+0000","last_fullsized":"2024-09-16T10:30:56.511400+0000","mapping_epoch":43,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17732550,"num_objects":4331,"num_object_clones":0,"num_object_copies":8662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16130,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16737","reported_seq":17013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.683580+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:57.683580+0000","last_peered":"2024-09-16T10:30:57.683580+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:57.683580+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17626054,"num_objects":4305,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16735,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-32}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16166","reported_seq":16363,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.041078+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:30:57.041078+0000","last_peered":"2024-09-16T10:30:57.041078+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:30:57.041078+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17421254,"num_objects":4255,"num_object_clones":0,"num_object_copies":8510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16164,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":22}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16262","reported_seq":22748,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.044579+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:30:57.044579+0000","last_peered":"2024-09-16T10:30:57.044579+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:30:57.044579+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17671110,"num_objects":4316,"num_object_clones":0,"num_object_copies":8632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":997,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16260,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3199,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3319}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15893","reported_seq":16144,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.751280+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:57.751280+0000","last_peered":"2024-09-16T10:30:57.751280+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:57.751280+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16962019,"num_objects":4142,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15892,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":67}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16568","reported_seq":16845,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.779698+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:57.779698+0000","last_peered":"2024-09-16T10:30:57.779698+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:57.779698+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17802665,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16565,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":82}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15971","reported_seq":16203,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.593014+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:57.593014+0000","last_peered":"2024-09-16T10:30:57.593014+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:57.593014+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17077190,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15969,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16682","reported_seq":16924,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.768698+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:57.768698+0000","last_peered":"2024-09-16T10:30:57.768698+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:57.768698+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17262476,"num_objects":4218,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16679,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":278123022,"num_objects":67918,"num_object_clones":0,"num_object_copies":135836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67918,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":238387,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12280,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":767602688,"data_stored":767518671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161022,"ondisk_log_size":161022,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018982,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572272,"kb_used_data":274020,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93799568,"statfs":{"total":96636764160,"available":96050757632,"internally_reserved":0,"allocated":280596480,"data_stored":280328329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":305393328},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.35899999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.2090000000000001},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.379}]}]},{"osd":1,"up_from":40,"seq":171798691926,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552796,"kb_used_data":206160,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93819044,"statfs":{"total":96636764160,"available":96070701056,"internally_reserved":0,"allocated":211107840,"data_stored":210774655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":354938538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":4.218}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.33500000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215209,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690972,"kb_used_data":272192,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93680868,"statfs":{"total":96636764160,"available":95929208832,"internally_reserved":0,"allocated":278724608,"data_stored":278464143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":428797608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.5920000000000001},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":14.301}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":3.0910000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.5630000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277233664,"data_stored":277230051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":210325504,"data_stored":210252785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":280043520,"data_stored":280035835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:03.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:03.716 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:03.719 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:03.741 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-16T10:31:02.720084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274653729,"num_objects":67073,"num_object_clones":0,"num_object_copies":134146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67073,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239236,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12450,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161072,"ondisk_log_size":161072,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1796636,"kb_used_data":741168,"kb_used_omap":12,"kb_used_meta":1055411,"kb_avail":281318884,"statfs":{"total":289910292480,"available":288070537216,"internally_reserved":0,"allocated":758956032,"data_stored":758094231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1080740866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,6,4,19,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22376448,"num_objects":-5463,"num_object_clones":0,"num_object_copies":-10926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5463,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"2.f","version":"46'16256","reported_seq":24721,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.674115+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:57.674115+0000","last_peered":"2024-09-16T10:30:57.674115+0000","last_clean":"2024-09-16T10:30:57.674115+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:57.674115+0000","last_undegraded":"2024-09-16T10:30:57.674115+0000","last_fullsized":"2024-09-16T10:30:57.674115+0000","mapping_epoch":45,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17252352,"num_objects":4212,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16256,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16358","reported_seq":16519,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:02.061034+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:02.061034+0000","last_peered":"2024-09-16T10:31:02.061034+0000","last_clean":"2024-09-16T10:31:02.061034+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:02.061034+0000","last_undegraded":"2024-09-16T10:31:02.061034+0000","last_fullsized":"2024-09-16T10:31:02.061034+0000","mapping_epoch":34,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16809984,"num_objects":4104,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16545","reported_seq":16798,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.050642+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:02.050642+0000","last_peered":"2024-09-16T10:31:02.050642+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:02.050642+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16867352,"num_objects":4119,"num_object_clones":0,"num_object_copies":8238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12602,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-184}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16392","reported_seq":16451,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:02.109118+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:02.109118+0000","last_peered":"2024-09-16T10:31:02.109118+0000","last_clean":"2024-09-16T10:31:02.109118+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:02.109118+0000","last_undegraded":"2024-09-16T10:31:02.109118+0000","last_fullsized":"2024-09-16T10:31:02.109118+0000","mapping_epoch":34,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16670720,"num_objects":4070,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16392,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16279","reported_seq":16439,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.766161+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:57.766161+0000","last_peered":"2024-09-16T10:30:57.766161+0000","last_clean":"2024-09-16T10:30:57.766161+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:57.766161+0000","last_undegraded":"2024-09-16T10:30:57.766161+0000","last_fullsized":"2024-09-16T10:30:57.766161+0000","mapping_epoch":33,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17309696,"num_objects":4226,"num_object_clones":0,"num_object_copies":8452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12921,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16134","reported_seq":16464,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.701899+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:57.701899+0000","last_peered":"2024-09-16T10:30:57.701899+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:57.701899+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17354752,"num_objects":4237,"num_object_clones":0,"num_object_copies":8474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12759,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":103}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16686","reported_seq":17095,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.737877+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:57.737877+0000","last_peered":"2024-09-16T10:30:57.737877+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:57.737877+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17342464,"num_objects":4234,"num_object_clones":0,"num_object_copies":8468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12928,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-53}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16196","reported_seq":16375,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.701139+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:57.701139+0000","last_peered":"2024-09-16T10:30:57.701139+0000","last_clean":"2024-09-16T10:30:57.701139+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:57.701139+0000","last_undegraded":"2024-09-16T10:30:57.701139+0000","last_fullsized":"2024-09-16T10:30:57.701139+0000","mapping_epoch":33,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17178624,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11999,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16274","reported_seq":24781,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:01.601625+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:01.601625+0000","last_peered":"2024-09-16T10:31:01.601625+0000","last_clean":"2024-09-16T10:31:01.601625+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:01.601625+0000","last_undegraded":"2024-09-16T10:31:01.601625+0000","last_fullsized":"2024-09-16T10:31:01.601625+0000","mapping_epoch":43,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17150918,"num_objects":4189,"num_object_clones":0,"num_object_copies":8378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16272,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16737","reported_seq":17013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.683580+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:57.683580+0000","last_peered":"2024-09-16T10:30:57.683580+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:57.683580+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17626054,"num_objects":4305,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16735,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-32}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16291","reported_seq":16488,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.091497+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:02.091497+0000","last_peered":"2024-09-16T10:31:02.091497+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:02.091497+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16909254,"num_objects":4130,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16289,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-103}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16407","reported_seq":23231,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.109645+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:02.109645+0000","last_peered":"2024-09-16T10:31:02.109645+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:02.109645+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17077190,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16405,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3368,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3343}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15893","reported_seq":16144,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.751280+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:57.751280+0000","last_peered":"2024-09-16T10:30:57.751280+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:57.751280+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16962019,"num_objects":4142,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15892,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":67}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16568","reported_seq":16845,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.779698+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:57.779698+0000","last_peered":"2024-09-16T10:30:57.779698+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:57.779698+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17802665,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16565,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":82}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15971","reported_seq":16203,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.593014+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:57.593014+0000","last_peered":"2024-09-16T10:30:57.593014+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:57.593014+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17077190,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15969,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16682","reported_seq":16924,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.768698+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:57.768698+0000","last_peered":"2024-09-16T10:30:57.768698+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:57.768698+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17262476,"num_objects":4218,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16679,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":274653710,"num_objects":67071,"num_object_clones":0,"num_object_copies":134142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67071,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239234,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12449,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":755961856,"data_stored":755877839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161069,"ondisk_log_size":161069,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018983,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571972,"kb_used_data":269628,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93799868,"statfs":{"total":96636764160,"available":96051064832,"internally_reserved":0,"allocated":276099072,"data_stored":275830921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.7639999999999998},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.59499999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":7.5819999999999999},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.33900000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691927,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533692,"kb_used_data":199348,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93838148,"statfs":{"total":96636764160,"available":96090263552,"internally_reserved":0,"allocated":204132352,"data_stored":203799167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":342355626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.972},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.0059999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.28599999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215209,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690972,"kb_used_data":272192,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93680868,"statfs":{"total":96636764160,"available":95929208832,"internally_reserved":0,"allocated":278724608,"data_stored":278464143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":428797608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.5920000000000001},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":14.301}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":3.0910000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.5630000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277233664,"data_stored":277230051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":203243520,"data_stored":203170801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275484672,"data_stored":275476987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:03.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:04.302 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:04.304 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:04.335 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-16T10:31:02.720084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":274653729,"num_objects":67073,"num_object_clones":0,"num_object_copies":134146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67073,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239236,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12450,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161072,"ondisk_log_size":161072,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1796636,"kb_used_data":741168,"kb_used_omap":12,"kb_used_meta":1055411,"kb_avail":281318884,"statfs":{"total":289910292480,"available":288070537216,"internally_reserved":0,"allocated":758956032,"data_stored":758094231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1080740866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,6,4,19,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22376448,"num_objects":-5463,"num_object_clones":0,"num_object_copies":-10926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5463,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"2.f","version":"46'16256","reported_seq":24721,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.674115+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:30:57.674115+0000","last_peered":"2024-09-16T10:30:57.674115+0000","last_clean":"2024-09-16T10:30:57.674115+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:30:57.674115+0000","last_undegraded":"2024-09-16T10:30:57.674115+0000","last_fullsized":"2024-09-16T10:30:57.674115+0000","mapping_epoch":45,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17252352,"num_objects":4212,"num_object_clones":0,"num_object_copies":8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16256,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16358","reported_seq":16519,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:02.061034+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:02.061034+0000","last_peered":"2024-09-16T10:31:02.061034+0000","last_clean":"2024-09-16T10:31:02.061034+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:02.061034+0000","last_undegraded":"2024-09-16T10:31:02.061034+0000","last_fullsized":"2024-09-16T10:31:02.061034+0000","mapping_epoch":34,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16809984,"num_objects":4104,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16545","reported_seq":16798,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.050642+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:02.050642+0000","last_peered":"2024-09-16T10:31:02.050642+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:02.050642+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16867352,"num_objects":4119,"num_object_clones":0,"num_object_copies":8238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12602,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-184}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16392","reported_seq":16451,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:02.109118+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:02.109118+0000","last_peered":"2024-09-16T10:31:02.109118+0000","last_clean":"2024-09-16T10:31:02.109118+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:02.109118+0000","last_undegraded":"2024-09-16T10:31:02.109118+0000","last_fullsized":"2024-09-16T10:31:02.109118+0000","mapping_epoch":34,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16670720,"num_objects":4070,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16392,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16279","reported_seq":16439,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.766161+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:30:57.766161+0000","last_peered":"2024-09-16T10:30:57.766161+0000","last_clean":"2024-09-16T10:30:57.766161+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:30:57.766161+0000","last_undegraded":"2024-09-16T10:30:57.766161+0000","last_fullsized":"2024-09-16T10:30:57.766161+0000","mapping_epoch":33,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17309696,"num_objects":4226,"num_object_clones":0,"num_object_copies":8452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12921,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16134","reported_seq":16464,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.701899+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:30:57.701899+0000","last_peered":"2024-09-16T10:30:57.701899+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:30:57.701899+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17354752,"num_objects":4237,"num_object_clones":0,"num_object_copies":8474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12759,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":103}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16686","reported_seq":17095,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.737877+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:30:57.737877+0000","last_peered":"2024-09-16T10:30:57.737877+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:30:57.737877+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17342464,"num_objects":4234,"num_object_clones":0,"num_object_copies":8468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12928,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-53}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16196","reported_seq":16375,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:30:57.701139+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:30:57.701139+0000","last_peered":"2024-09-16T10:30:57.701139+0000","last_clean":"2024-09-16T10:30:57.701139+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:30:57.701139+0000","last_undegraded":"2024-09-16T10:30:57.701139+0000","last_fullsized":"2024-09-16T10:30:57.701139+0000","mapping_epoch":33,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17178624,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11999,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16274","reported_seq":24781,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:01.601625+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:01.601625+0000","last_peered":"2024-09-16T10:31:01.601625+0000","last_clean":"2024-09-16T10:31:01.601625+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:01.601625+0000","last_undegraded":"2024-09-16T10:31:01.601625+0000","last_fullsized":"2024-09-16T10:31:01.601625+0000","mapping_epoch":43,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17150918,"num_objects":4189,"num_object_clones":0,"num_object_copies":8378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16272,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16737","reported_seq":17013,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.683580+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:30:57.683580+0000","last_peered":"2024-09-16T10:30:57.683580+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:30:57.683580+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17626054,"num_objects":4305,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16735,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-32}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16291","reported_seq":16488,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.091497+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:02.091497+0000","last_peered":"2024-09-16T10:31:02.091497+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:02.091497+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16909254,"num_objects":4130,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16289,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-103}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16407","reported_seq":23231,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.109645+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:02.109645+0000","last_peered":"2024-09-16T10:31:02.109645+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:02.109645+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17077190,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16405,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3368,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3343}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'15893","reported_seq":16144,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.751280+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:30:57.751280+0000","last_peered":"2024-09-16T10:30:57.751280+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:30:57.751280+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'5800","ondisk_log_start":"42'5800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16962019,"num_objects":4142,"num_object_clones":0,"num_object_copies":8284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15892,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":67}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16568","reported_seq":16845,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.779698+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:30:57.779698+0000","last_peered":"2024-09-16T10:30:57.779698+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:30:57.779698+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17802665,"num_objects":4349,"num_object_clones":0,"num_object_copies":8698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16565,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":82}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'15971","reported_seq":16203,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.593014+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:30:57.593014+0000","last_peered":"2024-09-16T10:30:57.593014+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:30:57.593014+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'5900","ondisk_log_start":"42'5900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17077190,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15969,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":78}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16682","reported_seq":16924,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:30:57.768698+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:30:57.768698+0000","last_peered":"2024-09-16T10:30:57.768698+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:30:57.768698+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17262476,"num_objects":4218,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16679,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":274653710,"num_objects":67071,"num_object_clones":0,"num_object_copies":134142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67071,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":239234,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12449,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":755961856,"data_stored":755877839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161069,"ondisk_log_size":161069,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018983,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571972,"kb_used_data":269628,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93799868,"statfs":{"total":96636764160,"available":96051064832,"internally_reserved":0,"allocated":276099072,"data_stored":275830921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.7639999999999998},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.59499999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":7.5819999999999999},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.33900000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691927,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533692,"kb_used_data":199348,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93838148,"statfs":{"total":96636764160,"available":96090263552,"internally_reserved":0,"allocated":204132352,"data_stored":203799167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":342355626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.972},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.0059999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.28599999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215209,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":690972,"kb_used_data":272192,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93680868,"statfs":{"total":96636764160,"available":95929208832,"internally_reserved":0,"allocated":278724608,"data_stored":278464143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":428797608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.5920000000000001},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":14.301}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":3.0910000000000002},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.5630000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":277233664,"data_stored":277230051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":203243520,"data_stored":203170801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275484672,"data_stored":275476987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:04.336 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:04.336 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'16545", 'reported_seq': 16798, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:02.050642+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:02.050642+0000', 'last_peered': '2024-09-16T10:31:02.050642+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:02.050642+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'6500", 'ondisk_log_start': "42'6500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16867352, 'num_objects': 4119, 'num_object_clones': 0, 'num_object_copies': 8238, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4119, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12602, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -184}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:04.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:04.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'16134", 'reported_seq': 16464, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.701899+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:30:57.701899+0000', 'last_peered': '2024-09-16T10:30:57.701899+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:30:57.701899+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'6100", 'ondisk_log_start': "42'6100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17354752, 'num_objects': 4237, 'num_object_clones': 0, 'num_object_copies': 8474, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12759, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 103}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:04.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:04.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'16686", 'reported_seq': 17095, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.737877+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:30:57.737877+0000', 'last_peered': '2024-09-16T10:30:57.737877+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:30:57.737877+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'6600", 'ondisk_log_start': "42'6600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17342464, 'num_objects': 4234, 'num_object_clones': 0, 'num_object_copies': 8468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12928, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -53}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:04.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:04.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'16737", 'reported_seq': 17013, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.683580+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:30:57.683580+0000', 'last_peered': '2024-09-16T10:30:57.683580+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:30:57.683580+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'6700", 'ondisk_log_start': "42'6700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17626054, 'num_objects': 4305, 'num_object_clones': 0, 'num_object_copies': 8610, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16735, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -32}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:04.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:04.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'16291", 'reported_seq': 16488, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:02.091497+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:02.091497+0000', 'last_peered': '2024-09-16T10:31:02.091497+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:02.091497+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'6200", 'ondisk_log_start': "42'6200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16909254, 'num_objects': 4130, 'num_object_clones': 0, 'num_object_copies': 8260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16289, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -103}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:04.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:31:04.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'16407", 'reported_seq': 23231, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:02.109645+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:31:02.109645+0000', 'last_peered': '2024-09-16T10:31:02.109645+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:31:02.109645+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'6400", 'ondisk_log_start': "42'6400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17077190, 'num_objects': 4171, 'num_object_clones': 0, 'num_object_copies': 8342, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 828, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16405, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3368, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3343}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:04.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:04.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'15893", 'reported_seq': 16144, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.751280+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:30:57.751280+0000', 'last_peered': '2024-09-16T10:30:57.751280+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:30:57.751280+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'5800", 'ondisk_log_start': "42'5800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16962019, 'num_objects': 4142, 'num_object_clones': 0, 'num_object_copies': 8284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15892, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 67}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:04.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:04.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'16568", 'reported_seq': 16845, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.779698+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:30:57.779698+0000', 'last_peered': '2024-09-16T10:30:57.779698+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:30:57.779698+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'6500", 'ondisk_log_start': "42'6500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17802665, 'num_objects': 4349, 'num_object_clones': 0, 'num_object_copies': 8698, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16565, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 82}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:04.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:04.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'15971", 'reported_seq': 16203, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.593014+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:30:57.593014+0000', 'last_peered': '2024-09-16T10:30:57.593014+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:30:57.593014+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'5900", 'ondisk_log_start': "42'5900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17077190, 'num_objects': 4171, 'num_object_clones': 0, 'num_object_copies': 8342, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15969, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': 78}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:04.338 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:04.339 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'16682", 'reported_seq': 16924, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:30:57.768698+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:30:57.768698+0000', 'last_peered': '2024-09-16T10:30:57.768698+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:30:57.768698+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'6600", 'ondisk_log_start': "42'6600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17262476, 'num_objects': 4218, 'num_object_clones': 0, 'num_object_copies': 8436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16679, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:04.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:05.019 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:05.019 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 38897/134146 objects degraded (28.996%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":553,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":67073,"data_bytes":274653729,"bytes_used":1839755264,"bytes_avail":288070537216,"bytes_total":289910292480,"degraded_objects":38897,"degraded_total":134146,"degraded_ratio":0.28996019262594486,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":458,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":455},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:31:05.044 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:31:05.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:05.615 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:05.618 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:05.636 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-16T10:31:04.720611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":268673569,"num_objects":65613,"num_object_clones":0,"num_object_copies":131226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240696,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12450,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160832,"ondisk_log_size":160832,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1777304,"kb_used_data":734140,"kb_used_omap":12,"kb_used_meta":1043123,"kb_avail":281338216,"statfs":{"total":289910292480,"available":288090333184,"internally_reserved":0,"allocated":751759360,"data_stored":750897559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1068157954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,9,5,11,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24793088,"num_objects":-6053,"num_object_clones":0,"num_object_copies":-12106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6053,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002988"},"pg_stats":[{"pgid":"2.f","version":"46'16399","reported_seq":24864,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:02.837945+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:02.837945+0000","last_peered":"2024-09-16T10:31:02.837945+0000","last_clean":"2024-09-16T10:31:02.837945+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:02.837945+0000","last_undegraded":"2024-09-16T10:31:02.837945+0000","last_fullsized":"2024-09-16T10:31:02.837945+0000","mapping_epoch":45,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16666624,"num_objects":4069,"num_object_clones":0,"num_object_copies":8138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16399,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16358","reported_seq":16519,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:02.061034+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:02.061034+0000","last_peered":"2024-09-16T10:31:02.061034+0000","last_clean":"2024-09-16T10:31:02.061034+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:02.061034+0000","last_undegraded":"2024-09-16T10:31:02.061034+0000","last_fullsized":"2024-09-16T10:31:02.061034+0000","mapping_epoch":34,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16809984,"num_objects":4104,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12571,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16545","reported_seq":16798,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.050642+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:02.050642+0000","last_peered":"2024-09-16T10:31:02.050642+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:02.050642+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16867352,"num_objects":4119,"num_object_clones":0,"num_object_copies":8238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12602,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-184}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16392","reported_seq":16451,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:02.109118+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:02.109118+0000","last_peered":"2024-09-16T10:31:02.109118+0000","last_clean":"2024-09-16T10:31:02.109118+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:02.109118+0000","last_undegraded":"2024-09-16T10:31:02.109118+0000","last_fullsized":"2024-09-16T10:31:02.109118+0000","mapping_epoch":34,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16670720,"num_objects":4070,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16392,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16427","reported_seq":16585,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:02.867365+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:02.867365+0000","last_peered":"2024-09-16T10:31:02.867365+0000","last_clean":"2024-09-16T10:31:02.867365+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:02.867365+0000","last_undegraded":"2024-09-16T10:31:02.867365+0000","last_fullsized":"2024-09-16T10:31:02.867365+0000","mapping_epoch":33,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16703488,"num_objects":4078,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13069,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16281","reported_seq":16614,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.768889+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:02.768889+0000","last_peered":"2024-09-16T10:31:02.768889+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:02.768889+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16752640,"num_objects":4090,"num_object_clones":0,"num_object_copies":8180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12906,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-44}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16842","reported_seq":17250,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.820615+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:02.820615+0000","last_peered":"2024-09-16T10:31:02.820615+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:02.820615+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16703488,"num_objects":4078,"num_object_clones":0,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13084,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16327","reported_seq":16506,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:02.795359+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:02.795359+0000","last_peered":"2024-09-16T10:31:02.795359+0000","last_clean":"2024-09-16T10:31:02.795359+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:02.795359+0000","last_undegraded":"2024-09-16T10:31:02.795359+0000","last_fullsized":"2024-09-16T10:31:02.795359+0000","mapping_epoch":33,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16642048,"num_objects":4063,"num_object_clones":0,"num_object_copies":8126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12130,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16274","reported_seq":24781,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:01.601625+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:01.601625+0000","last_peered":"2024-09-16T10:31:01.601625+0000","last_clean":"2024-09-16T10:31:01.601625+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:01.601625+0000","last_undegraded":"2024-09-16T10:31:01.601625+0000","last_fullsized":"2024-09-16T10:31:01.601625+0000","mapping_epoch":43,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17150918,"num_objects":4189,"num_object_clones":0,"num_object_copies":8378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16272,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'16891","reported_seq":17168,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.824314+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:02.824314+0000","last_peered":"2024-09-16T10:31:02.824314+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:02.824314+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16995270,"num_objects":4151,"num_object_clones":0,"num_object_copies":8302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16889,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-186}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16291","reported_seq":16488,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.091497+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:02.091497+0000","last_peered":"2024-09-16T10:31:02.091497+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:02.091497+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16909254,"num_objects":4130,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16289,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-103}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16407","reported_seq":23231,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.109645+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:02.109645+0000","last_peered":"2024-09-16T10:31:02.109645+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:02.109645+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17077190,"num_objects":4171,"num_object_clones":0,"num_object_copies":8342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16405,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3368,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3343}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16042","reported_seq":16293,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.838074+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:02.838074+0000","last_peered":"2024-09-16T10:31:02.838074+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:02.838074+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6000","ondisk_log_start":"42'6000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16351715,"num_objects":3993,"num_object_clones":0,"num_object_copies":7986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16041,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-82}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16723","reported_seq":17000,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.821734+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:02.821734+0000","last_peered":"2024-09-16T10:31:02.821734+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:02.821734+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17167785,"num_objects":4194,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16720,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-73}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16123","reported_seq":16355,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.774716+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:02.774716+0000","last_peered":"2024-09-16T10:31:02.774716+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:02.774716+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16454598,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16121,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-74}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16807","reported_seq":17047,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:02.798933+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:02.798933+0000","last_peered":"2024-09-16T10:31:02.798933+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:02.798933+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16750476,"num_objects":4093,"num_object_clones":0,"num_object_copies":8186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16804,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-229}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":268673550,"num_objects":65611,"num_object_clones":0,"num_object_copies":131222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65611,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240694,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12449,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":748797952,"data_stored":748713935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160829,"ondisk_log_size":160829,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018983,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571972,"kb_used_data":269628,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93799868,"statfs":{"total":96636764160,"available":96051064832,"internally_reserved":0,"allocated":276099072,"data_stored":275830921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":309587632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4199999999999999,"5min":1.6950000000000001,"15min":1.1819999999999999},"min":{"1min":0.29599999999999999,"5min":0.19,"15min":0.184},"max":{"1min":5.1260000000000003,"5min":6.6299999999999999,"15min":6.6299999999999999},"last":2.7639999999999998},{"interface":"front","average":{"1min":2.0070000000000001,"5min":2.2519999999999998,"15min":1.476},"min":{"1min":0.22900000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.917,"5min":13.917,"15min":13.917},"last":0.59499999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:30:03 2024","interfaces":[{"interface":"back","average":{"1min":3.097,"5min":2.8679999999999999,"15min":2.8879999999999999},"min":{"1min":0.20999999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.336,"5min":15.157999999999999,"15min":15.157999999999999},"last":7.5819999999999999},{"interface":"front","average":{"1min":3.8690000000000002,"5min":3.073,"15min":2.5179999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":17.219999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.33900000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691927,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533692,"kb_used_data":199348,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93838148,"statfs":{"total":96636764160,"available":96090263552,"internally_reserved":0,"allocated":204132352,"data_stored":203799167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":342355626},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.972},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":3.0059999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22500000000000001},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.28599999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215210,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":671640,"kb_used_data":265164,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93700200,"statfs":{"total":96636764160,"available":95949004800,"internally_reserved":0,"allocated":271527936,"data_stored":271267471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,3,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.097},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":0.40699999999999997},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.198}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270069760,"data_stored":270066147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":203243520,"data_stored":203170801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":275484672,"data_stored":275476987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:08.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:09.207 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:09.209 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:09.234 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-09-16T10:31:08.721533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":259301921,"num_objects":63325,"num_object_clones":0,"num_object_copies":126650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242984,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12599,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1797740,"kb_used_data":716052,"kb_used_omap":12,"kb_used_meta":1081651,"kb_avail":281317780,"statfs":{"total":289910292480,"available":288069406720,"internally_reserved":0,"allocated":733237248,"data_stored":732375447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1107610626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,6,5,3,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27787264,"num_objects":-6784,"num_object_clones":0,"num_object_copies":-13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002903"},"pg_stats":[{"pgid":"2.f","version":"46'16542","reported_seq":25007,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.915580+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:07.915580+0000","last_peered":"2024-09-16T10:31:07.915580+0000","last_clean":"2024-09-16T10:31:07.915580+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:07.915580+0000","last_undegraded":"2024-09-16T10:31:07.915580+0000","last_fullsized":"2024-09-16T10:31:07.915580+0000","mapping_epoch":45,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":3926,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16542,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16513","reported_seq":16674,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.133152+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:07.133152+0000","last_peered":"2024-09-16T10:31:07.133152+0000","last_clean":"2024-09-16T10:31:07.133152+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:07.133152+0000","last_undegraded":"2024-09-16T10:31:07.133152+0000","last_fullsized":"2024-09-16T10:31:07.133152+0000","mapping_epoch":34,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16175104,"num_objects":3949,"num_object_clones":0,"num_object_copies":7898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16678","reported_seq":16932,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.053640+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:07.053640+0000","last_peered":"2024-09-16T10:31:07.053640+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:07.053640+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16322584,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12735,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-317}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16539","reported_seq":16599,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.097226+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:07.097226+0000","last_peered":"2024-09-16T10:31:07.097226+0000","last_clean":"2024-09-16T10:31:07.097226+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:07.097226+0000","last_undegraded":"2024-09-16T10:31:07.097226+0000","last_fullsized":"2024-09-16T10:31:07.097226+0000","mapping_epoch":34,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16068608,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16539,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16564","reported_seq":16724,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.825146+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:07.825146+0000","last_peered":"2024-09-16T10:31:07.825146+0000","last_clean":"2024-09-16T10:31:07.825146+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:07.825146+0000","last_undegraded":"2024-09-16T10:31:07.825146+0000","last_fullsized":"2024-09-16T10:31:07.825146+0000","mapping_epoch":33,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16142336,"num_objects":3941,"num_object_clones":0,"num_object_copies":7882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13206,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16420","reported_seq":16753,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.899890+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:07.899890+0000","last_peered":"2024-09-16T10:31:07.899890+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:07.899890+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16183296,"num_objects":3951,"num_object_clones":0,"num_object_copies":7902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13045,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16982","reported_seq":17391,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.812541+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:07.812541+0000","last_peered":"2024-09-16T10:31:07.812541+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:07.812541+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16130048,"num_objects":3938,"num_object_clones":0,"num_object_copies":7876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13224,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16464","reported_seq":16643,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.894176+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:07.894176+0000","last_peered":"2024-09-16T10:31:07.894176+0000","last_clean":"2024-09-16T10:31:07.894176+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:07.894176+0000","last_undegraded":"2024-09-16T10:31:07.894176+0000","last_fullsized":"2024-09-16T10:31:07.894176+0000","mapping_epoch":33,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":3926,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12267,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16409","reported_seq":24914,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:06.718774+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:06.718774+0000","last_peered":"2024-09-16T10:31:06.718774+0000","last_clean":"2024-09-16T10:31:06.718774+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:06.718774+0000","last_undegraded":"2024-09-16T10:31:06.718774+0000","last_fullsized":"2024-09-16T10:31:06.718774+0000","mapping_epoch":43,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16597958,"num_objects":4054,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16407,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17023","reported_seq":17300,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.900957+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:07.900957+0000","last_peered":"2024-09-16T10:31:07.900957+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:07.900957+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16454598,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17021,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-318}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16447","reported_seq":16644,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.044980+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:07.044980+0000","last_peered":"2024-09-16T10:31:07.044980+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:07.044980+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16270278,"num_objects":3974,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16445,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16565","reported_seq":23687,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.091758+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:07.091758+0000","last_peered":"2024-09-16T10:31:07.091758+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:07.091758+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430022,"num_objects":4013,"num_object_clones":0,"num_object_copies":8026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16563,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3517,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16178","reported_seq":16429,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.895963+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:07.895963+0000","last_peered":"2024-09-16T10:31:07.895963+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:07.895963+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15794659,"num_objects":3857,"num_object_clones":0,"num_object_copies":7714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16177,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16881","reported_seq":17157,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.833659+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:07.833659+0000","last_peered":"2024-09-16T10:31:07.833659+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:07.833659+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16520617,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16878,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16258","reported_seq":16490,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.812844+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:07.812844+0000","last_peered":"2024-09-16T10:31:07.812844+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:07.812844+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15901638,"num_objects":3884,"num_object_clones":0,"num_object_copies":7768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16256,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16954","reported_seq":17196,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.875886+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:07.875886+0000","last_peered":"2024-09-16T10:31:07.875886+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:07.875886+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16148364,"num_objects":3946,"num_object_clones":0,"num_object_copies":7892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16951,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":259301902,"num_objects":63323,"num_object_clones":0,"num_object_copies":126646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242982,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12598,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":729923584,"data_stored":729839567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589460,"kb_used_data":264972,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93782380,"statfs":{"total":96636764160,"available":96033157120,"internally_reserved":0,"allocated":271331328,"data_stored":271063177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":0.32500000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":3.6059999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.45400000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691928,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535084,"kb_used_data":192548,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93836756,"statfs":{"total":96636764160,"available":96088838144,"internally_reserved":0,"allocated":197169152,"data_stored":196835967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":350744234},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":4.0250000000000004},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":9.077}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.036}]}]},{"osd":0,"up_from":24,"seq":103079215211,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673196,"kb_used_data":258532,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93698644,"statfs":{"total":96636764160,"available":95947411456,"internally_reserved":0,"allocated":264736768,"data_stored":264476303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424603304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,2,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.0750000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.2729999999999997},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":263061504,"data_stored":263057891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196149248,"data_stored":196076529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":270712832,"data_stored":270705147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:09.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:09.803 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:09.806 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:09.830 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-09-16T10:31:08.721533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":259301921,"num_objects":63325,"num_object_clones":0,"num_object_copies":126650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242984,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12599,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1797740,"kb_used_data":716052,"kb_used_omap":12,"kb_used_meta":1081651,"kb_avail":281317780,"statfs":{"total":289910292480,"available":288069406720,"internally_reserved":0,"allocated":733237248,"data_stored":732375447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1107610626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,6,5,3,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27787264,"num_objects":-6784,"num_object_clones":0,"num_object_copies":-13568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6784,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002903"},"pg_stats":[{"pgid":"2.f","version":"46'16542","reported_seq":25007,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.915580+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:07.915580+0000","last_peered":"2024-09-16T10:31:07.915580+0000","last_clean":"2024-09-16T10:31:07.915580+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:07.915580+0000","last_undegraded":"2024-09-16T10:31:07.915580+0000","last_fullsized":"2024-09-16T10:31:07.915580+0000","mapping_epoch":45,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":3926,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16542,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16513","reported_seq":16674,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.133152+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:07.133152+0000","last_peered":"2024-09-16T10:31:07.133152+0000","last_clean":"2024-09-16T10:31:07.133152+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:07.133152+0000","last_undegraded":"2024-09-16T10:31:07.133152+0000","last_fullsized":"2024-09-16T10:31:07.133152+0000","mapping_epoch":34,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16175104,"num_objects":3949,"num_object_clones":0,"num_object_copies":7898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16678","reported_seq":16932,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.053640+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:07.053640+0000","last_peered":"2024-09-16T10:31:07.053640+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:07.053640+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16322584,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12735,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-317}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16539","reported_seq":16599,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.097226+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:07.097226+0000","last_peered":"2024-09-16T10:31:07.097226+0000","last_clean":"2024-09-16T10:31:07.097226+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:07.097226+0000","last_undegraded":"2024-09-16T10:31:07.097226+0000","last_fullsized":"2024-09-16T10:31:07.097226+0000","mapping_epoch":34,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16068608,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16539,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16564","reported_seq":16724,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.825146+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:07.825146+0000","last_peered":"2024-09-16T10:31:07.825146+0000","last_clean":"2024-09-16T10:31:07.825146+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:07.825146+0000","last_undegraded":"2024-09-16T10:31:07.825146+0000","last_fullsized":"2024-09-16T10:31:07.825146+0000","mapping_epoch":33,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16142336,"num_objects":3941,"num_object_clones":0,"num_object_copies":7882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13206,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16420","reported_seq":16753,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.899890+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:07.899890+0000","last_peered":"2024-09-16T10:31:07.899890+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:07.899890+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16183296,"num_objects":3951,"num_object_clones":0,"num_object_copies":7902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13045,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16982","reported_seq":17391,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.812541+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:07.812541+0000","last_peered":"2024-09-16T10:31:07.812541+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:07.812541+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16130048,"num_objects":3938,"num_object_clones":0,"num_object_copies":7876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13224,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16464","reported_seq":16643,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.894176+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:07.894176+0000","last_peered":"2024-09-16T10:31:07.894176+0000","last_clean":"2024-09-16T10:31:07.894176+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:07.894176+0000","last_undegraded":"2024-09-16T10:31:07.894176+0000","last_fullsized":"2024-09-16T10:31:07.894176+0000","mapping_epoch":33,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":3926,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12267,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16409","reported_seq":24914,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:06.718774+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:06.718774+0000","last_peered":"2024-09-16T10:31:06.718774+0000","last_clean":"2024-09-16T10:31:06.718774+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:06.718774+0000","last_undegraded":"2024-09-16T10:31:06.718774+0000","last_fullsized":"2024-09-16T10:31:06.718774+0000","mapping_epoch":43,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16597958,"num_objects":4054,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16407,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17023","reported_seq":17300,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.900957+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:07.900957+0000","last_peered":"2024-09-16T10:31:07.900957+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:07.900957+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16454598,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17021,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-318}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16447","reported_seq":16644,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.044980+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:07.044980+0000","last_peered":"2024-09-16T10:31:07.044980+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:07.044980+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16270278,"num_objects":3974,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16445,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16565","reported_seq":23687,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.091758+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:07.091758+0000","last_peered":"2024-09-16T10:31:07.091758+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:07.091758+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430022,"num_objects":4013,"num_object_clones":0,"num_object_copies":8026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16563,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3517,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16178","reported_seq":16429,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.895963+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:07.895963+0000","last_peered":"2024-09-16T10:31:07.895963+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:07.895963+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15794659,"num_objects":3857,"num_object_clones":0,"num_object_copies":7714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16177,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16881","reported_seq":17157,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.833659+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:07.833659+0000","last_peered":"2024-09-16T10:31:07.833659+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:07.833659+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16520617,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16878,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16258","reported_seq":16490,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.812844+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:07.812844+0000","last_peered":"2024-09-16T10:31:07.812844+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:07.812844+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15901638,"num_objects":3884,"num_object_clones":0,"num_object_copies":7768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16256,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16954","reported_seq":17196,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.875886+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:07.875886+0000","last_peered":"2024-09-16T10:31:07.875886+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:07.875886+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16148364,"num_objects":3946,"num_object_clones":0,"num_object_copies":7892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16951,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":259301902,"num_objects":63323,"num_object_clones":0,"num_object_copies":126646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242982,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12598,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":729923584,"data_stored":729839567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589460,"kb_used_data":264972,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93782380,"statfs":{"total":96636764160,"available":96033157120,"internally_reserved":0,"allocated":271331328,"data_stored":271063177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":0.32500000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":3.6059999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.45400000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691928,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535084,"kb_used_data":192548,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93836756,"statfs":{"total":96636764160,"available":96088838144,"internally_reserved":0,"allocated":197169152,"data_stored":196835967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":350744234},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":4.0250000000000004},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":9.077}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.036}]}]},{"osd":0,"up_from":24,"seq":103079215211,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673196,"kb_used_data":258532,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93698644,"statfs":{"total":96636764160,"available":95947411456,"internally_reserved":0,"allocated":264736768,"data_stored":264476303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424603304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,2,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.0750000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.2729999999999997},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":263061504,"data_stored":263057891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196149248,"data_stored":196076529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":270712832,"data_stored":270705147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:09.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:09.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'16678", 'reported_seq': 16932, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:07.053640+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:07.053640+0000', 'last_peered': '2024-09-16T10:31:07.053640+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:07.053640+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'6600", 'ondisk_log_start': "42'6600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16322584, 'num_objects': 3986, 'num_object_clones': 0, 'num_object_copies': 7972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3986, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12735, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -317}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:09.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:09.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'16420", 'reported_seq': 16753, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:07.899890+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:07.899890+0000', 'last_peered': '2024-09-16T10:31:07.899890+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:07.899890+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'6400", 'ondisk_log_start': "42'6400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16183296, 'num_objects': 3951, 'num_object_clones': 0, 'num_object_copies': 7902, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3951, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13045, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -183}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:09.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:09.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'16982", 'reported_seq': 17391, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:07.812541+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:07.812541+0000', 'last_peered': '2024-09-16T10:31:07.812541+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:07.812541+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'6900", 'ondisk_log_start': "42'6900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16130048, 'num_objects': 3938, 'num_object_clones': 0, 'num_object_copies': 7876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13224, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -349}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:09.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:09.832 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'17023", 'reported_seq': 17300, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:07.900957+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:07.900957+0000', 'last_peered': '2024-09-16T10:31:07.900957+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:07.900957+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'7000", 'ondisk_log_start': "42'7000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16454598, 'num_objects': 4019, 'num_object_clones': 0, 'num_object_copies': 8038, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17021, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -318}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:09.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:09.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'16447", 'reported_seq': 16644, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:07.044980+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:07.044980+0000', 'last_peered': '2024-09-16T10:31:07.044980+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:07.044980+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'6400", 'ondisk_log_start': "42'6400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16270278, 'num_objects': 3974, 'num_object_clones': 0, 'num_object_copies': 7948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16445, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -259}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:09.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:31:09.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'16565", 'reported_seq': 23687, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:07.091758+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:31:07.091758+0000', 'last_peered': '2024-09-16T10:31:07.091758+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:31:07.091758+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'6500", 'ondisk_log_start': "42'6500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16430022, 'num_objects': 4013, 'num_object_clones': 0, 'num_object_copies': 8026, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 679, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4013, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16563, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3517, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3334}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:09.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:09.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'16178", 'reported_seq': 16429, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:07.895963+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:07.895963+0000', 'last_peered': '2024-09-16T10:31:07.895963+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:07.895963+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'6100", 'ondisk_log_start': "42'6100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15794659, 'num_objects': 3857, 'num_object_clones': 0, 'num_object_copies': 7714, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16177, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:09.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:09.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'16881", 'reported_seq': 17157, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:07.833659+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:07.833659+0000', 'last_peered': '2024-09-16T10:31:07.833659+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:07.833659+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'6800", 'ondisk_log_start': "42'6800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16520617, 'num_objects': 4036, 'num_object_clones': 0, 'num_object_copies': 8072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16878, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:09.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:09.833 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'16258", 'reported_seq': 16490, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:07.812844+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:31:07.812844+0000', 'last_peered': '2024-09-16T10:31:07.812844+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:07.812844+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'6200", 'ondisk_log_start': "42'6200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15901638, 'num_objects': 3884, 'num_object_clones': 0, 'num_object_copies': 7768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16256, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -209}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:09.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:09.834 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'16954", 'reported_seq': 17196, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:07.875886+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:07.875886+0000', 'last_peered': '2024-09-16T10:31:07.875886+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:07.875886+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'6900", 'ondisk_log_start': "42'6900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16148364, 'num_objects': 3946, 'num_object_clones': 0, 'num_object_copies': 7892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3946, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16951, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -376}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:09.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:10.553 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:10.553 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 38748/126650 objects degraded (30.595%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":559,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":63325,"data_bytes":259301921,"bytes_used":1840885760,"bytes_avail":288069406720,"bytes_total":289910292480,"degraded_objects":38748,"degraded_total":126650,"degraded_ratio":0.30594551914725621,"recovering_objects_per_sec":37,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":565},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:31:10.580 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:31:10.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:11.137 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:11.162 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-16T10:31:10.722091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":259301921,"num_objects":63325,"num_object_clones":0,"num_object_copies":126650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242984,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12599,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1797740,"kb_used_data":716052,"kb_used_omap":12,"kb_used_meta":1081651,"kb_avail":281317780,"statfs":{"total":289910292480,"available":288069406720,"internally_reserved":0,"allocated":733237248,"data_stored":732375447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1107610626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,6,5,3,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18821120,"num_objects":-4595,"num_object_clones":0,"num_object_copies":-9190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4595,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002970"},"pg_stats":[{"pgid":"2.f","version":"46'16542","reported_seq":25007,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.915580+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:07.915580+0000","last_peered":"2024-09-16T10:31:07.915580+0000","last_clean":"2024-09-16T10:31:07.915580+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:07.915580+0000","last_undegraded":"2024-09-16T10:31:07.915580+0000","last_fullsized":"2024-09-16T10:31:07.915580+0000","mapping_epoch":45,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":3926,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16542,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16513","reported_seq":16674,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.133152+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:07.133152+0000","last_peered":"2024-09-16T10:31:07.133152+0000","last_clean":"2024-09-16T10:31:07.133152+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:07.133152+0000","last_undegraded":"2024-09-16T10:31:07.133152+0000","last_fullsized":"2024-09-16T10:31:07.133152+0000","mapping_epoch":34,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16175104,"num_objects":3949,"num_object_clones":0,"num_object_copies":7898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16678","reported_seq":16932,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.053640+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:07.053640+0000","last_peered":"2024-09-16T10:31:07.053640+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:07.053640+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16322584,"num_objects":3986,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12735,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-317}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16539","reported_seq":16599,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.097226+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:07.097226+0000","last_peered":"2024-09-16T10:31:07.097226+0000","last_clean":"2024-09-16T10:31:07.097226+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:07.097226+0000","last_undegraded":"2024-09-16T10:31:07.097226+0000","last_fullsized":"2024-09-16T10:31:07.097226+0000","mapping_epoch":34,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16068608,"num_objects":3923,"num_object_clones":0,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16539,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16564","reported_seq":16724,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.825146+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:07.825146+0000","last_peered":"2024-09-16T10:31:07.825146+0000","last_clean":"2024-09-16T10:31:07.825146+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:07.825146+0000","last_undegraded":"2024-09-16T10:31:07.825146+0000","last_fullsized":"2024-09-16T10:31:07.825146+0000","mapping_epoch":33,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16142336,"num_objects":3941,"num_object_clones":0,"num_object_copies":7882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13206,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16420","reported_seq":16753,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.899890+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:07.899890+0000","last_peered":"2024-09-16T10:31:07.899890+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:07.899890+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16183296,"num_objects":3951,"num_object_clones":0,"num_object_copies":7902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13045,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16982","reported_seq":17391,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.812541+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:07.812541+0000","last_peered":"2024-09-16T10:31:07.812541+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:07.812541+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16130048,"num_objects":3938,"num_object_clones":0,"num_object_copies":7876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13224,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16464","reported_seq":16643,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.894176+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:07.894176+0000","last_peered":"2024-09-16T10:31:07.894176+0000","last_clean":"2024-09-16T10:31:07.894176+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:07.894176+0000","last_undegraded":"2024-09-16T10:31:07.894176+0000","last_fullsized":"2024-09-16T10:31:07.894176+0000","mapping_epoch":33,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":3926,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12267,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16409","reported_seq":24914,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:06.718774+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:06.718774+0000","last_peered":"2024-09-16T10:31:06.718774+0000","last_clean":"2024-09-16T10:31:06.718774+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:06.718774+0000","last_undegraded":"2024-09-16T10:31:06.718774+0000","last_fullsized":"2024-09-16T10:31:06.718774+0000","mapping_epoch":43,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16597958,"num_objects":4054,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16407,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988366+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:03.988366+0000","last_peered":"2024-09-16T10:29:03.988366+0000","last_clean":"2024-09-16T10:29:03.988366+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:03.988366+0000","last_undegraded":"2024-09-16T10:29:03.988366+0000","last_fullsized":"2024-09-16T10:29:03.988366+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17023","reported_seq":17300,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.900957+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:07.900957+0000","last_peered":"2024-09-16T10:31:07.900957+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:07.900957+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16454598,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17021,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-318}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16447","reported_seq":16644,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.044980+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:07.044980+0000","last_peered":"2024-09-16T10:31:07.044980+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:07.044980+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16270278,"num_objects":3974,"num_object_clones":0,"num_object_copies":7948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16445,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-259}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16565","reported_seq":23687,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.091758+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:07.091758+0000","last_peered":"2024-09-16T10:31:07.091758+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:07.091758+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16430022,"num_objects":4013,"num_object_clones":0,"num_object_copies":8026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16563,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3517,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3334}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16178","reported_seq":16429,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.895963+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:07.895963+0000","last_peered":"2024-09-16T10:31:07.895963+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:07.895963+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15794659,"num_objects":3857,"num_object_clones":0,"num_object_copies":7714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16177,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16881","reported_seq":17157,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.833659+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:07.833659+0000","last_peered":"2024-09-16T10:31:07.833659+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:07.833659+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16520617,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16878,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16258","reported_seq":16490,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.812844+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:07.812844+0000","last_peered":"2024-09-16T10:31:07.812844+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:07.812844+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15901638,"num_objects":3884,"num_object_clones":0,"num_object_copies":7768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16256,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16954","reported_seq":17196,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.875886+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:07.875886+0000","last_peered":"2024-09-16T10:31:07.875886+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:07.875886+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16148364,"num_objects":3946,"num_object_clones":0,"num_object_copies":7892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16951,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":259301902,"num_objects":63323,"num_object_clones":0,"num_object_copies":126646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":242982,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12598,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":729923584,"data_stored":729839567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160817,"ondisk_log_size":160817,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018984,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589460,"kb_used_data":264972,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93782380,"statfs":{"total":96636764160,"available":96033157120,"internally_reserved":0,"allocated":271331328,"data_stored":271063177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":0.32500000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":3.6059999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.45400000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691928,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535084,"kb_used_data":192548,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93836756,"statfs":{"total":96636764160,"available":96088838144,"internally_reserved":0,"allocated":197169152,"data_stored":196835967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":350744234},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":4.0250000000000004},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":9.077}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.036}]}]},{"osd":0,"up_from":24,"seq":103079215211,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673196,"kb_used_data":258532,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93698644,"statfs":{"total":96636764160,"available":95947411456,"internally_reserved":0,"allocated":264736768,"data_stored":264476303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424603304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,2,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.0750000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.2729999999999997},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":263061504,"data_stored":263057891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":196149248,"data_stored":196076529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":270712832,"data_stored":270705147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:14.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:14.713 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:14.714 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:14.742 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-16T10:31:12.722524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":255808033,"num_objects":62472,"num_object_clones":0,"num_object_copies":124944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62472,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":243837,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12767,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160873,"ondisk_log_size":160873,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1794740,"kb_used_data":704856,"kb_used_omap":12,"kb_used_meta":1089843,"kb_avail":281320780,"statfs":{"total":289910292480,"available":288072478720,"internally_reserved":0,"allocated":721772544,"data_stored":720910743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1115999234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,1,9,8,6,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22315008,"num_objects":-5448,"num_object_clones":0,"num_object_copies":-10896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5448,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":486,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002908"},"pg_stats":[{"pgid":"2.f","version":"46'16542","reported_seq":25007,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.915580+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:07.915580+0000","last_peered":"2024-09-16T10:31:07.915580+0000","last_clean":"2024-09-16T10:31:07.915580+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:07.915580+0000","last_undegraded":"2024-09-16T10:31:07.915580+0000","last_fullsized":"2024-09-16T10:31:07.915580+0000","mapping_epoch":45,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":3926,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16542,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16664","reported_seq":16825,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.088182+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:12.088182+0000","last_peered":"2024-09-16T10:31:12.088182+0000","last_clean":"2024-09-16T10:31:12.088182+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:12.088182+0000","last_undegraded":"2024-09-16T10:31:12.088182+0000","last_fullsized":"2024-09-16T10:31:12.088182+0000","mapping_epoch":34,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15556608,"num_objects":3798,"num_object_clones":0,"num_object_copies":7596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12877,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16824","reported_seq":17078,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.146064+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:12.146064+0000","last_peered":"2024-09-16T10:31:12.146064+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:12.146064+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15724568,"num_objects":3840,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12881,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-463}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16685","reported_seq":16745,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.154201+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:12.154201+0000","last_peered":"2024-09-16T10:31:12.154201+0000","last_clean":"2024-09-16T10:31:12.154201+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:12.154201+0000","last_undegraded":"2024-09-16T10:31:12.154201+0000","last_fullsized":"2024-09-16T10:31:12.154201+0000","mapping_epoch":34,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15470592,"num_objects":3777,"num_object_clones":0,"num_object_copies":7554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16685,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16564","reported_seq":16724,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.825146+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:07.825146+0000","last_peered":"2024-09-16T10:31:07.825146+0000","last_clean":"2024-09-16T10:31:07.825146+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:07.825146+0000","last_undegraded":"2024-09-16T10:31:07.825146+0000","last_fullsized":"2024-09-16T10:31:07.825146+0000","mapping_epoch":33,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16142336,"num_objects":3941,"num_object_clones":0,"num_object_copies":7882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13206,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16420","reported_seq":16753,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.899890+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:07.899890+0000","last_peered":"2024-09-16T10:31:07.899890+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:07.899890+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16183296,"num_objects":3951,"num_object_clones":0,"num_object_copies":7902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13045,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-183}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'16982","reported_seq":17391,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.812541+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:07.812541+0000","last_peered":"2024-09-16T10:31:07.812541+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:07.812541+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16130048,"num_objects":3938,"num_object_clones":0,"num_object_copies":7876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13224,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-349}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16464","reported_seq":16643,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:07.894176+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:07.894176+0000","last_peered":"2024-09-16T10:31:07.894176+0000","last_clean":"2024-09-16T10:31:07.894176+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:07.894176+0000","last_undegraded":"2024-09-16T10:31:07.894176+0000","last_fullsized":"2024-09-16T10:31:07.894176+0000","mapping_epoch":33,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16080896,"num_objects":3926,"num_object_clones":0,"num_object_copies":7852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12267,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16541","reported_seq":25048,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:11.663389+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:11.663389+0000","last_peered":"2024-09-16T10:31:11.663389+0000","last_clean":"2024-09-16T10:31:11.663389+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:11.663389+0000","last_undegraded":"2024-09-16T10:31:11.663389+0000","last_fullsized":"2024-09-16T10:31:11.663389+0000","mapping_epoch":43,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16057286,"num_objects":3922,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16539,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17023","reported_seq":17300,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.900957+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:07.900957+0000","last_peered":"2024-09-16T10:31:07.900957+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:07.900957+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16454598,"num_objects":4019,"num_object_clones":0,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17021,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-318}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16583","reported_seq":16780,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.182331+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:12.182331+0000","last_peered":"2024-09-16T10:31:12.182331+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:12.182331+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15713222,"num_objects":3838,"num_object_clones":0,"num_object_copies":7676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16581,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16707","reported_seq":24165,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.148652+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:12.148652+0000","last_peered":"2024-09-16T10:31:12.148652+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:12.148652+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15848390,"num_objects":3871,"num_object_clones":0,"num_object_copies":7742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16705,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3685,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16178","reported_seq":16429,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.895963+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:07.895963+0000","last_peered":"2024-09-16T10:31:07.895963+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:07.895963+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6100","ondisk_log_start":"42'6100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15794659,"num_objects":3857,"num_object_clones":0,"num_object_copies":7714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16177,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'16881","reported_seq":17157,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.833659+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:07.833659+0000","last_peered":"2024-09-16T10:31:07.833659+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:07.833659+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16520617,"num_objects":4036,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16878,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16258","reported_seq":16490,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.812844+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:07.812844+0000","last_peered":"2024-09-16T10:31:07.812844+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:07.812844+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6200","ondisk_log_start":"42'6200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15901638,"num_objects":3884,"num_object_clones":0,"num_object_copies":7768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16256,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'16954","reported_seq":17196,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:07.875886+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:07.875886+0000","last_peered":"2024-09-16T10:31:07.875886+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:07.875886+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16148364,"num_objects":3946,"num_object_clones":0,"num_object_copies":7892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16951,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-376}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":255808014,"num_objects":62470,"num_object_clones":0,"num_object_copies":124940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":243835,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12766,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":718204928,"data_stored":718120911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160870,"ondisk_log_size":160870,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018985,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589132,"kb_used_data":260548,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93782708,"statfs":{"total":96636764160,"available":96033492992,"internally_reserved":0,"allocated":266801152,"data_stored":266533001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,0,4,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":7.3310000000000004},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.095}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.2859999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691929,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532412,"kb_used_data":185776,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93839428,"statfs":{"total":96636764160,"available":96091574272,"internally_reserved":0,"allocated":190234624,"data_stored":189901439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":354938538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,2,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.318},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.663}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.645},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.238}]}]},{"osd":0,"up_from":24,"seq":103079215211,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673196,"kb_used_data":258532,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93698644,"statfs":{"total":96636764160,"available":95947411456,"internally_reserved":0,"allocated":264736768,"data_stored":264476303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424603304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,2,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.4020000000000001,"5min":2.6899999999999999,"15min":2.77},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.726,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":2.4950000000000001,"5min":2.2989999999999999,"15min":2.1150000000000002},"min":{"1min":0.247,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":10.009,"5min":12.999000000000001,"15min":12.999000000000001},"last":4.0750000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:30:08 2024","interfaces":[{"interface":"back","average":{"1min":2.532,"5min":2.7749999999999999,"15min":1.4490000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9160000000000004,"5min":18.093,"15min":18.093},"last":4.2729999999999997},{"interface":"front","average":{"1min":2.2210000000000001,"5min":2.3620000000000001,"15min":1.6579999999999999},"min":{"1min":0.23599999999999999,"5min":0.214,"15min":0.192},"max":{"1min":7.7679999999999998,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":263061504,"data_stored":263057891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189120512,"data_stored":189047793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266022912,"data_stored":266015227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:14.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:15.279 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:15.280 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:15.313 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-16T10:31:14.723008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":249782817,"num_objects":61001,"num_object_clones":0,"num_object_copies":122002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245308,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12767,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160744,"ondisk_log_size":160744,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1791780,"kb_used_data":697800,"kb_used_omap":12,"kb_used_meta":1093939,"kb_avail":281323740,"statfs":{"total":289910292480,"available":288075509760,"internally_reserved":0,"allocated":714547200,"data_stored":713685399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1120193538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,1,6,10,12,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24870912,"num_objects":-6072,"num_object_clones":0,"num_object_copies":-12144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6072,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002923"},"pg_stats":[{"pgid":"2.f","version":"46'16681","reported_seq":25146,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.903605+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:12.903605+0000","last_peered":"2024-09-16T10:31:12.903605+0000","last_clean":"2024-09-16T10:31:12.903605+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:12.903605+0000","last_undegraded":"2024-09-16T10:31:12.903605+0000","last_fullsized":"2024-09-16T10:31:12.903605+0000","mapping_epoch":45,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15511552,"num_objects":3787,"num_object_clones":0,"num_object_copies":7574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16681,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16664","reported_seq":16825,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.088182+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:12.088182+0000","last_peered":"2024-09-16T10:31:12.088182+0000","last_clean":"2024-09-16T10:31:12.088182+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:12.088182+0000","last_undegraded":"2024-09-16T10:31:12.088182+0000","last_fullsized":"2024-09-16T10:31:12.088182+0000","mapping_epoch":34,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15556608,"num_objects":3798,"num_object_clones":0,"num_object_copies":7596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12877,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16824","reported_seq":17078,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.146064+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:12.146064+0000","last_peered":"2024-09-16T10:31:12.146064+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:12.146064+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15724568,"num_objects":3840,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12881,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-463}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16685","reported_seq":16745,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.154201+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:12.154201+0000","last_peered":"2024-09-16T10:31:12.154201+0000","last_clean":"2024-09-16T10:31:12.154201+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:12.154201+0000","last_undegraded":"2024-09-16T10:31:12.154201+0000","last_fullsized":"2024-09-16T10:31:12.154201+0000","mapping_epoch":34,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15470592,"num_objects":3777,"num_object_clones":0,"num_object_copies":7554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16685,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16707","reported_seq":16867,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.860851+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:12.860851+0000","last_peered":"2024-09-16T10:31:12.860851+0000","last_clean":"2024-09-16T10:31:12.860851+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:12.860851+0000","last_undegraded":"2024-09-16T10:31:12.860851+0000","last_fullsized":"2024-09-16T10:31:12.860851+0000","mapping_epoch":33,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15556608,"num_objects":3798,"num_object_clones":0,"num_object_copies":7596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13349,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16564","reported_seq":16888,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.771311+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:12.771311+0000","last_peered":"2024-09-16T10:31:12.771311+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:12.771311+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15593472,"num_objects":3807,"num_object_clones":0,"num_object_copies":7614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13189,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17129","reported_seq":17538,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.858051+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:12.858051+0000","last_peered":"2024-09-16T10:31:12.858051+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:12.858051+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15527936,"num_objects":3791,"num_object_clones":0,"num_object_copies":7582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13371,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16603","reported_seq":16782,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.900147+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:12.900147+0000","last_peered":"2024-09-16T10:31:12.900147+0000","last_clean":"2024-09-16T10:31:12.900147+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:12.900147+0000","last_undegraded":"2024-09-16T10:31:12.900147+0000","last_fullsized":"2024-09-16T10:31:12.900147+0000","mapping_epoch":33,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15511552,"num_objects":3787,"num_object_clones":0,"num_object_copies":7574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16541","reported_seq":25048,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:11.663389+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:11.663389+0000","last_peered":"2024-09-16T10:31:11.663389+0000","last_clean":"2024-09-16T10:31:11.663389+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:11.663389+0000","last_undegraded":"2024-09-16T10:31:11.663389+0000","last_fullsized":"2024-09-16T10:31:11.663389+0000","mapping_epoch":43,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16057286,"num_objects":3922,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16539,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17174","reported_seq":17450,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.790326+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:12.790326+0000","last_peered":"2024-09-16T10:31:12.790326+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:12.790326+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15836102,"num_objects":3868,"num_object_clones":0,"num_object_copies":7736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17172,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-469}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16583","reported_seq":16780,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.182331+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:12.182331+0000","last_peered":"2024-09-16T10:31:12.182331+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:12.182331+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15713222,"num_objects":3838,"num_object_clones":0,"num_object_copies":7676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16581,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16707","reported_seq":24165,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.148652+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:12.148652+0000","last_peered":"2024-09-16T10:31:12.148652+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:12.148652+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15848390,"num_objects":3871,"num_object_clones":0,"num_object_copies":7742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16705,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3685,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16326","reported_seq":16577,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.876775+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:12.876775+0000","last_peered":"2024-09-16T10:31:12.876775+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:12.876775+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15188451,"num_objects":3709,"num_object_clones":0,"num_object_copies":7418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16325,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17053","reported_seq":17330,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.809791+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:12.809791+0000","last_peered":"2024-09-16T10:31:12.809791+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:12.809791+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15816105,"num_objects":3864,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17050,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16412","reported_seq":16642,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.902705+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:12.902705+0000","last_peered":"2024-09-16T10:31:12.902705+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:12.902705+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15270854,"num_objects":3730,"num_object_clones":0,"num_object_copies":7460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16410,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17088","reported_seq":17331,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.920981+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:12.920981+0000","last_peered":"2024-09-16T10:31:12.920981+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:12.920981+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15599500,"num_objects":3812,"num_object_clones":0,"num_object_copies":7624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17085,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":249782798,"num_objects":60999,"num_object_clones":0,"num_object_copies":121998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245306,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12766,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":710995968,"data_stored":710911951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018985,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589132,"kb_used_data":260548,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93782708,"statfs":{"total":96636764160,"available":96033492992,"internally_reserved":0,"allocated":266801152,"data_stored":266533001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,0,4,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":7.3310000000000004},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.095}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.2859999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691929,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532412,"kb_used_data":185776,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93839428,"statfs":{"total":96636764160,"available":96091574272,"internally_reserved":0,"allocated":190234624,"data_stored":189901439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":354938538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,2,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.318},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.663}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.645},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.238}]}]},{"osd":0,"up_from":24,"seq":103079215212,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670236,"kb_used_data":251476,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93701604,"statfs":{"total":96636764160,"available":95950442496,"internally_reserved":0,"allocated":257511424,"data_stored":257250959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":428797608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.7709999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.496},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":255852544,"data_stored":255848931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189120512,"data_stored":189047793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266022912,"data_stored":266015227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:15.314 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:15.314 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'16824", 'reported_seq': 17078, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:12.146064+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:12.146064+0000', 'last_peered': '2024-09-16T10:31:12.146064+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:12.146064+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'6800", 'ondisk_log_start': "42'6800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15724568, 'num_objects': 3840, 'num_object_clones': 0, 'num_object_copies': 7680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3840, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12881, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -463}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:15.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:15.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'16564", 'reported_seq': 16888, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:12.771311+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:12.771311+0000', 'last_peered': '2024-09-16T10:31:12.771311+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:12.771311+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'6500", 'ondisk_log_start': "42'6500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15593472, 'num_objects': 3807, 'num_object_clones': 0, 'num_object_copies': 7614, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3807, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13189, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -327}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:15.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:15.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'17129", 'reported_seq': 17538, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:12.858051+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:12.858051+0000', 'last_peered': '2024-09-16T10:31:12.858051+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:12.858051+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'7100", 'ondisk_log_start': "42'7100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15527936, 'num_objects': 3791, 'num_object_clones': 0, 'num_object_copies': 7582, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13371, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -496}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:15.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:15.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'17174", 'reported_seq': 17450, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:12.790326+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:12.790326+0000', 'last_peered': '2024-09-16T10:31:12.790326+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:12.790326+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'7100", 'ondisk_log_start': "42'7100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15836102, 'num_objects': 3868, 'num_object_clones': 0, 'num_object_copies': 7736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17172, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -469}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:15.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:15.315 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'16583", 'reported_seq': 16780, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:12.182331+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:12.182331+0000', 'last_peered': '2024-09-16T10:31:12.182331+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:12.182331+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'6500", 'ondisk_log_start': "42'6500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15713222, 'num_objects': 3838, 'num_object_clones': 0, 'num_object_copies': 7676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3838, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16581, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -395}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:15.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:31:15.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'16707", 'reported_seq': 24165, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:12.148652+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:31:12.148652+0000', 'last_peered': '2024-09-16T10:31:12.148652+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:31:12.148652+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'6700", 'ondisk_log_start': "42'6700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15848390, 'num_objects': 3871, 'num_object_clones': 0, 'num_object_copies': 7742, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 511, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16705, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3685, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3360}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:15.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:15.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'16326", 'reported_seq': 16577, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:12.876775+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:12.876775+0000', 'last_peered': '2024-09-16T10:31:12.876775+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:12.876775+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'6300", 'ondisk_log_start': "42'6300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15188451, 'num_objects': 3709, 'num_object_clones': 0, 'num_object_copies': 7418, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16325, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -366}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:15.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:15.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'17053", 'reported_seq': 17330, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:12.809791+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:12.809791+0000', 'last_peered': '2024-09-16T10:31:12.809791+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:12.809791+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'7000", 'ondisk_log_start': "42'7000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15816105, 'num_objects': 3864, 'num_object_clones': 0, 'num_object_copies': 7728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17050, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:15.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:15.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'16412", 'reported_seq': 16642, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:12.902705+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:31:12.902705+0000', 'last_peered': '2024-09-16T10:31:12.902705+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:12.902705+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'6400", 'ondisk_log_start': "42'6400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15270854, 'num_objects': 3730, 'num_object_clones': 0, 'num_object_copies': 7460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16410, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -363}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:15.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:15.316 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'17088", 'reported_seq': 17331, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:12.920981+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:12.920981+0000', 'last_peered': '2024-09-16T10:31:12.920981+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:12.920981+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'7000", 'ondisk_log_start': "42'7000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15599500, 'num_objects': 3812, 'num_object_clones': 0, 'num_object_copies': 7624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17085, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -510}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:15.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:15.993 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:15.993 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 38580/122002 objects degraded (31.622%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":564,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":61001,"data_bytes":249782817,"bytes_used":1834782720,"bytes_avail":288075509760,"bytes_total":289910292480,"degraded_objects":38580,"degraded_total":122002,"degraded_ratio":0.31622432419140672,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":317,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":505},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:31:16.021 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:31:16.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:16.626 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:16.626 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:16.647 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-16T10:31:14.723008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":249782817,"num_objects":61001,"num_object_clones":0,"num_object_copies":122002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245308,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12767,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160744,"ondisk_log_size":160744,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1791780,"kb_used_data":697800,"kb_used_omap":12,"kb_used_meta":1093939,"kb_avail":281323740,"statfs":{"total":289910292480,"available":288075509760,"internally_reserved":0,"allocated":714547200,"data_stored":713685399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1120193538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,1,6,10,12,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24870912,"num_objects":-6072,"num_object_clones":0,"num_object_copies":-12144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6072,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002923"},"pg_stats":[{"pgid":"2.f","version":"46'16681","reported_seq":25146,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.903605+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:12.903605+0000","last_peered":"2024-09-16T10:31:12.903605+0000","last_clean":"2024-09-16T10:31:12.903605+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:12.903605+0000","last_undegraded":"2024-09-16T10:31:12.903605+0000","last_fullsized":"2024-09-16T10:31:12.903605+0000","mapping_epoch":45,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15511552,"num_objects":3787,"num_object_clones":0,"num_object_copies":7574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16681,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16664","reported_seq":16825,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.088182+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:12.088182+0000","last_peered":"2024-09-16T10:31:12.088182+0000","last_clean":"2024-09-16T10:31:12.088182+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:12.088182+0000","last_undegraded":"2024-09-16T10:31:12.088182+0000","last_fullsized":"2024-09-16T10:31:12.088182+0000","mapping_epoch":34,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15556608,"num_objects":3798,"num_object_clones":0,"num_object_copies":7596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12877,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16824","reported_seq":17078,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.146064+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:12.146064+0000","last_peered":"2024-09-16T10:31:12.146064+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:12.146064+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15724568,"num_objects":3840,"num_object_clones":0,"num_object_copies":7680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3840,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12881,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-463}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16685","reported_seq":16745,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.154201+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:12.154201+0000","last_peered":"2024-09-16T10:31:12.154201+0000","last_clean":"2024-09-16T10:31:12.154201+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:12.154201+0000","last_undegraded":"2024-09-16T10:31:12.154201+0000","last_fullsized":"2024-09-16T10:31:12.154201+0000","mapping_epoch":34,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15470592,"num_objects":3777,"num_object_clones":0,"num_object_copies":7554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16685,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16707","reported_seq":16867,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.860851+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:12.860851+0000","last_peered":"2024-09-16T10:31:12.860851+0000","last_clean":"2024-09-16T10:31:12.860851+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:12.860851+0000","last_undegraded":"2024-09-16T10:31:12.860851+0000","last_fullsized":"2024-09-16T10:31:12.860851+0000","mapping_epoch":33,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15556608,"num_objects":3798,"num_object_clones":0,"num_object_copies":7596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13349,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16564","reported_seq":16888,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.771311+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:12.771311+0000","last_peered":"2024-09-16T10:31:12.771311+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:12.771311+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15593472,"num_objects":3807,"num_object_clones":0,"num_object_copies":7614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13189,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17129","reported_seq":17538,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.858051+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:12.858051+0000","last_peered":"2024-09-16T10:31:12.858051+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:12.858051+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15527936,"num_objects":3791,"num_object_clones":0,"num_object_copies":7582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13371,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-496}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16603","reported_seq":16782,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:12.900147+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:12.900147+0000","last_peered":"2024-09-16T10:31:12.900147+0000","last_clean":"2024-09-16T10:31:12.900147+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:12.900147+0000","last_undegraded":"2024-09-16T10:31:12.900147+0000","last_fullsized":"2024-09-16T10:31:12.900147+0000","mapping_epoch":33,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15511552,"num_objects":3787,"num_object_clones":0,"num_object_copies":7574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12406,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16541","reported_seq":25048,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:11.663389+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:11.663389+0000","last_peered":"2024-09-16T10:31:11.663389+0000","last_clean":"2024-09-16T10:31:11.663389+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:11.663389+0000","last_undegraded":"2024-09-16T10:31:11.663389+0000","last_fullsized":"2024-09-16T10:31:11.663389+0000","mapping_epoch":43,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16057286,"num_objects":3922,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16539,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17174","reported_seq":17450,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.790326+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:12.790326+0000","last_peered":"2024-09-16T10:31:12.790326+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:12.790326+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15836102,"num_objects":3868,"num_object_clones":0,"num_object_copies":7736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17172,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-469}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16583","reported_seq":16780,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.182331+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:12.182331+0000","last_peered":"2024-09-16T10:31:12.182331+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:12.182331+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15713222,"num_objects":3838,"num_object_clones":0,"num_object_copies":7676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16581,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-395}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16707","reported_seq":24165,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.148652+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:12.148652+0000","last_peered":"2024-09-16T10:31:12.148652+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:12.148652+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15848390,"num_objects":3871,"num_object_clones":0,"num_object_copies":7742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":511,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16705,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3685,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3360}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16326","reported_seq":16577,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.876775+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:12.876775+0000","last_peered":"2024-09-16T10:31:12.876775+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:12.876775+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6300","ondisk_log_start":"42'6300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15188451,"num_objects":3709,"num_object_clones":0,"num_object_copies":7418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16325,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-366}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17053","reported_seq":17330,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.809791+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:12.809791+0000","last_peered":"2024-09-16T10:31:12.809791+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:12.809791+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15816105,"num_objects":3864,"num_object_clones":0,"num_object_copies":7728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17050,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":60,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.988851+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:03.988851+0000","last_peered":"2024-09-16T10:29:03.988851+0000","last_clean":"2024-09-16T10:29:03.988851+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:03.988851+0000","last_undegraded":"2024-09-16T10:29:03.988851+0000","last_fullsized":"2024-09-16T10:29:03.988851+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16412","reported_seq":16642,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.902705+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:12.902705+0000","last_peered":"2024-09-16T10:31:12.902705+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:12.902705+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15270854,"num_objects":3730,"num_object_clones":0,"num_object_copies":7460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16410,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-363}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17088","reported_seq":17331,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:12.920981+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:12.920981+0000","last_peered":"2024-09-16T10:31:12.920981+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:12.920981+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15599500,"num_objects":3812,"num_object_clones":0,"num_object_copies":7624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17085,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":249782798,"num_objects":60999,"num_object_clones":0,"num_object_copies":121998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38580,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":245306,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12766,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":710995968,"data_stored":710911951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018985,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589132,"kb_used_data":260548,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93782708,"statfs":{"total":96636764160,"available":96033492992,"internally_reserved":0,"allocated":266801152,"data_stored":266533001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,0,4,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":7.3310000000000004},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.095}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.2859999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691929,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532412,"kb_used_data":185776,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93839428,"statfs":{"total":96636764160,"available":96091574272,"internally_reserved":0,"allocated":190234624,"data_stored":189901439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":354938538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,2,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":4.2549999999999999,"5min":3.4060000000000001,"15min":3.5350000000000001},"min":{"1min":0.30199999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.093,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.318},{"interface":"front","average":{"1min":3.3479999999999999,"5min":3.1920000000000002,"15min":2.6680000000000001},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":12.132,"5min":16.850999999999999,"15min":16.850999999999999},"last":1.663}]},{"osd":2,"last update":"Mon Sep 16 10:30:12 2024","interfaces":[{"interface":"back","average":{"1min":2.4740000000000002,"5min":2.8700000000000001,"15min":2.5019999999999998},"min":{"1min":0.28000000000000003,"5min":0.27700000000000002,"15min":0.26200000000000001},"max":{"1min":5.968,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.645},{"interface":"front","average":{"1min":3.9489999999999998,"5min":3.056,"15min":2.9470000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":11.074,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.238}]}]},{"osd":0,"up_from":24,"seq":103079215212,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670236,"kb_used_data":251476,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93701604,"statfs":{"total":96636764160,"available":95950442496,"internally_reserved":0,"allocated":257511424,"data_stored":257250959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":428797608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,4,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.7709999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.496},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":255852544,"data_stored":255848931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189120512,"data_stored":189047793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266022912,"data_stored":266015227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:19.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:20.211 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:20.213 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:20.233 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-09-16T10:31:18.723936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":240525857,"num_objects":58741,"num_object_clones":0,"num_object_copies":117482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247568,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12927,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1740844,"kb_used_data":679624,"kb_used_omap":12,"kb_used_meta":1061171,"kb_avail":281374676,"statfs":{"total":289910292480,"available":288127668224,"internally_reserved":0,"allocated":695934976,"data_stored":695073175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1086639106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,4,0,6,12,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28147712,"num_objects":-6872,"num_object_clones":0,"num_object_copies":-13744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002825"},"pg_stats":[{"pgid":"2.f","version":"46'16824","reported_seq":25289,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.918024+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:17.918024+0000","last_peered":"2024-09-16T10:31:17.918024+0000","last_clean":"2024-09-16T10:31:17.918024+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:17.918024+0000","last_undegraded":"2024-09-16T10:31:17.918024+0000","last_fullsized":"2024-09-16T10:31:17.918024+0000","mapping_epoch":45,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14925824,"num_objects":3644,"num_object_clones":0,"num_object_copies":7288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16824,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16841","reported_seq":17002,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.054403+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:17.054403+0000","last_peered":"2024-09-16T10:31:17.054403+0000","last_clean":"2024-09-16T10:31:17.054403+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:17.054403+0000","last_undegraded":"2024-09-16T10:31:17.054403+0000","last_fullsized":"2024-09-16T10:31:17.054403+0000","mapping_epoch":34,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14831616,"num_objects":3621,"num_object_clones":0,"num_object_copies":7242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13054,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16939","reported_seq":17193,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.083965+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:17.083965+0000","last_peered":"2024-09-16T10:31:17.083965+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:17.083965+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15253528,"num_objects":3725,"num_object_clones":0,"num_object_copies":7450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12996,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-578}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16809","reported_seq":16869,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.184797+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:17.184797+0000","last_peered":"2024-09-16T10:31:17.184797+0000","last_clean":"2024-09-16T10:31:17.184797+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:17.184797+0000","last_undegraded":"2024-09-16T10:31:17.184797+0000","last_fullsized":"2024-09-16T10:31:17.184797+0000","mapping_epoch":34,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14962688,"num_objects":3653,"num_object_clones":0,"num_object_copies":7306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16809,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16843","reported_seq":17003,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.891418+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:17.891418+0000","last_peered":"2024-09-16T10:31:17.891418+0000","last_clean":"2024-09-16T10:31:17.891418+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:17.891418+0000","last_undegraded":"2024-09-16T10:31:17.891418+0000","last_fullsized":"2024-09-16T10:31:17.891418+0000","mapping_epoch":33,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14999552,"num_objects":3662,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13485,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16697","reported_seq":17030,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.889605+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:17.889605+0000","last_peered":"2024-09-16T10:31:17.889605+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:17.889605+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15048704,"num_objects":3674,"num_object_clones":0,"num_object_copies":7348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13322,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-460}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17271","reported_seq":17680,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.893386+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:17.893386+0000","last_peered":"2024-09-16T10:31:17.893386+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:17.893386+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14946304,"num_objects":3649,"num_object_clones":0,"num_object_copies":7298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13513,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16747","reported_seq":16926,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.804536+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:17.804536+0000","last_peered":"2024-09-16T10:31:17.804536+0000","last_clean":"2024-09-16T10:31:17.804536+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:17.804536+0000","last_undegraded":"2024-09-16T10:31:17.804536+0000","last_fullsized":"2024-09-16T10:31:17.804536+0000","mapping_epoch":33,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14921728,"num_objects":3643,"num_object_clones":0,"num_object_copies":7286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12550,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16690","reported_seq":25197,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:16.860508+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:16.860508+0000","last_peered":"2024-09-16T10:31:16.860508+0000","last_clean":"2024-09-16T10:31:16.860508+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:16.860508+0000","last_undegraded":"2024-09-16T10:31:16.860508+0000","last_fullsized":"2024-09-16T10:31:16.860508+0000","mapping_epoch":43,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15446982,"num_objects":3773,"num_object_clones":0,"num_object_copies":7546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16688,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17318","reported_seq":17595,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.837869+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:17.837869+0000","last_peered":"2024-09-16T10:31:17.837869+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:17.837869+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15246278,"num_objects":3724,"num_object_clones":0,"num_object_copies":7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17316,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-613}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16714","reported_seq":16912,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.168686+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:17.168686+0000","last_peered":"2024-09-16T10:31:17.168686+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:17.168686+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15176646,"num_objects":3707,"num_object_clones":0,"num_object_copies":7414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16712,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-526}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16849","reported_seq":24627,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.168580+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:17.168580+0000","last_peered":"2024-09-16T10:31:17.168580+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:17.168580+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15266758,"num_objects":3729,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16847,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3845,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16465","reported_seq":16716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.917904+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:17.917904+0000","last_peered":"2024-09-16T10:31:17.917904+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:17.917904+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14619107,"num_objects":3570,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16464,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17207","reported_seq":17482,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.903266+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:17.903266+0000","last_peered":"2024-09-16T10:31:17.903266+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:17.903266+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15185321,"num_objects":3710,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17204,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.559321+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:11.559321+0000","last_peered":"2024-09-16T10:29:11.559321+0000","last_clean":"2024-09-16T10:29:11.559321+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:11.559321+0000","last_undegraded":"2024-09-16T10:29:11.559321+0000","last_fullsized":"2024-09-16T10:29:11.559321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16550","reported_seq":16782,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.930965+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:17.930965+0000","last_peered":"2024-09-16T10:31:17.930965+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:17.930965+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14705606,"num_objects":3592,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16548,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17237","reported_seq":17479,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.889532+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:17.889532+0000","last_peered":"2024-09-16T10:31:17.889532+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:17.889532+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14989196,"num_objects":3663,"num_object_clones":0,"num_object_copies":7326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17234,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-659}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":240525838,"num_objects":58739,"num_object_clones":0,"num_object_copies":117478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58739,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247566,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12926,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":692441088,"data_stored":692357071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018986,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572296,"kb_used_data":256000,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93799544,"statfs":{"total":96636764160,"available":96050733056,"internally_reserved":0,"allocated":262144000,"data_stored":261875849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":323874480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,0,4,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":7.3310000000000004},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.095}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.2859999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691930,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517436,"kb_used_data":178996,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93854404,"statfs":{"total":96636764160,"available":96106909696,"internally_reserved":0,"allocated":183291904,"data_stored":182958719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":346549930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.39000000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215213,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651112,"kb_used_data":244628,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93720728,"statfs":{"total":96636764160,"available":95970025472,"internally_reserved":0,"allocated":250499072,"data_stored":250238607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,3,0,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":2.464}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":4.9580000000000002},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":248852480,"data_stored":248848867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182067200,"data_stored":181994481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261521408,"data_stored":261513723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:20.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:20.835 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:20.838 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:20.856 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-16T10:31:20.724475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":240525857,"num_objects":58741,"num_object_clones":0,"num_object_copies":117482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247568,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12927,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1740844,"kb_used_data":679624,"kb_used_omap":12,"kb_used_meta":1061171,"kb_avail":281374676,"statfs":{"total":289910292480,"available":288127668224,"internally_reserved":0,"allocated":695934976,"data_stored":695073175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1086639106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,4,0,6,12,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18776064,"num_objects":-4584,"num_object_clones":0,"num_object_copies":-9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4584,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002942"},"pg_stats":[{"pgid":"2.f","version":"46'16824","reported_seq":25289,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.918024+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:17.918024+0000","last_peered":"2024-09-16T10:31:17.918024+0000","last_clean":"2024-09-16T10:31:17.918024+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:17.918024+0000","last_undegraded":"2024-09-16T10:31:17.918024+0000","last_fullsized":"2024-09-16T10:31:17.918024+0000","mapping_epoch":45,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14925824,"num_objects":3644,"num_object_clones":0,"num_object_copies":7288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16824,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16841","reported_seq":17002,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.054403+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:17.054403+0000","last_peered":"2024-09-16T10:31:17.054403+0000","last_clean":"2024-09-16T10:31:17.054403+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:17.054403+0000","last_undegraded":"2024-09-16T10:31:17.054403+0000","last_fullsized":"2024-09-16T10:31:17.054403+0000","mapping_epoch":34,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14831616,"num_objects":3621,"num_object_clones":0,"num_object_copies":7242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13054,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16939","reported_seq":17193,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.083965+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:17.083965+0000","last_peered":"2024-09-16T10:31:17.083965+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:17.083965+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15253528,"num_objects":3725,"num_object_clones":0,"num_object_copies":7450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12996,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-578}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16809","reported_seq":16869,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.184797+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:17.184797+0000","last_peered":"2024-09-16T10:31:17.184797+0000","last_clean":"2024-09-16T10:31:17.184797+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:17.184797+0000","last_undegraded":"2024-09-16T10:31:17.184797+0000","last_fullsized":"2024-09-16T10:31:17.184797+0000","mapping_epoch":34,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14962688,"num_objects":3653,"num_object_clones":0,"num_object_copies":7306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16809,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16843","reported_seq":17003,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.891418+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:17.891418+0000","last_peered":"2024-09-16T10:31:17.891418+0000","last_clean":"2024-09-16T10:31:17.891418+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:17.891418+0000","last_undegraded":"2024-09-16T10:31:17.891418+0000","last_fullsized":"2024-09-16T10:31:17.891418+0000","mapping_epoch":33,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14999552,"num_objects":3662,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13485,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16697","reported_seq":17030,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.889605+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:17.889605+0000","last_peered":"2024-09-16T10:31:17.889605+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:17.889605+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15048704,"num_objects":3674,"num_object_clones":0,"num_object_copies":7348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13322,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-460}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17271","reported_seq":17680,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.893386+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:17.893386+0000","last_peered":"2024-09-16T10:31:17.893386+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:17.893386+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14946304,"num_objects":3649,"num_object_clones":0,"num_object_copies":7298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13513,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16747","reported_seq":16926,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.804536+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:17.804536+0000","last_peered":"2024-09-16T10:31:17.804536+0000","last_clean":"2024-09-16T10:31:17.804536+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:17.804536+0000","last_undegraded":"2024-09-16T10:31:17.804536+0000","last_fullsized":"2024-09-16T10:31:17.804536+0000","mapping_epoch":33,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14921728,"num_objects":3643,"num_object_clones":0,"num_object_copies":7286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12550,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16690","reported_seq":25197,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:16.860508+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:16.860508+0000","last_peered":"2024-09-16T10:31:16.860508+0000","last_clean":"2024-09-16T10:31:16.860508+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:16.860508+0000","last_undegraded":"2024-09-16T10:31:16.860508+0000","last_fullsized":"2024-09-16T10:31:16.860508+0000","mapping_epoch":43,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15446982,"num_objects":3773,"num_object_clones":0,"num_object_copies":7546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16688,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17318","reported_seq":17595,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.837869+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:17.837869+0000","last_peered":"2024-09-16T10:31:17.837869+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:17.837869+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15246278,"num_objects":3724,"num_object_clones":0,"num_object_copies":7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17316,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-613}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16714","reported_seq":16912,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.168686+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:17.168686+0000","last_peered":"2024-09-16T10:31:17.168686+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:17.168686+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15176646,"num_objects":3707,"num_object_clones":0,"num_object_copies":7414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16712,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-526}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16849","reported_seq":24627,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.168580+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:17.168580+0000","last_peered":"2024-09-16T10:31:17.168580+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:17.168580+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15266758,"num_objects":3729,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16847,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3845,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16465","reported_seq":16716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.917904+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:17.917904+0000","last_peered":"2024-09-16T10:31:17.917904+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:17.917904+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14619107,"num_objects":3570,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16464,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17207","reported_seq":17482,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.903266+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:17.903266+0000","last_peered":"2024-09-16T10:31:17.903266+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:17.903266+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15185321,"num_objects":3710,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17204,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.559321+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:11.559321+0000","last_peered":"2024-09-16T10:29:11.559321+0000","last_clean":"2024-09-16T10:29:11.559321+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:11.559321+0000","last_undegraded":"2024-09-16T10:29:11.559321+0000","last_fullsized":"2024-09-16T10:29:11.559321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16550","reported_seq":16782,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.930965+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:17.930965+0000","last_peered":"2024-09-16T10:31:17.930965+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:17.930965+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14705606,"num_objects":3592,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16548,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17237","reported_seq":17479,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.889532+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:17.889532+0000","last_peered":"2024-09-16T10:31:17.889532+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:17.889532+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14989196,"num_objects":3663,"num_object_clones":0,"num_object_copies":7326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17234,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-659}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":240525838,"num_objects":58739,"num_object_clones":0,"num_object_copies":117478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58739,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247566,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12926,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":692441088,"data_stored":692357071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018986,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572296,"kb_used_data":256000,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93799544,"statfs":{"total":96636764160,"available":96050733056,"internally_reserved":0,"allocated":262144000,"data_stored":261875849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":323874480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,0,4,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":7.3310000000000004},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.095}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.2859999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691930,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517436,"kb_used_data":178996,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93854404,"statfs":{"total":96636764160,"available":96106909696,"internally_reserved":0,"allocated":183291904,"data_stored":182958719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":346549930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.39000000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215213,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651112,"kb_used_data":244628,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93720728,"statfs":{"total":96636764160,"available":95970025472,"internally_reserved":0,"allocated":250499072,"data_stored":250238607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,3,0,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":2.464}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":4.9580000000000002},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":248852480,"data_stored":248848867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182067200,"data_stored":181994481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261521408,"data_stored":261513723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:20.858 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:20.858 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'16939", 'reported_seq': 17193, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:17.083965+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:17.083965+0000', 'last_peered': '2024-09-16T10:31:17.083965+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:17.083965+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'6900", 'ondisk_log_start': "42'6900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15253528, 'num_objects': 3725, 'num_object_clones': 0, 'num_object_copies': 7450, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3725, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12996, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -578}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:20.858 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:20.858 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'16697", 'reported_seq': 17030, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:17.889605+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:17.889605+0000', 'last_peered': '2024-09-16T10:31:17.889605+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:17.889605+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'6600", 'ondisk_log_start': "42'6600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15048704, 'num_objects': 3674, 'num_object_clones': 0, 'num_object_copies': 7348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13322, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -460}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:20.858 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:20.858 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'17271", 'reported_seq': 17680, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:17.893386+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:17.893386+0000', 'last_peered': '2024-09-16T10:31:17.893386+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:17.893386+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'7200", 'ondisk_log_start': "42'7200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14946304, 'num_objects': 3649, 'num_object_clones': 0, 'num_object_copies': 7298, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13513, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -638}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:20.859 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:20.859 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'17318", 'reported_seq': 17595, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:17.837869+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:17.837869+0000', 'last_peered': '2024-09-16T10:31:17.837869+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:17.837869+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'7300", 'ondisk_log_start': "42'7300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15246278, 'num_objects': 3724, 'num_object_clones': 0, 'num_object_copies': 7448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17316, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -613}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:20.859 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:20.859 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'16714", 'reported_seq': 16912, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:17.168686+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:17.168686+0000', 'last_peered': '2024-09-16T10:31:17.168686+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:17.168686+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'6700", 'ondisk_log_start': "42'6700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15176646, 'num_objects': 3707, 'num_object_clones': 0, 'num_object_copies': 7414, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16712, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -526}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:20.859 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:31:20.859 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'16849", 'reported_seq': 24627, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:17.168580+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:31:17.168580+0000', 'last_peered': '2024-09-16T10:31:17.168580+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:31:17.168580+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'6800", 'ondisk_log_start': "42'6800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15266758, 'num_objects': 3729, 'num_object_clones': 0, 'num_object_copies': 7458, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 351, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16847, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3845, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3378}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:20.859 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:20.859 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'16465", 'reported_seq': 16716, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:17.917904+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:17.917904+0000', 'last_peered': '2024-09-16T10:31:17.917904+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:17.917904+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'6400", 'ondisk_log_start': "42'6400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14619107, 'num_objects': 3570, 'num_object_clones': 0, 'num_object_copies': 7140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16464, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -505}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:20.859 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:20.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'17207", 'reported_seq': 17482, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:17.903266+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:17.903266+0000', 'last_peered': '2024-09-16T10:31:17.903266+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:17.903266+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'7200", 'ondisk_log_start': "42'7200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15185321, 'num_objects': 3710, 'num_object_clones': 0, 'num_object_copies': 7420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17204, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -557}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:20.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:20.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'16550", 'reported_seq': 16782, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:17.930965+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:31:17.930965+0000', 'last_peered': '2024-09-16T10:31:17.930965+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:17.930965+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'6500", 'ondisk_log_start': "42'6500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14705606, 'num_objects': 3592, 'num_object_clones': 0, 'num_object_copies': 7184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3592, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16548, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:20.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:20.860 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'17237", 'reported_seq': 17479, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:17.889532+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:17.889532+0000', 'last_peered': '2024-09-16T10:31:17.889532+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:17.889532+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'7200", 'ondisk_log_start': "42'7200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14989196, 'num_objects': 3663, 'num_object_clones': 0, 'num_object_copies': 7326, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3663, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17234, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -659}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:20.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:21.542 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:21.543 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 38420/117482 objects degraded (32.703%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":570,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":58741,"data_bytes":240525857,"bytes_used":1782624256,"bytes_avail":288127668224,"bytes_total":289910292480,"degraded_objects":38420,"degraded_total":117482,"degraded_ratio":0.32702882143647538,"recovering_objects_per_sec":27,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":328,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":381},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:31:21.567 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:31:21.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:22.139 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:22.142 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:22.168 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-16T10:31:20.724475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":240525857,"num_objects":58741,"num_object_clones":0,"num_object_copies":117482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58741,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247568,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12927,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1740844,"kb_used_data":679624,"kb_used_omap":12,"kb_used_meta":1061171,"kb_avail":281374676,"statfs":{"total":289910292480,"available":288127668224,"internally_reserved":0,"allocated":695934976,"data_stored":695073175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1086639106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,4,0,6,12,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18776064,"num_objects":-4584,"num_object_clones":0,"num_object_copies":-9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4584,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002942"},"pg_stats":[{"pgid":"2.f","version":"46'16824","reported_seq":25289,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.918024+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:17.918024+0000","last_peered":"2024-09-16T10:31:17.918024+0000","last_clean":"2024-09-16T10:31:17.918024+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:17.918024+0000","last_undegraded":"2024-09-16T10:31:17.918024+0000","last_fullsized":"2024-09-16T10:31:17.918024+0000","mapping_epoch":45,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14925824,"num_objects":3644,"num_object_clones":0,"num_object_copies":7288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16824,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16841","reported_seq":17002,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.054403+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:17.054403+0000","last_peered":"2024-09-16T10:31:17.054403+0000","last_clean":"2024-09-16T10:31:17.054403+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:17.054403+0000","last_undegraded":"2024-09-16T10:31:17.054403+0000","last_fullsized":"2024-09-16T10:31:17.054403+0000","mapping_epoch":34,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14831616,"num_objects":3621,"num_object_clones":0,"num_object_copies":7242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13054,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'16939","reported_seq":17193,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.083965+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:17.083965+0000","last_peered":"2024-09-16T10:31:17.083965+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:17.083965+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15253528,"num_objects":3725,"num_object_clones":0,"num_object_copies":7450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12996,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-578}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16809","reported_seq":16869,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.184797+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:17.184797+0000","last_peered":"2024-09-16T10:31:17.184797+0000","last_clean":"2024-09-16T10:31:17.184797+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:17.184797+0000","last_undegraded":"2024-09-16T10:31:17.184797+0000","last_fullsized":"2024-09-16T10:31:17.184797+0000","mapping_epoch":34,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14962688,"num_objects":3653,"num_object_clones":0,"num_object_copies":7306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16809,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16843","reported_seq":17003,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.891418+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:17.891418+0000","last_peered":"2024-09-16T10:31:17.891418+0000","last_clean":"2024-09-16T10:31:17.891418+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:17.891418+0000","last_undegraded":"2024-09-16T10:31:17.891418+0000","last_fullsized":"2024-09-16T10:31:17.891418+0000","mapping_epoch":33,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14999552,"num_objects":3662,"num_object_clones":0,"num_object_copies":7324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13485,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16697","reported_seq":17030,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.889605+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:17.889605+0000","last_peered":"2024-09-16T10:31:17.889605+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:17.889605+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15048704,"num_objects":3674,"num_object_clones":0,"num_object_copies":7348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13322,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-460}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17271","reported_seq":17680,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.893386+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:17.893386+0000","last_peered":"2024-09-16T10:31:17.893386+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:17.893386+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14946304,"num_objects":3649,"num_object_clones":0,"num_object_copies":7298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13513,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16747","reported_seq":16926,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:17.804536+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:17.804536+0000","last_peered":"2024-09-16T10:31:17.804536+0000","last_clean":"2024-09-16T10:31:17.804536+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:17.804536+0000","last_undegraded":"2024-09-16T10:31:17.804536+0000","last_fullsized":"2024-09-16T10:31:17.804536+0000","mapping_epoch":33,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14921728,"num_objects":3643,"num_object_clones":0,"num_object_copies":7286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12550,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16690","reported_seq":25197,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:16.860508+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:16.860508+0000","last_peered":"2024-09-16T10:31:16.860508+0000","last_clean":"2024-09-16T10:31:16.860508+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:16.860508+0000","last_undegraded":"2024-09-16T10:31:16.860508+0000","last_fullsized":"2024-09-16T10:31:16.860508+0000","mapping_epoch":43,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15446982,"num_objects":3773,"num_object_clones":0,"num_object_copies":7546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16688,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17318","reported_seq":17595,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.837869+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:17.837869+0000","last_peered":"2024-09-16T10:31:17.837869+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:17.837869+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15246278,"num_objects":3724,"num_object_clones":0,"num_object_copies":7448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17316,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-613}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16714","reported_seq":16912,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.168686+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:17.168686+0000","last_peered":"2024-09-16T10:31:17.168686+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:17.168686+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15176646,"num_objects":3707,"num_object_clones":0,"num_object_copies":7414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16712,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-526}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'16849","reported_seq":24627,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.168580+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:17.168580+0000","last_peered":"2024-09-16T10:31:17.168580+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:17.168580+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15266758,"num_objects":3729,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16847,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3845,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3378}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16465","reported_seq":16716,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.917904+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:17.917904+0000","last_peered":"2024-09-16T10:31:17.917904+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:17.917904+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6400","ondisk_log_start":"42'6400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14619107,"num_objects":3570,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16464,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-505}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17207","reported_seq":17482,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.903266+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:17.903266+0000","last_peered":"2024-09-16T10:31:17.903266+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:17.903266+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15185321,"num_objects":3710,"num_object_clones":0,"num_object_copies":7420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17204,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-557}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.559321+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:11.559321+0000","last_peered":"2024-09-16T10:29:11.559321+0000","last_clean":"2024-09-16T10:29:11.559321+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:11.559321+0000","last_undegraded":"2024-09-16T10:29:11.559321+0000","last_fullsized":"2024-09-16T10:29:11.559321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16550","reported_seq":16782,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.930965+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:17.930965+0000","last_peered":"2024-09-16T10:31:17.930965+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:17.930965+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6500","ondisk_log_start":"42'6500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14705606,"num_objects":3592,"num_object_clones":0,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16548,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17237","reported_seq":17479,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:17.889532+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:17.889532+0000","last_peered":"2024-09-16T10:31:17.889532+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:17.889532+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14989196,"num_objects":3663,"num_object_clones":0,"num_object_copies":7326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17234,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-659}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":240525838,"num_objects":58739,"num_object_clones":0,"num_object_copies":117478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38420,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58739,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":247566,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12926,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":692441088,"data_stored":692357071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018986,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572296,"kb_used_data":256000,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93799544,"statfs":{"total":96636764160,"available":96050733056,"internally_reserved":0,"allocated":262144000,"data_stored":261875849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":323874480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,0,4,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":7.3310000000000004},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.095}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.2859999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691930,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517436,"kb_used_data":178996,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93854404,"statfs":{"total":96636764160,"available":96106909696,"internally_reserved":0,"allocated":183291904,"data_stored":182958719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":346549930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.39000000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215213,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":651112,"kb_used_data":244628,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93720728,"statfs":{"total":96636764160,"available":95970025472,"internally_reserved":0,"allocated":250499072,"data_stored":250238607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,3,0,0,4,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":2.464}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":4.9580000000000002},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":248852480,"data_stored":248848867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":182067200,"data_stored":181994481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261521408,"data_stored":261513723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:25.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:25.734 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:25.735 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:25.762 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-16T10:31:24.725369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231027233,"num_objects":56422,"num_object_clones":0,"num_object_copies":112844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249887,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13081,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1756736,"kb_used_data":661088,"kb_used_omap":12,"kb_used_meta":1095603,"kb_avail":281358784,"statfs":{"total":289910292480,"available":288111394816,"internally_reserved":0,"allocated":676954112,"data_stored":676092311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1121897474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,9,5,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24780800,"num_objects":-6050,"num_object_clones":0,"num_object_copies":-12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002845"},"pg_stats":[{"pgid":"2.f","version":"46'16962","reported_seq":25427,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.971424+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:22.971424+0000","last_peered":"2024-09-16T10:31:22.971424+0000","last_clean":"2024-09-16T10:31:22.971424+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:22.971424+0000","last_undegraded":"2024-09-16T10:31:22.971424+0000","last_fullsized":"2024-09-16T10:31:22.971424+0000","mapping_epoch":45,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14360576,"num_objects":3506,"num_object_clones":0,"num_object_copies":7012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16962,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16967","reported_seq":17128,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.146862+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:22.146862+0000","last_peered":"2024-09-16T10:31:22.146862+0000","last_clean":"2024-09-16T10:31:22.146862+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:22.146862+0000","last_undegraded":"2024-09-16T10:31:22.146862+0000","last_fullsized":"2024-09-16T10:31:22.146862+0000","mapping_epoch":34,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14315520,"num_objects":3495,"num_object_clones":0,"num_object_copies":6990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13180,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'17073","reported_seq":17327,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.092566+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:22.092566+0000","last_peered":"2024-09-16T10:31:22.092566+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:22.092566+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14704664,"num_objects":3591,"num_object_clones":0,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13130,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-712}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16948","reported_seq":17008,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.130433+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:22.130433+0000","last_peered":"2024-09-16T10:31:22.130433+0000","last_clean":"2024-09-16T10:31:22.130433+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:22.130433+0000","last_undegraded":"2024-09-16T10:31:22.130433+0000","last_fullsized":"2024-09-16T10:31:22.130433+0000","mapping_epoch":34,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14393344,"num_objects":3514,"num_object_clones":0,"num_object_copies":7028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16948,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16994","reported_seq":17155,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.971806+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:22.971806+0000","last_peered":"2024-09-16T10:31:22.971806+0000","last_clean":"2024-09-16T10:31:22.971806+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:22.971806+0000","last_undegraded":"2024-09-16T10:31:22.971806+0000","last_fullsized":"2024-09-16T10:31:22.971806+0000","mapping_epoch":33,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14381056,"num_objects":3511,"num_object_clones":0,"num_object_copies":7022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13636,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16844","reported_seq":17176,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.978165+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:22.978165+0000","last_peered":"2024-09-16T10:31:22.978165+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:22.978165+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14446592,"num_objects":3527,"num_object_clones":0,"num_object_copies":7054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13469,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-607}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17394","reported_seq":17803,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.964721+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:22.964721+0000","last_peered":"2024-09-16T10:31:22.964721+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:22.964721+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14442496,"num_objects":3526,"num_object_clones":0,"num_object_copies":7052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13636,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16905","reported_seq":17084,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.855864+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:22.855864+0000","last_peered":"2024-09-16T10:31:22.855864+0000","last_clean":"2024-09-16T10:31:22.855864+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:22.855864+0000","last_undegraded":"2024-09-16T10:31:22.855864+0000","last_fullsized":"2024-09-16T10:31:22.855864+0000","mapping_epoch":33,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14274560,"num_objects":3485,"num_object_clones":0,"num_object_copies":6970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12708,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16852","reported_seq":25359,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:21.952946+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:21.952946+0000","last_peered":"2024-09-16T10:31:21.952946+0000","last_clean":"2024-09-16T10:31:21.952946+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:21.952946+0000","last_undegraded":"2024-09-16T10:31:21.952946+0000","last_fullsized":"2024-09-16T10:31:21.952946+0000","mapping_epoch":43,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14783430,"num_objects":3611,"num_object_clones":0,"num_object_copies":7222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16850,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17464","reported_seq":17739,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.949487+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:22.949487+0000","last_peered":"2024-09-16T10:31:22.949487+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:22.949487+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14648262,"num_objects":3578,"num_object_clones":0,"num_object_copies":7156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17462,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-759}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16845","reported_seq":17043,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.177318+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:22.177318+0000","last_peered":"2024-09-16T10:31:22.177318+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:22.177318+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14640070,"num_objects":3576,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16843,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'17011","reported_seq":25095,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.147663+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:22.147663+0000","last_peered":"2024-09-16T10:31:22.147663+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:22.147663+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14603206,"num_objects":3567,"num_object_clones":0,"num_object_copies":7134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17009,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3999,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3370}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16601","reported_seq":16852,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.957952+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:22.957952+0000","last_peered":"2024-09-16T10:31:22.957952+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:22.957952+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14062051,"num_objects":3434,"num_object_clones":0,"num_object_copies":6868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16600,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17366","reported_seq":17643,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.924108+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:22.924108+0000","last_peered":"2024-09-16T10:31:22.924108+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:22.924108+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14534057,"num_objects":3551,"num_object_clones":0,"num_object_copies":7102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17363,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-716}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.559321+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:11.559321+0000","last_peered":"2024-09-16T10:29:11.559321+0000","last_clean":"2024-09-16T10:29:11.559321+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:11.559321+0000","last_undegraded":"2024-09-16T10:29:11.559321+0000","last_fullsized":"2024-09-16T10:29:11.559321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16701","reported_seq":16933,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.904893+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:22.904893+0000","last_peered":"2024-09-16T10:31:22.904893+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:22.904893+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14087110,"num_objects":3441,"num_object_clones":0,"num_object_copies":6882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16699,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17393","reported_seq":17635,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.943877+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:22.943877+0000","last_peered":"2024-09-16T10:31:22.943877+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:22.943877+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14350220,"num_objects":3507,"num_object_clones":0,"num_object_copies":7014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17390,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":231027214,"num_objects":56420,"num_object_clones":0,"num_object_copies":112840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249885,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13080,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":673239040,"data_stored":673155023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018987,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571632,"kb_used_data":251240,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93800208,"statfs":{"total":96636764160,"available":96051412992,"internally_reserved":0,"allocated":257269760,"data_stored":257001609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":5.3970000000000002},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":3.694}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":5.1139999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":8.7400000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691931,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532648,"kb_used_data":172064,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93839192,"statfs":{"total":96636764160,"available":96091332608,"internally_reserved":0,"allocated":176193536,"data_stored":175860351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":369225386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":10.157}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":9.2720000000000002},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":8.1489999999999991}]}]},{"osd":0,"up_from":24,"seq":103079215214,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652456,"kb_used_data":237784,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93719384,"statfs":{"total":96636764160,"available":95968649216,"internally_reserved":0,"allocated":243490816,"data_stored":243230351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424603304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,5,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":7.4329999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":3.9609999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":4.226},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":9.2590000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241680384,"data_stored":241676771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174796800,"data_stored":174724081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256761856,"data_stored":256754171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:25.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:26.336 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:26.338 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:26.363 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-16T10:31:24.725369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231027233,"num_objects":56422,"num_object_clones":0,"num_object_copies":112844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249887,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13081,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1756736,"kb_used_data":661088,"kb_used_omap":12,"kb_used_meta":1095603,"kb_avail":281358784,"statfs":{"total":289910292480,"available":288111394816,"internally_reserved":0,"allocated":676954112,"data_stored":676092311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1121897474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,9,5,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24780800,"num_objects":-6050,"num_object_clones":0,"num_object_copies":-12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6050,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002845"},"pg_stats":[{"pgid":"2.f","version":"46'16962","reported_seq":25427,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.971424+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:22.971424+0000","last_peered":"2024-09-16T10:31:22.971424+0000","last_clean":"2024-09-16T10:31:22.971424+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:22.971424+0000","last_undegraded":"2024-09-16T10:31:22.971424+0000","last_fullsized":"2024-09-16T10:31:22.971424+0000","mapping_epoch":45,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14360576,"num_objects":3506,"num_object_clones":0,"num_object_copies":7012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16962,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16967","reported_seq":17128,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.146862+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:22.146862+0000","last_peered":"2024-09-16T10:31:22.146862+0000","last_clean":"2024-09-16T10:31:22.146862+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:22.146862+0000","last_undegraded":"2024-09-16T10:31:22.146862+0000","last_fullsized":"2024-09-16T10:31:22.146862+0000","mapping_epoch":34,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14315520,"num_objects":3495,"num_object_clones":0,"num_object_copies":6990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13180,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'17073","reported_seq":17327,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.092566+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:22.092566+0000","last_peered":"2024-09-16T10:31:22.092566+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:22.092566+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14704664,"num_objects":3591,"num_object_clones":0,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13130,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-712}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16948","reported_seq":17008,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.130433+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:22.130433+0000","last_peered":"2024-09-16T10:31:22.130433+0000","last_clean":"2024-09-16T10:31:22.130433+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:22.130433+0000","last_undegraded":"2024-09-16T10:31:22.130433+0000","last_fullsized":"2024-09-16T10:31:22.130433+0000","mapping_epoch":34,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14393344,"num_objects":3514,"num_object_clones":0,"num_object_copies":7028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16948,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16994","reported_seq":17155,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.971806+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:22.971806+0000","last_peered":"2024-09-16T10:31:22.971806+0000","last_clean":"2024-09-16T10:31:22.971806+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:22.971806+0000","last_undegraded":"2024-09-16T10:31:22.971806+0000","last_fullsized":"2024-09-16T10:31:22.971806+0000","mapping_epoch":33,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14381056,"num_objects":3511,"num_object_clones":0,"num_object_copies":7022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13636,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16844","reported_seq":17176,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.978165+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:22.978165+0000","last_peered":"2024-09-16T10:31:22.978165+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:22.978165+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14446592,"num_objects":3527,"num_object_clones":0,"num_object_copies":7054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13469,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-607}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17394","reported_seq":17803,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.964721+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:22.964721+0000","last_peered":"2024-09-16T10:31:22.964721+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:22.964721+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14442496,"num_objects":3526,"num_object_clones":0,"num_object_copies":7052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13636,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16905","reported_seq":17084,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.855864+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:22.855864+0000","last_peered":"2024-09-16T10:31:22.855864+0000","last_clean":"2024-09-16T10:31:22.855864+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:22.855864+0000","last_undegraded":"2024-09-16T10:31:22.855864+0000","last_fullsized":"2024-09-16T10:31:22.855864+0000","mapping_epoch":33,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14274560,"num_objects":3485,"num_object_clones":0,"num_object_copies":6970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12708,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16852","reported_seq":25359,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:21.952946+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:21.952946+0000","last_peered":"2024-09-16T10:31:21.952946+0000","last_clean":"2024-09-16T10:31:21.952946+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:21.952946+0000","last_undegraded":"2024-09-16T10:31:21.952946+0000","last_fullsized":"2024-09-16T10:31:21.952946+0000","mapping_epoch":43,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14783430,"num_objects":3611,"num_object_clones":0,"num_object_copies":7222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16850,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17464","reported_seq":17739,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.949487+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:22.949487+0000","last_peered":"2024-09-16T10:31:22.949487+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:22.949487+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14648262,"num_objects":3578,"num_object_clones":0,"num_object_copies":7156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17462,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-759}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16845","reported_seq":17043,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.177318+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:22.177318+0000","last_peered":"2024-09-16T10:31:22.177318+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:22.177318+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14640070,"num_objects":3576,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16843,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'17011","reported_seq":25095,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.147663+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:22.147663+0000","last_peered":"2024-09-16T10:31:22.147663+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:22.147663+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14603206,"num_objects":3567,"num_object_clones":0,"num_object_copies":7134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17009,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3999,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3370}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16601","reported_seq":16852,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.957952+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:22.957952+0000","last_peered":"2024-09-16T10:31:22.957952+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:22.957952+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14062051,"num_objects":3434,"num_object_clones":0,"num_object_copies":6868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16600,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17366","reported_seq":17643,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.924108+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:22.924108+0000","last_peered":"2024-09-16T10:31:22.924108+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:22.924108+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14534057,"num_objects":3551,"num_object_clones":0,"num_object_copies":7102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17363,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-716}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.559321+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:11.559321+0000","last_peered":"2024-09-16T10:29:11.559321+0000","last_clean":"2024-09-16T10:29:11.559321+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:11.559321+0000","last_undegraded":"2024-09-16T10:29:11.559321+0000","last_fullsized":"2024-09-16T10:29:11.559321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16701","reported_seq":16933,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.904893+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:22.904893+0000","last_peered":"2024-09-16T10:31:22.904893+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:22.904893+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14087110,"num_objects":3441,"num_object_clones":0,"num_object_copies":6882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16699,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17393","reported_seq":17635,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.943877+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:22.943877+0000","last_peered":"2024-09-16T10:31:22.943877+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:22.943877+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14350220,"num_objects":3507,"num_object_clones":0,"num_object_copies":7014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17390,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":231027214,"num_objects":56420,"num_object_clones":0,"num_object_copies":112840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249885,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13080,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":673239040,"data_stored":673155023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018987,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571632,"kb_used_data":251240,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93800208,"statfs":{"total":96636764160,"available":96051412992,"internally_reserved":0,"allocated":257269760,"data_stored":257001609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":5.3970000000000002},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":3.694}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":5.1139999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":8.7400000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691931,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532648,"kb_used_data":172064,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93839192,"statfs":{"total":96636764160,"available":96091332608,"internally_reserved":0,"allocated":176193536,"data_stored":175860351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":369225386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":10.157}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":9.2720000000000002},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":8.1489999999999991}]}]},{"osd":0,"up_from":24,"seq":103079215214,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652456,"kb_used_data":237784,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93719384,"statfs":{"total":96636764160,"available":95968649216,"internally_reserved":0,"allocated":243490816,"data_stored":243230351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424603304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,5,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":7.4329999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":3.9609999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":4.226},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":9.2590000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241680384,"data_stored":241676771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174796800,"data_stored":174724081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256761856,"data_stored":256754171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:26.364 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:26.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'17073", 'reported_seq': 17327, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:22.092566+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:22.092566+0000', 'last_peered': '2024-09-16T10:31:22.092566+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:22.092566+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'7000", 'ondisk_log_start': "42'7000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14704664, 'num_objects': 3591, 'num_object_clones': 0, 'num_object_copies': 7182, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3591, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13130, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -712}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:26.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:26.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'16844", 'reported_seq': 17176, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:22.978165+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:22.978165+0000', 'last_peered': '2024-09-16T10:31:22.978165+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:22.978165+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'6800", 'ondisk_log_start': "42'6800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14446592, 'num_objects': 3527, 'num_object_clones': 0, 'num_object_copies': 7054, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13469, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -607}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:26.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:26.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'17394", 'reported_seq': 17803, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:22.964721+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:22.964721+0000', 'last_peered': '2024-09-16T10:31:22.964721+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:22.964721+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'7300", 'ondisk_log_start': "42'7300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14442496, 'num_objects': 3526, 'num_object_clones': 0, 'num_object_copies': 7052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13636, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -761}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:26.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:26.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'17464", 'reported_seq': 17739, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:22.949487+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:22.949487+0000', 'last_peered': '2024-09-16T10:31:22.949487+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:22.949487+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'7400", 'ondisk_log_start': "42'7400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14648262, 'num_objects': 3578, 'num_object_clones': 0, 'num_object_copies': 7156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17462, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -759}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:26.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:26.365 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'16845", 'reported_seq': 17043, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:22.177318+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:22.177318+0000', 'last_peered': '2024-09-16T10:31:22.177318+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:22.177318+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'6800", 'ondisk_log_start': "42'6800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14640070, 'num_objects': 3576, 'num_object_clones': 0, 'num_object_copies': 7152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16843, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -657}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:26.366 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:31:26.366 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'17011", 'reported_seq': 25095, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:22.147663+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:31:22.147663+0000', 'last_peered': '2024-09-16T10:31:22.147663+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:31:22.147663+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'7000", 'ondisk_log_start': "42'7000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14603206, 'num_objects': 3567, 'num_object_clones': 0, 'num_object_copies': 7134, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 197, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3567, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17009, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3999, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3370}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:26.366 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:26.366 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'16601", 'reported_seq': 16852, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:22.957952+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:22.957952+0000', 'last_peered': '2024-09-16T10:31:22.957952+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:22.957952+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'6600", 'ondisk_log_start': "42'6600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14062051, 'num_objects': 3434, 'num_object_clones': 0, 'num_object_copies': 6868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16600, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -641}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:26.366 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:26.366 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'17366", 'reported_seq': 17643, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:22.924108+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:22.924108+0000', 'last_peered': '2024-09-16T10:31:22.924108+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:22.924108+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'7300", 'ondisk_log_start': "42'7300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14534057, 'num_objects': 3551, 'num_object_clones': 0, 'num_object_copies': 7102, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3551, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17363, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -716}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:26.366 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:26.366 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'16701", 'reported_seq': 16933, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:22.904893+0000', 'last_change': '2024-09-16T10:23:56.904486+0000', 'last_active': '2024-09-16T10:31:22.904893+0000', 'last_peered': '2024-09-16T10:31:22.904893+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:22.904893+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'6700", 'ondisk_log_start': "42'6700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14087110, 'num_objects': 3441, 'num_object_clones': 0, 'num_object_copies': 6882, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4095, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3441, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16699, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 76, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -652}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:26.366 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:26.366 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'17393", 'reported_seq': 17635, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:22.943877+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:22.943877+0000', 'last_peered': '2024-09-16T10:31:22.943877+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:22.943877+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'7300", 'ondisk_log_start': "42'7300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14350220, 'num_objects': 3507, 'num_object_clones': 0, 'num_object_copies': 7014, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17390, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:26.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:27.075 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:27.076 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 38266/112844 objects degraded (33.911%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":575,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":46,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":56422,"data_bytes":231027233,"bytes_used":1798897664,"bytes_avail":288111394816,"bytes_total":289910292480,"degraded_objects":38266,"degraded_total":112844,"degraded_ratio":0.33910531353018331,"recovering_objects_per_sec":26,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":314,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":504},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:31:27.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:31:27.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:27.688 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:27.691 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:27.717 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-16T10:31:26.725840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231027233,"num_objects":56422,"num_object_clones":0,"num_object_copies":112844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56422,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249887,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13081,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1756736,"kb_used_data":661088,"kb_used_omap":12,"kb_used_meta":1095603,"kb_avail":281358784,"statfs":{"total":289910292480,"available":288111394816,"internally_reserved":0,"allocated":676954112,"data_stored":676092311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1121897474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4,9,5,8,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18755584,"num_objects":-4579,"num_object_clones":0,"num_object_copies":-9158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4579,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002832"},"pg_stats":[{"pgid":"2.f","version":"46'16962","reported_seq":25427,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.971424+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:22.971424+0000","last_peered":"2024-09-16T10:31:22.971424+0000","last_clean":"2024-09-16T10:31:22.971424+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:22.971424+0000","last_undegraded":"2024-09-16T10:31:22.971424+0000","last_fullsized":"2024-09-16T10:31:22.971424+0000","mapping_epoch":45,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14360576,"num_objects":3506,"num_object_clones":0,"num_object_copies":7012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16962,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'16967","reported_seq":17128,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.146862+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:22.146862+0000","last_peered":"2024-09-16T10:31:22.146862+0000","last_clean":"2024-09-16T10:31:22.146862+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:22.146862+0000","last_undegraded":"2024-09-16T10:31:22.146862+0000","last_fullsized":"2024-09-16T10:31:22.146862+0000","mapping_epoch":34,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14315520,"num_objects":3495,"num_object_clones":0,"num_object_copies":6990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13180,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'17073","reported_seq":17327,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.092566+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:22.092566+0000","last_peered":"2024-09-16T10:31:22.092566+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:22.092566+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14704664,"num_objects":3591,"num_object_clones":0,"num_object_copies":7182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3591,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13130,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-712}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'16948","reported_seq":17008,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.130433+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:22.130433+0000","last_peered":"2024-09-16T10:31:22.130433+0000","last_clean":"2024-09-16T10:31:22.130433+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:22.130433+0000","last_undegraded":"2024-09-16T10:31:22.130433+0000","last_fullsized":"2024-09-16T10:31:22.130433+0000","mapping_epoch":34,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14393344,"num_objects":3514,"num_object_clones":0,"num_object_copies":7028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16948,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'16994","reported_seq":17155,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.971806+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:22.971806+0000","last_peered":"2024-09-16T10:31:22.971806+0000","last_clean":"2024-09-16T10:31:22.971806+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:22.971806+0000","last_undegraded":"2024-09-16T10:31:22.971806+0000","last_fullsized":"2024-09-16T10:31:22.971806+0000","mapping_epoch":33,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14381056,"num_objects":3511,"num_object_clones":0,"num_object_copies":7022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13636,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16844","reported_seq":17176,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.978165+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:22.978165+0000","last_peered":"2024-09-16T10:31:22.978165+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:22.978165+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14446592,"num_objects":3527,"num_object_clones":0,"num_object_copies":7054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13469,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-607}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17394","reported_seq":17803,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.964721+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:22.964721+0000","last_peered":"2024-09-16T10:31:22.964721+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:22.964721+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14442496,"num_objects":3526,"num_object_clones":0,"num_object_copies":7052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13636,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'16905","reported_seq":17084,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:22.855864+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:22.855864+0000","last_peered":"2024-09-16T10:31:22.855864+0000","last_clean":"2024-09-16T10:31:22.855864+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:22.855864+0000","last_undegraded":"2024-09-16T10:31:22.855864+0000","last_fullsized":"2024-09-16T10:31:22.855864+0000","mapping_epoch":33,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14274560,"num_objects":3485,"num_object_clones":0,"num_object_copies":6970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12708,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16852","reported_seq":25359,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:21.952946+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:21.952946+0000","last_peered":"2024-09-16T10:31:21.952946+0000","last_clean":"2024-09-16T10:31:21.952946+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:21.952946+0000","last_undegraded":"2024-09-16T10:31:21.952946+0000","last_fullsized":"2024-09-16T10:31:21.952946+0000","mapping_epoch":43,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14783430,"num_objects":3611,"num_object_clones":0,"num_object_copies":7222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16850,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17464","reported_seq":17739,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.949487+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:22.949487+0000","last_peered":"2024-09-16T10:31:22.949487+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:22.949487+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14648262,"num_objects":3578,"num_object_clones":0,"num_object_copies":7156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17462,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-759}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'16845","reported_seq":17043,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.177318+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:22.177318+0000","last_peered":"2024-09-16T10:31:22.177318+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:22.177318+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14640070,"num_objects":3576,"num_object_clones":0,"num_object_copies":7152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16843,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-657}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'17011","reported_seq":25095,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.147663+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:22.147663+0000","last_peered":"2024-09-16T10:31:22.147663+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:22.147663+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14603206,"num_objects":3567,"num_object_clones":0,"num_object_copies":7134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17009,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3999,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3370}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16601","reported_seq":16852,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.957952+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:22.957952+0000","last_peered":"2024-09-16T10:31:22.957952+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:22.957952+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6600","ondisk_log_start":"42'6600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14062051,"num_objects":3434,"num_object_clones":0,"num_object_copies":6868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16600,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-641}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17366","reported_seq":17643,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.924108+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:22.924108+0000","last_peered":"2024-09-16T10:31:22.924108+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:22.924108+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14534057,"num_objects":3551,"num_object_clones":0,"num_object_copies":7102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17363,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-716}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.559321+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:11.559321+0000","last_peered":"2024-09-16T10:29:11.559321+0000","last_clean":"2024-09-16T10:29:11.559321+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:11.559321+0000","last_undegraded":"2024-09-16T10:29:11.559321+0000","last_fullsized":"2024-09-16T10:29:11.559321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16701","reported_seq":16933,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.904893+0000","last_change":"2024-09-16T10:23:56.904486+0000","last_active":"2024-09-16T10:31:22.904893+0000","last_peered":"2024-09-16T10:31:22.904893+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:22.904893+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14087110,"num_objects":3441,"num_object_clones":0,"num_object_copies":6882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4095,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16699,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-652}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17393","reported_seq":17635,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:22.943877+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:22.943877+0000","last_peered":"2024-09-16T10:31:22.943877+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:22.943877+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14350220,"num_objects":3507,"num_object_clones":0,"num_object_copies":7014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17390,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":231027214,"num_objects":56420,"num_object_clones":0,"num_object_copies":112840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":249885,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13080,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":673239040,"data_stored":673155023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018987,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571632,"kb_used_data":251240,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93800208,"statfs":{"total":96636764160,"available":96051412992,"internally_reserved":0,"allocated":257269760,"data_stored":257001609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":5.3970000000000002},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":3.694}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":5.1139999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":8.7400000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691931,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532648,"kb_used_data":172064,"kb_used_omap":4,"kb_used_meta":360571,"kb_avail":93839192,"statfs":{"total":96636764160,"available":96091332608,"internally_reserved":0,"allocated":176193536,"data_stored":175860351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":369225386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":10.157}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":9.2720000000000002},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":8.1489999999999991}]}]},{"osd":0,"up_from":24,"seq":103079215214,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":652456,"kb_used_data":237784,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93719384,"statfs":{"total":96636764160,"available":95968649216,"internally_reserved":0,"allocated":243490816,"data_stored":243230351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424603304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,5,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":7.4329999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":3.9609999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":4.226},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":9.2590000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":241680384,"data_stored":241676771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174796800,"data_stored":174724081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256761856,"data_stored":256754171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:30.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:31.269 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:31.270 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:31.292 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-16T10:31:30.726773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":221590049,"num_objects":54118,"num_object_clones":0,"num_object_copies":108236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252191,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13287,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160627,"ondisk_log_size":160627,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1736988,"kb_used_data":641344,"kb_used_omap":12,"kb_used_meta":1095603,"kb_avail":281378532,"statfs":{"total":289910292480,"available":288131616768,"internally_reserved":0,"allocated":656736256,"data_stored":655874455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1121897474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,4,2,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15437824,"num_objects":-3769,"num_object_clones":0,"num_object_copies":-7538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001880"},"pg_stats":[{"pgid":"2.f","version":"46'17113","reported_seq":25578,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.031523+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:28.031523+0000","last_peered":"2024-09-16T10:31:28.031523+0000","last_clean":"2024-09-16T10:31:28.031523+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:28.031523+0000","last_undegraded":"2024-09-16T10:31:28.031523+0000","last_fullsized":"2024-09-16T10:31:28.031523+0000","mapping_epoch":45,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13742080,"num_objects":3355,"num_object_clones":0,"num_object_copies":6710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17113,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'17115","reported_seq":17276,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:27.105034+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:27.105034+0000","last_peered":"2024-09-16T10:31:27.105034+0000","last_clean":"2024-09-16T10:31:27.105034+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:27.105034+0000","last_undegraded":"2024-09-16T10:31:27.105034+0000","last_fullsized":"2024-09-16T10:31:27.105034+0000","mapping_epoch":34,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13709312,"num_objects":3347,"num_object_clones":0,"num_object_copies":6694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13328,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'17215","reported_seq":17469,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.169364+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:27.169364+0000","last_peered":"2024-09-16T10:31:27.169364+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:27.169364+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14123032,"num_objects":3449,"num_object_clones":0,"num_object_copies":6898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13272,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'17088","reported_seq":17148,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:27.118321+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:27.118321+0000","last_peered":"2024-09-16T10:31:27.118321+0000","last_clean":"2024-09-16T10:31:27.118321+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:27.118321+0000","last_undegraded":"2024-09-16T10:31:27.118321+0000","last_fullsized":"2024-09-16T10:31:27.118321+0000","mapping_epoch":34,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13819904,"num_objects":3374,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17088,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'17140","reported_seq":17299,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:27.968050+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:27.968050+0000","last_peered":"2024-09-16T10:31:27.968050+0000","last_clean":"2024-09-16T10:31:27.968050+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:27.968050+0000","last_undegraded":"2024-09-16T10:31:27.968050+0000","last_fullsized":"2024-09-16T10:31:27.968050+0000","mapping_epoch":33,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13783040,"num_objects":3365,"num_object_clones":0,"num_object_copies":6730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13782,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16983","reported_seq":17316,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.990676+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:27.990676+0000","last_peered":"2024-09-16T10:31:27.990676+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:27.990676+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13877248,"num_objects":3388,"num_object_clones":0,"num_object_copies":6776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13608,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17547","reported_seq":17954,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.885116+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:27.885116+0000","last_peered":"2024-09-16T10:31:27.885116+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:27.885116+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13815808,"num_objects":3373,"num_object_clones":0,"num_object_copies":6746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13789,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-914}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'17039","reported_seq":17218,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.002249+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:28.002249+0000","last_peered":"2024-09-16T10:31:28.002249+0000","last_clean":"2024-09-16T10:31:28.002249+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:28.002249+0000","last_undegraded":"2024-09-16T10:31:28.002249+0000","last_fullsized":"2024-09-16T10:31:28.002249+0000","mapping_epoch":33,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13725696,"num_objects":3351,"num_object_clones":0,"num_object_copies":6702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12842,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16997","reported_seq":25504,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:26.853910+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:26.853910+0000","last_peered":"2024-09-16T10:31:26.853910+0000","last_clean":"2024-09-16T10:31:26.853910+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:26.853910+0000","last_undegraded":"2024-09-16T10:31:26.853910+0000","last_fullsized":"2024-09-16T10:31:26.853910+0000","mapping_epoch":43,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14189510,"num_objects":3466,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16995,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17606","reported_seq":17883,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.968884+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:27.968884+0000","last_peered":"2024-09-16T10:31:27.968884+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:27.968884+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14066630,"num_objects":3436,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17604,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-901}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'17005","reported_seq":17203,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.129002+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:27.129002+0000","last_peered":"2024-09-16T10:31:27.129002+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:27.129002+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13984710,"num_objects":3416,"num_object_clones":0,"num_object_copies":6832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17003,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'17150","reported_seq":25624,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.198912+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:27.198912+0000","last_peered":"2024-09-16T10:31:27.198912+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:27.198912+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14033862,"num_objects":3428,"num_object_clones":0,"num_object_copies":6856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17148,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4193,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3425}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16754","reported_seq":17005,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:28.009945+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:28.009945+0000","last_peered":"2024-09-16T10:31:28.009945+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:28.009945+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13435363,"num_objects":3281,"num_object_clones":0,"num_object_copies":6562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16753,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17510","reported_seq":17787,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.961084+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:27.961084+0000","last_peered":"2024-09-16T10:31:27.961084+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:27.961084+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13944233,"num_objects":3407,"num_object_clones":0,"num_object_copies":6814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17507,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.559321+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:11.559321+0000","last_peered":"2024-09-16T10:29:11.559321+0000","last_clean":"2024-09-16T10:29:11.559321+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:11.559321+0000","last_undegraded":"2024-09-16T10:29:11.559321+0000","last_fullsized":"2024-09-16T10:29:11.559321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16844","reported_seq":17102,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:28.045617+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:28.045617+0000","last_peered":"2024-09-16T10:31:28.045617+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:28.045617+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13501382,"num_objects":3298,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16842,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17518","reported_seq":17761,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.987313+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:27.987313+0000","last_peered":"2024-09-16T10:31:27.987313+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:27.987313+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13838220,"num_objects":3382,"num_object_clones":0,"num_object_copies":6764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17515,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":221590030,"num_objects":54116,"num_object_clones":0,"num_object_copies":108232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252189,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13286,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":654278656,"data_stored":654194639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160624,"ondisk_log_size":160624,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018988,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571088,"kb_used_data":246600,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93800752,"statfs":{"total":96636764160,"available":96051970048,"internally_reserved":0,"allocated":252518400,"data_stored":252250249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":3.5800000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.2789999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.248}]}]},{"osd":1,"up_from":40,"seq":171798691932,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517516,"kb_used_data":165124,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93854324,"statfs":{"total":96636764160,"available":96106827776,"internally_reserved":0,"allocated":169086976,"data_stored":168753791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":360836778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":3.4830000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.9729999999999999},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.2830000000000004}]}]},{"osd":0,"up_from":24,"seq":103079215215,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":648384,"kb_used_data":229620,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93723456,"statfs":{"total":96636764160,"available":95972818944,"internally_reserved":0,"allocated":235130880,"data_stored":234870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":428797608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.375},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":9.734}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":1.7410000000000001},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.34599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234622976,"data_stored":234619363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167641088,"data_stored":167568369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":252014592,"data_stored":252006907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:31.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:31.870 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:31.871 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:31.901 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-16T10:31:30.726773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":221590049,"num_objects":54118,"num_object_clones":0,"num_object_copies":108236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252191,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13287,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160627,"ondisk_log_size":160627,"up":48,"acting":38,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1736988,"kb_used_data":641344,"kb_used_omap":12,"kb_used_meta":1095603,"kb_avail":281378532,"statfs":{"total":289910292480,"available":288131616768,"internally_reserved":0,"allocated":656736256,"data_stored":655874455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1121897474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,4,2,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15437824,"num_objects":-3769,"num_object_clones":0,"num_object_copies":-7538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001880"},"pg_stats":[{"pgid":"2.f","version":"46'17113","reported_seq":25578,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.031523+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:28.031523+0000","last_peered":"2024-09-16T10:31:28.031523+0000","last_clean":"2024-09-16T10:31:28.031523+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:28.031523+0000","last_undegraded":"2024-09-16T10:31:28.031523+0000","last_fullsized":"2024-09-16T10:31:28.031523+0000","mapping_epoch":45,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13742080,"num_objects":3355,"num_object_clones":0,"num_object_copies":6710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17113,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"46'17115","reported_seq":17276,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:27.105034+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:27.105034+0000","last_peered":"2024-09-16T10:31:27.105034+0000","last_clean":"2024-09-16T10:31:27.105034+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:27.105034+0000","last_undegraded":"2024-09-16T10:31:27.105034+0000","last_fullsized":"2024-09-16T10:31:27.105034+0000","mapping_epoch":34,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13709312,"num_objects":3347,"num_object_clones":0,"num_object_copies":6694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13328,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"46'17215","reported_seq":17469,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.169364+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:27.169364+0000","last_peered":"2024-09-16T10:31:27.169364+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:27.169364+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14123032,"num_objects":3449,"num_object_clones":0,"num_object_copies":6898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3449,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13272,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"46'17088","reported_seq":17148,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:27.118321+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:27.118321+0000","last_peered":"2024-09-16T10:31:27.118321+0000","last_clean":"2024-09-16T10:31:27.118321+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:27.118321+0000","last_undegraded":"2024-09-16T10:31:27.118321+0000","last_fullsized":"2024-09-16T10:31:27.118321+0000","mapping_epoch":34,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13819904,"num_objects":3374,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17088,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'17140","reported_seq":17299,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:27.968050+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:27.968050+0000","last_peered":"2024-09-16T10:31:27.968050+0000","last_clean":"2024-09-16T10:31:27.968050+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:27.968050+0000","last_undegraded":"2024-09-16T10:31:27.968050+0000","last_fullsized":"2024-09-16T10:31:27.968050+0000","mapping_epoch":33,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13783040,"num_objects":3365,"num_object_clones":0,"num_object_copies":6730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13782,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16983","reported_seq":17316,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.990676+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:27.990676+0000","last_peered":"2024-09-16T10:31:27.990676+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:27.990676+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13877248,"num_objects":3388,"num_object_clones":0,"num_object_copies":6776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13608,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17547","reported_seq":17954,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.885116+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:27.885116+0000","last_peered":"2024-09-16T10:31:27.885116+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:27.885116+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13815808,"num_objects":3373,"num_object_clones":0,"num_object_copies":6746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13789,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-914}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'17039","reported_seq":17218,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.002249+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:28.002249+0000","last_peered":"2024-09-16T10:31:28.002249+0000","last_clean":"2024-09-16T10:31:28.002249+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:28.002249+0000","last_undegraded":"2024-09-16T10:31:28.002249+0000","last_fullsized":"2024-09-16T10:31:28.002249+0000","mapping_epoch":33,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13725696,"num_objects":3351,"num_object_clones":0,"num_object_copies":6702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12842,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973813+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:29:03.973813+0000","last_peered":"2024-09-16T10:29:03.973813+0000","last_clean":"2024-09-16T10:29:03.973813+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:29:03.973813+0000","last_undegraded":"2024-09-16T10:29:03.973813+0000","last_fullsized":"2024-09-16T10:29:03.973813+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"46'16997","reported_seq":25504,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:26.853910+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:26.853910+0000","last_peered":"2024-09-16T10:31:26.853910+0000","last_clean":"2024-09-16T10:31:26.853910+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:26.853910+0000","last_undegraded":"2024-09-16T10:31:26.853910+0000","last_fullsized":"2024-09-16T10:31:26.853910+0000","mapping_epoch":43,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14189510,"num_objects":3466,"num_object_clones":0,"num_object_copies":6932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16995,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:06.542998+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:29:06.542998+0000","last_peered":"2024-09-16T10:29:06.542998+0000","last_clean":"2024-09-16T10:29:06.542998+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:29:06.542998+0000","last_undegraded":"2024-09-16T10:29:06.542998+0000","last_fullsized":"2024-09-16T10:29:06.542998+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17606","reported_seq":17883,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.968884+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:27.968884+0000","last_peered":"2024-09-16T10:31:27.968884+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:27.968884+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14066630,"num_objects":3436,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17604,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-901}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"46'17005","reported_seq":17203,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.129002+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:27.129002+0000","last_peered":"2024-09-16T10:31:27.129002+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:27.129002+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13984710,"num_objects":3416,"num_object_clones":0,"num_object_copies":6832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17003,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":72,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.973960+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:29:03.973960+0000","last_peered":"2024-09-16T10:29:03.973960+0000","last_clean":"2024-09-16T10:29:03.973960+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:29:03.973960+0000","last_undegraded":"2024-09-16T10:29:03.973960+0000","last_fullsized":"2024-09-16T10:29:03.973960+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"46'17150","reported_seq":25624,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.198912+0000","last_change":"2024-09-16T10:29:02.427092+0000","last_active":"2024-09-16T10:31:27.198912+0000","last_peered":"2024-09-16T10:31:27.198912+0000","last_clean":"2024-09-16T10:23:54.357802+0000","last_became_active":"2024-09-16T10:23:56.801992+0000","last_became_peered":"2024-09-16T10:23:56.801992+0000","last_unstale":"2024-09-16T10:31:27.198912+0000","last_undegraded":"2024-09-16T10:23:56.408365+0000","last_fullsized":"2024-09-16T10:23:56.363164+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14033862,"num_objects":3428,"num_object_clones":0,"num_object_copies":6856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17148,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4193,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":3425}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16754","reported_seq":17005,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:28.009945+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:28.009945+0000","last_peered":"2024-09-16T10:31:28.009945+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:28.009945+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13435363,"num_objects":3281,"num_object_clones":0,"num_object_copies":6562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16753,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974284+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:29:03.974284+0000","last_peered":"2024-09-16T10:29:03.974284+0000","last_clean":"2024-09-16T10:29:03.974284+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:29:03.974284+0000","last_undegraded":"2024-09-16T10:29:03.974284+0000","last_fullsized":"2024-09-16T10:29:03.974284+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17510","reported_seq":17787,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.961084+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:27.961084+0000","last_peered":"2024-09-16T10:31:27.961084+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:27.961084+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13944233,"num_objects":3407,"num_object_clones":0,"num_object_copies":6814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17507,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":61,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:11.559321+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:29:11.559321+0000","last_peered":"2024-09-16T10:29:11.559321+0000","last_clean":"2024-09-16T10:29:11.559321+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:29:11.559321+0000","last_undegraded":"2024-09-16T10:29:11.559321+0000","last_fullsized":"2024-09-16T10:29:11.559321+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16844","reported_seq":17102,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:28.045617+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:28.045617+0000","last_peered":"2024-09-16T10:31:28.045617+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:28.045617+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13501382,"num_objects":3298,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16842,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":58,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974103+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:29:03.974103+0000","last_peered":"2024-09-16T10:29:03.974103+0000","last_clean":"2024-09-16T10:29:03.974103+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:29:03.974103+0000","last_undegraded":"2024-09-16T10:29:03.974103+0000","last_fullsized":"2024-09-16T10:29:03.974103+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":67,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.974232+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:29:03.974232+0000","last_peered":"2024-09-16T10:29:03.974232+0000","last_clean":"2024-09-16T10:29:03.974232+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:29:03.974232+0000","last_undegraded":"2024-09-16T10:29:03.974232+0000","last_fullsized":"2024-09-16T10:29:03.974232+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17518","reported_seq":17761,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.987313+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:27.987313+0000","last_peered":"2024-09-16T10:31:27.987313+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:27.987313+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13838220,"num_objects":3382,"num_object_clones":0,"num_object_copies":6764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17515,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":221590030,"num_objects":54116,"num_object_clones":0,"num_object_copies":108232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252189,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13286,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":654278656,"data_stored":654194639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160624,"ondisk_log_size":160624,"up":32,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018988,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":571088,"kb_used_data":246600,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93800752,"statfs":{"total":96636764160,"available":96051970048,"internally_reserved":0,"allocated":252518400,"data_stored":252250249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":3.5800000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.2789999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.248}]}]},{"osd":1,"up_from":40,"seq":171798691932,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517516,"kb_used_data":165124,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93854324,"statfs":{"total":96636764160,"available":96106827776,"internally_reserved":0,"allocated":169086976,"data_stored":168753791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":360836778},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":3.4830000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.9729999999999999},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.2830000000000004}]}]},{"osd":0,"up_from":24,"seq":103079215215,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":648384,"kb_used_data":229620,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93723456,"statfs":{"total":96636764160,"available":95972818944,"internally_reserved":0,"allocated":235130880,"data_stored":234870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":428797608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.375},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":9.734}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":1.7410000000000001},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.34599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234622976,"data_stored":234619363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167641088,"data_stored":167568369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":252014592,"data_stored":252006907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:31.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:31.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "46'17215", 'reported_seq': 17469, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:27.169364+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:27.169364+0000', 'last_peered': '2024-09-16T10:31:27.169364+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:27.169364+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'7200", 'ondisk_log_start': "42'7200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14123032, 'num_objects': 3449, 'num_object_clones': 0, 'num_object_copies': 6898, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3449, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13272, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -854}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:31.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:31.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "46'16983", 'reported_seq': 17316, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:27.990676+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:27.990676+0000', 'last_peered': '2024-09-16T10:31:27.990676+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:27.990676+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'6900", 'ondisk_log_start': "42'6900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13877248, 'num_objects': 3388, 'num_object_clones': 0, 'num_object_copies': 6776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13608, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -746}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:31.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:31.903 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "46'17547", 'reported_seq': 17954, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:27.885116+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:27.885116+0000', 'last_peered': '2024-09-16T10:31:27.885116+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:27.885116+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'7500", 'ondisk_log_start': "42'7500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13815808, 'num_objects': 3373, 'num_object_clones': 0, 'num_object_copies': 6746, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13789, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -914}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:31.904 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:31.904 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "46'17606", 'reported_seq': 17883, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:27.968884+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:27.968884+0000', 'last_peered': '2024-09-16T10:31:27.968884+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:27.968884+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'7600", 'ondisk_log_start': "42'7600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14066630, 'num_objects': 3436, 'num_object_clones': 0, 'num_object_copies': 6872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3436, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17604, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -901}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:31.904 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:31.904 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "46'17005", 'reported_seq': 17203, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:27.129002+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:27.129002+0000', 'last_peered': '2024-09-16T10:31:27.129002+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:27.129002+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'7000", 'ondisk_log_start': "42'7000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13984710, 'num_objects': 3416, 'num_object_clones': 0, 'num_object_copies': 6832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17003, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -817}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:31.904 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:31:31.904 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "46'17150", 'reported_seq': 25624, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:27.198912+0000', 'last_change': '2024-09-16T10:29:02.427092+0000', 'last_active': '2024-09-16T10:31:27.198912+0000', 'last_peered': '2024-09-16T10:31:27.198912+0000', 'last_clean': '2024-09-16T10:23:54.357802+0000', 'last_became_active': '2024-09-16T10:23:56.801992+0000', 'last_became_peered': '2024-09-16T10:23:56.801992+0000', 'last_unstale': '2024-09-16T10:31:27.198912+0000', 'last_undegraded': '2024-09-16T10:23:56.408365+0000', 'last_fullsized': '2024-09-16T10:23:56.363164+0000', 'mapping_epoch': 41, 'log_start': "42'7100", 'ondisk_log_start': "42'7100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14033862, 'num_objects': 3428, 'num_object_clones': 0, 'num_object_copies': 6856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17148, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4193, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': 3425}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:31.905 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:31.905 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "46'16754", 'reported_seq': 17005, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:28.009945+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:28.009945+0000', 'last_peered': '2024-09-16T10:31:28.009945+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:28.009945+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'6700", 'ondisk_log_start': "42'6700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13435363, 'num_objects': 3281, 'num_object_clones': 0, 'num_object_copies': 6562, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16753, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:31.905 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:31.905 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "46'17510", 'reported_seq': 17787, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:27.961084+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:27.961084+0000', 'last_peered': '2024-09-16T10:31:27.961084+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:27.961084+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'7500", 'ondisk_log_start': "42'7500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13944233, 'num_objects': 3407, 'num_object_clones': 0, 'num_object_copies': 6814, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17507, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:31.905 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:31.905 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "46'16844", 'reported_seq': 17102, 'reported_epoch': 46, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:28.045617+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:31:28.045617+0000', 'last_peered': '2024-09-16T10:31:28.045617+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:28.045617+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'6800", 'ondisk_log_start': "42'6800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13501382, 'num_objects': 3298, 'num_object_clones': 0, 'num_object_copies': 6596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16842, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 88, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -785}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:31.905 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:31.905 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "46'17518", 'reported_seq': 17761, 'reported_epoch': 46, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:27.987313+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:27.987313+0000', 'last_peered': '2024-09-16T10:31:27.987313+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:27.987313+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'7500", 'ondisk_log_start': "42'7500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13838220, 'num_objects': 3382, 'num_object_clones': 0, 'num_object_copies': 6764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3382, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17515, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:31.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:32.586 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:32.586 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 38060/108236 objects degraded (35.164%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":581,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":24,"num_pools":2,"num_objects":54118,"data_bytes":221590049,"bytes_used":1778675712,"bytes_avail":288131616768,"bytes_total":289910292480,"degraded_objects":38060,"degraded_total":108236,"degraded_ratio":0.35163901104992795,"recovering_objects_per_sec":25,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":206,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":471},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:31:32.610 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:31:32.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:33.160 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:33.161 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:33.187 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-16T10:31:32.727225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218178081,"num_objects":53285,"num_object_clones":0,"num_object_copies":106570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53285,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253024,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13290,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160660,"ondisk_log_size":160660,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1734696,"kb_used_data":630856,"kb_used_omap":12,"kb_used_meta":1103795,"kb_avail":281380824,"statfs":{"total":289910292480,"available":288133963776,"internally_reserved":0,"allocated":645996544,"data_stored":645120979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1130286082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,2,8,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12849152,"num_objects":-3137,"num_object_clones":0,"num_object_copies":-6274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-209,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":209,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001855"},"pg_stats":[{"pgid":"2.f","version":"46'17113","reported_seq":25578,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.031523+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:28.031523+0000","last_peered":"2024-09-16T10:31:28.031523+0000","last_clean":"2024-09-16T10:31:28.031523+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:28.031523+0000","last_undegraded":"2024-09-16T10:31:28.031523+0000","last_fullsized":"2024-09-16T10:31:28.031523+0000","mapping_epoch":45,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13742080,"num_objects":3355,"num_object_clones":0,"num_object_copies":6710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17113,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17267","reported_seq":17430,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.126039+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:32.126039+0000","last_peered":"2024-09-16T10:31:32.126039+0000","last_clean":"2024-09-16T10:31:32.126039+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:32.126039+0000","last_undegraded":"2024-09-16T10:31:32.126039+0000","last_fullsized":"2024-09-16T10:31:32.126039+0000","mapping_epoch":34,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13086720,"num_objects":3195,"num_object_clones":0,"num_object_copies":6390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17336","reported_seq":17591,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.208208+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:32.208208+0000","last_peered":"2024-09-16T10:31:32.208208+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:32.208208+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13627416,"num_objects":3328,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13393,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17238","reported_seq":17300,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.143990+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:32.143990+0000","last_peered":"2024-09-16T10:31:32.143990+0000","last_clean":"2024-09-16T10:31:32.143990+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:32.143990+0000","last_undegraded":"2024-09-16T10:31:32.143990+0000","last_fullsized":"2024-09-16T10:31:32.143990+0000","mapping_epoch":34,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13205504,"num_objects":3224,"num_object_clones":0,"num_object_copies":6448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17238,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"46'17140","reported_seq":17299,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:27.968050+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:27.968050+0000","last_peered":"2024-09-16T10:31:27.968050+0000","last_clean":"2024-09-16T10:31:27.968050+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:27.968050+0000","last_undegraded":"2024-09-16T10:31:27.968050+0000","last_fullsized":"2024-09-16T10:31:27.968050+0000","mapping_epoch":33,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13783040,"num_objects":3365,"num_object_clones":0,"num_object_copies":6730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13782,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"46'16983","reported_seq":17316,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.990676+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:27.990676+0000","last_peered":"2024-09-16T10:31:27.990676+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:27.990676+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13877248,"num_objects":3388,"num_object_clones":0,"num_object_copies":6776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13608,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"46'17547","reported_seq":17954,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.885116+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:27.885116+0000","last_peered":"2024-09-16T10:31:27.885116+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:27.885116+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13815808,"num_objects":3373,"num_object_clones":0,"num_object_copies":6746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13789,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-914}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"46'17039","reported_seq":17218,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.002249+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:28.002249+0000","last_peered":"2024-09-16T10:31:28.002249+0000","last_clean":"2024-09-16T10:31:28.002249+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:28.002249+0000","last_undegraded":"2024-09-16T10:31:28.002249+0000","last_fullsized":"2024-09-16T10:31:28.002249+0000","mapping_epoch":33,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13725696,"num_objects":3351,"num_object_clones":0,"num_object_copies":6702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12842,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17136","reported_seq":25645,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.066101+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:32.066101+0000","last_peered":"2024-09-16T10:31:32.066101+0000","last_clean":"2024-09-16T10:31:32.066101+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:32.066101+0000","last_undegraded":"2024-09-16T10:31:32.066101+0000","last_fullsized":"2024-09-16T10:31:32.066101+0000","mapping_epoch":43,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13620166,"num_objects":3327,"num_object_clones":0,"num_object_copies":6654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17134,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"46'17606","reported_seq":17883,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.968884+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:27.968884+0000","last_peered":"2024-09-16T10:31:27.968884+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:27.968884+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14066630,"num_objects":3436,"num_object_clones":0,"num_object_copies":6872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17604,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-901}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":92,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-16T10:29:03.969392+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:29:03.969392+0000","last_peered":"2024-09-16T10:29:03.969392+0000","last_clean":"2024-09-16T10:29:03.969392+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:29:03.969392+0000","last_undegraded":"2024-09-16T10:29:03.969392+0000","last_fullsized":"2024-09-16T10:29:03.969392+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17141","reported_seq":17341,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.151544+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:32.151544+0000","last_peered":"2024-09-16T10:31:32.151544+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:32.151544+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13427654,"num_objects":3280,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17139,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-953}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17285","reported_seq":25773,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.121682+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:32.121682+0000","last_peered":"2024-09-16T10:31:32.121682+0000","last_clean":"2024-09-16T10:31:32.121682+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:32.121682+0000","last_undegraded":"2024-09-16T10:31:32.121682+0000","last_fullsized":"2024-09-16T10:31:32.121682+0000","mapping_epoch":47,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13480902,"num_objects":3293,"num_object_clones":0,"num_object_copies":6586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17283,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"46'16754","reported_seq":17005,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:28.009945+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:28.009945+0000","last_peered":"2024-09-16T10:31:28.009945+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:28.009945+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6700","ondisk_log_start":"42'6700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13435363,"num_objects":3281,"num_object_clones":0,"num_object_copies":6562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16753,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"46'17510","reported_seq":17787,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.961084+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:27.961084+0000","last_peered":"2024-09-16T10:31:27.961084+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:27.961084+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13944233,"num_objects":3407,"num_object_clones":0,"num_object_copies":6814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17507,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-860}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"46'16844","reported_seq":17102,"reported_epoch":46,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:28.045617+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:28.045617+0000","last_peered":"2024-09-16T10:31:28.045617+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:28.045617+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13501382,"num_objects":3298,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16842,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-785}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"46'17518","reported_seq":17761,"reported_epoch":46,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:27.987313+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:27.987313+0000","last_peered":"2024-09-16T10:31:27.987313+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:27.987313+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13838220,"num_objects":3382,"num_object_clones":0,"num_object_copies":6764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17515,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":218178062,"num_objects":53283,"num_object_clones":0,"num_object_copies":106566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":38057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53283,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253022,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13289,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":643325952,"data_stored":643249161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160657,"ondisk_log_size":160657,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018989,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570784,"kb_used_data":242200,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93801056,"statfs":{"total":96636764160,"available":96052281344,"internally_reserved":0,"allocated":248012800,"data_stored":247734154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":5.4379999999999997},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":0.30199999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.0129999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.0619999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691933,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515528,"kb_used_data":159036,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93856312,"statfs":{"total":96636764160,"available":96108863488,"internally_reserved":0,"allocated":162852864,"data_stored":162516410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":365031082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.8849999999999998},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":2.855}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.444},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.2410000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215215,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":648384,"kb_used_data":229620,"kb_used_omap":4,"kb_used_meta":418747,"kb_avail":93723456,"statfs":{"total":96636764160,"available":95972818944,"internally_reserved":0,"allocated":235130880,"data_stored":234870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":428797608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,1,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.375},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":9.734}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":1.7410000000000001},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.34599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234622976,"data_stored":234619363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161247232,"data_stored":161181739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247455744,"data_stored":247448059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:36.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:36.742 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:36.742 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:36.766 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-16T10:31:36.728196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":212914721,"num_objects":52000,"num_object_clones":0,"num_object_copies":104000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254309,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13420,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1716176,"kb_used_data":624628,"kb_used_omap":12,"kb_used_meta":1091507,"kb_avail":281399344,"statfs":{"total":289910292480,"available":288152928256,"internally_reserved":0,"allocated":639619072,"data_stored":638733012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1117703170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,9,6,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8675328,"num_objects":-2118,"num_object_clones":0,"num_object_copies":-4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2118,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.437947"},"pg_stats":[{"pgid":"2.f","version":"48'17242","reported_seq":25709,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.057720+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:33.057720+0000","last_peered":"2024-09-16T10:31:33.057720+0000","last_clean":"2024-09-16T10:31:33.057720+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:33.057720+0000","last_undegraded":"2024-09-16T10:31:33.057720+0000","last_fullsized":"2024-09-16T10:31:33.057720+0000","mapping_epoch":45,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13213696,"num_objects":3226,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17242,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17267","reported_seq":17430,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.126039+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:32.126039+0000","last_peered":"2024-09-16T10:31:32.126039+0000","last_clean":"2024-09-16T10:31:32.126039+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:32.126039+0000","last_undegraded":"2024-09-16T10:31:32.126039+0000","last_fullsized":"2024-09-16T10:31:32.126039+0000","mapping_epoch":34,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13086720,"num_objects":3195,"num_object_clones":0,"num_object_copies":6390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17336","reported_seq":17591,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.208208+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:32.208208+0000","last_peered":"2024-09-16T10:31:32.208208+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:32.208208+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13627416,"num_objects":3328,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13393,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17238","reported_seq":17300,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.143990+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:32.143990+0000","last_peered":"2024-09-16T10:31:32.143990+0000","last_clean":"2024-09-16T10:31:32.143990+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:32.143990+0000","last_undegraded":"2024-09-16T10:31:32.143990+0000","last_fullsized":"2024-09-16T10:31:32.143990+0000","mapping_epoch":34,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13205504,"num_objects":3224,"num_object_clones":0,"num_object_copies":6448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17238,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17263","reported_seq":17426,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.959948+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:32.959948+0000","last_peered":"2024-09-16T10:31:32.959948+0000","last_clean":"2024-09-16T10:31:32.959948+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:32.959948+0000","last_undegraded":"2024-09-16T10:31:32.959948+0000","last_fullsized":"2024-09-16T10:31:32.959948+0000","mapping_epoch":33,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13279232,"num_objects":3242,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13905,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17118","reported_seq":17453,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.059805+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:33.059805+0000","last_peered":"2024-09-16T10:31:33.059805+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:33.059805+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13324288,"num_objects":3253,"num_object_clones":0,"num_object_copies":6506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13743,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'17674","reported_seq":18086,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.957135+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:32.957135+0000","last_peered":"2024-09-16T10:31:32.957135+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:32.957135+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13295616,"num_objects":3246,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13916,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1041}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17198","reported_seq":17379,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.071554+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:33.071554+0000","last_peered":"2024-09-16T10:31:33.071554+0000","last_clean":"2024-09-16T10:31:33.071554+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:33.071554+0000","last_undegraded":"2024-09-16T10:31:33.071554+0000","last_fullsized":"2024-09-16T10:31:33.071554+0000","mapping_epoch":33,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13074432,"num_objects":3192,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13001,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17136","reported_seq":25645,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.066101+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:32.066101+0000","last_peered":"2024-09-16T10:31:32.066101+0000","last_clean":"2024-09-16T10:31:32.066101+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:32.066101+0000","last_undegraded":"2024-09-16T10:31:32.066101+0000","last_fullsized":"2024-09-16T10:31:32.066101+0000","mapping_epoch":43,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13620166,"num_objects":3327,"num_object_clones":0,"num_object_copies":6654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17134,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'17702","reported_seq":17981,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.056491+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:33.056491+0000","last_peered":"2024-09-16T10:31:33.056491+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:33.056491+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13673414,"num_objects":3340,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17700,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-997}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17141","reported_seq":17341,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.151544+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:32.151544+0000","last_peered":"2024-09-16T10:31:32.151544+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:32.151544+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13427654,"num_objects":3280,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17139,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-953}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17285","reported_seq":25773,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.121682+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:32.121682+0000","last_peered":"2024-09-16T10:31:32.121682+0000","last_clean":"2024-09-16T10:31:32.121682+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:32.121682+0000","last_undegraded":"2024-09-16T10:31:32.121682+0000","last_fullsized":"2024-09-16T10:31:32.121682+0000","mapping_epoch":47,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13480902,"num_objects":3293,"num_object_clones":0,"num_object_copies":6586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17283,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'16879","reported_seq":17133,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.873009+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:32.873009+0000","last_peered":"2024-09-16T10:31:32.873009+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:32.873009+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12923363,"num_objects":3156,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16878,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'17657","reported_seq":17935,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.015177+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:33.015177+0000","last_peered":"2024-09-16T10:31:33.015177+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:33.015177+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13342121,"num_objects":3260,"num_object_clones":0,"num_object_copies":6520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17654,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'16969","reported_seq":17487,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.075412+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:33.075412+0000","last_peered":"2024-09-16T10:31:33.075412+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:33.075412+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12989382,"num_objects":3173,"num_object_clones":0,"num_object_copies":6346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16967,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-780}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'17637","reported_seq":17882,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.025902+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:33.025902+0000","last_peered":"2024-09-16T10:31:33.025902+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:33.025902+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13350796,"num_objects":3263,"num_object_clones":0,"num_object_copies":6526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17634,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":212914702,"num_objects":51998,"num_object_clones":0,"num_object_copies":103996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254307,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13419,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":636932096,"data_stored":636855305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018989,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570784,"kb_used_data":242200,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93801056,"statfs":{"total":96636764160,"available":96052281344,"internally_reserved":0,"allocated":248012800,"data_stored":247734154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":5.4379999999999997},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":0.30199999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.0129999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.0619999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691933,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515528,"kb_used_data":159036,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93856312,"statfs":{"total":96636764160,"available":96108863488,"internally_reserved":0,"allocated":162852864,"data_stored":162516410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":365031082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.8849999999999998},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":2.855}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.444},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.2410000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215216,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629864,"kb_used_data":223392,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93741976,"statfs":{"total":96636764160,"available":95991783424,"internally_reserved":0,"allocated":228753408,"data_stored":228482448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.4849999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.76700000000000002},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.948}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228229120,"data_stored":228225507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161247232,"data_stored":161181739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247455744,"data_stored":247448059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:36.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:37.302 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:37.304 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:37.329 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-16T10:31:36.728196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":212914721,"num_objects":52000,"num_object_clones":0,"num_object_copies":104000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254309,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13420,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1716176,"kb_used_data":624628,"kb_used_omap":12,"kb_used_meta":1091507,"kb_avail":281399344,"statfs":{"total":289910292480,"available":288152928256,"internally_reserved":0,"allocated":639619072,"data_stored":638733012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1117703170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,9,6,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8675328,"num_objects":-2118,"num_object_clones":0,"num_object_copies":-4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2118,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.437947"},"pg_stats":[{"pgid":"2.f","version":"48'17242","reported_seq":25709,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.057720+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:33.057720+0000","last_peered":"2024-09-16T10:31:33.057720+0000","last_clean":"2024-09-16T10:31:33.057720+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:33.057720+0000","last_undegraded":"2024-09-16T10:31:33.057720+0000","last_fullsized":"2024-09-16T10:31:33.057720+0000","mapping_epoch":45,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13213696,"num_objects":3226,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17242,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17267","reported_seq":17430,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.126039+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:32.126039+0000","last_peered":"2024-09-16T10:31:32.126039+0000","last_clean":"2024-09-16T10:31:32.126039+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:32.126039+0000","last_undegraded":"2024-09-16T10:31:32.126039+0000","last_fullsized":"2024-09-16T10:31:32.126039+0000","mapping_epoch":34,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13086720,"num_objects":3195,"num_object_clones":0,"num_object_copies":6390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17336","reported_seq":17591,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.208208+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:32.208208+0000","last_peered":"2024-09-16T10:31:32.208208+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:32.208208+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13627416,"num_objects":3328,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13393,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17238","reported_seq":17300,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.143990+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:32.143990+0000","last_peered":"2024-09-16T10:31:32.143990+0000","last_clean":"2024-09-16T10:31:32.143990+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:32.143990+0000","last_undegraded":"2024-09-16T10:31:32.143990+0000","last_fullsized":"2024-09-16T10:31:32.143990+0000","mapping_epoch":34,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13205504,"num_objects":3224,"num_object_clones":0,"num_object_copies":6448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17238,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17263","reported_seq":17426,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.959948+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:32.959948+0000","last_peered":"2024-09-16T10:31:32.959948+0000","last_clean":"2024-09-16T10:31:32.959948+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:32.959948+0000","last_undegraded":"2024-09-16T10:31:32.959948+0000","last_fullsized":"2024-09-16T10:31:32.959948+0000","mapping_epoch":33,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13279232,"num_objects":3242,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13905,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17118","reported_seq":17453,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.059805+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:33.059805+0000","last_peered":"2024-09-16T10:31:33.059805+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:33.059805+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13324288,"num_objects":3253,"num_object_clones":0,"num_object_copies":6506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13743,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'17674","reported_seq":18086,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.957135+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:32.957135+0000","last_peered":"2024-09-16T10:31:32.957135+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:32.957135+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13295616,"num_objects":3246,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13916,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1041}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17198","reported_seq":17379,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.071554+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:33.071554+0000","last_peered":"2024-09-16T10:31:33.071554+0000","last_clean":"2024-09-16T10:31:33.071554+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:33.071554+0000","last_undegraded":"2024-09-16T10:31:33.071554+0000","last_fullsized":"2024-09-16T10:31:33.071554+0000","mapping_epoch":33,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13074432,"num_objects":3192,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13001,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17136","reported_seq":25645,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.066101+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:32.066101+0000","last_peered":"2024-09-16T10:31:32.066101+0000","last_clean":"2024-09-16T10:31:32.066101+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:32.066101+0000","last_undegraded":"2024-09-16T10:31:32.066101+0000","last_fullsized":"2024-09-16T10:31:32.066101+0000","mapping_epoch":43,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13620166,"num_objects":3327,"num_object_clones":0,"num_object_copies":6654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17134,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'17702","reported_seq":17981,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.056491+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:33.056491+0000","last_peered":"2024-09-16T10:31:33.056491+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:33.056491+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13673414,"num_objects":3340,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17700,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-997}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17141","reported_seq":17341,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.151544+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:32.151544+0000","last_peered":"2024-09-16T10:31:32.151544+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:32.151544+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13427654,"num_objects":3280,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17139,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-953}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17285","reported_seq":25773,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.121682+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:32.121682+0000","last_peered":"2024-09-16T10:31:32.121682+0000","last_clean":"2024-09-16T10:31:32.121682+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:32.121682+0000","last_undegraded":"2024-09-16T10:31:32.121682+0000","last_fullsized":"2024-09-16T10:31:32.121682+0000","mapping_epoch":47,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13480902,"num_objects":3293,"num_object_clones":0,"num_object_copies":6586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17283,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'16879","reported_seq":17133,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.873009+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:32.873009+0000","last_peered":"2024-09-16T10:31:32.873009+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:32.873009+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12923363,"num_objects":3156,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16878,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'17657","reported_seq":17935,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.015177+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:33.015177+0000","last_peered":"2024-09-16T10:31:33.015177+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:33.015177+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13342121,"num_objects":3260,"num_object_clones":0,"num_object_copies":6520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17654,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'16969","reported_seq":17487,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.075412+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:33.075412+0000","last_peered":"2024-09-16T10:31:33.075412+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:33.075412+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12989382,"num_objects":3173,"num_object_clones":0,"num_object_copies":6346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16967,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-780}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'17637","reported_seq":17882,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.025902+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:33.025902+0000","last_peered":"2024-09-16T10:31:33.025902+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:33.025902+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13350796,"num_objects":3263,"num_object_clones":0,"num_object_copies":6526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17634,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":212914702,"num_objects":51998,"num_object_clones":0,"num_object_copies":103996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254307,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13419,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":636932096,"data_stored":636855305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018989,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570784,"kb_used_data":242200,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93801056,"statfs":{"total":96636764160,"available":96052281344,"internally_reserved":0,"allocated":248012800,"data_stored":247734154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":5.4379999999999997},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":0.30199999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.0129999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.0619999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691933,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515528,"kb_used_data":159036,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93856312,"statfs":{"total":96636764160,"available":96108863488,"internally_reserved":0,"allocated":162852864,"data_stored":162516410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":365031082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.8849999999999998},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":2.855}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.444},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.2410000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215216,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629864,"kb_used_data":223392,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93741976,"statfs":{"total":96636764160,"available":95991783424,"internally_reserved":0,"allocated":228753408,"data_stored":228482448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.4849999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.76700000000000002},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.948}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228229120,"data_stored":228225507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161247232,"data_stored":161181739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247455744,"data_stored":247448059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:37.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:37.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'17336", 'reported_seq': 17591, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:32.208208+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:32.208208+0000', 'last_peered': '2024-09-16T10:31:32.208208+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:32.208208+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'7300", 'ondisk_log_start': "42'7300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13627416, 'num_objects': 3328, 'num_object_clones': 0, 'num_object_copies': 6656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3328, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13393, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:37.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:37.330 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'17118", 'reported_seq': 17453, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:33.059805+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:33.059805+0000', 'last_peered': '2024-09-16T10:31:33.059805+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:33.059805+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'7100", 'ondisk_log_start': "42'7100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13324288, 'num_objects': 3253, 'num_object_clones': 0, 'num_object_copies': 6506, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3253, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13743, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:37.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:37.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'17674", 'reported_seq': 18086, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:32.957135+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:32.957135+0000', 'last_peered': '2024-09-16T10:31:32.957135+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:32.957135+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'7600", 'ondisk_log_start': "42'7600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13295616, 'num_objects': 3246, 'num_object_clones': 0, 'num_object_copies': 6492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3246, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13916, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1041}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:37.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:37.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'17702", 'reported_seq': 17981, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:33.056491+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:33.056491+0000', 'last_peered': '2024-09-16T10:31:33.056491+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:33.056491+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'7700", 'ondisk_log_start': "42'7700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13673414, 'num_objects': 3340, 'num_object_clones': 0, 'num_object_copies': 6680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17700, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -997}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:37.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:37.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'17141", 'reported_seq': 17341, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:32.151544+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:32.151544+0000', 'last_peered': '2024-09-16T10:31:32.151544+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:32.151544+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'7100", 'ondisk_log_start': "42'7100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13427654, 'num_objects': 3280, 'num_object_clones': 0, 'num_object_copies': 6560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17139, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -953}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:37.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:37.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'16879", 'reported_seq': 17133, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:32.873009+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:32.873009+0000', 'last_peered': '2024-09-16T10:31:32.873009+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:32.873009+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'6800", 'ondisk_log_start': "42'6800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12923363, 'num_objects': 3156, 'num_object_clones': 0, 'num_object_copies': 6312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16878, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -919}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:37.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:37.331 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'17657", 'reported_seq': 17935, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:33.015177+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:33.015177+0000', 'last_peered': '2024-09-16T10:31:33.015177+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:33.015177+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'7600", 'ondisk_log_start': "42'7600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13342121, 'num_objects': 3260, 'num_object_clones': 0, 'num_object_copies': 6520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17654, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:37.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:37.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'16969", 'reported_seq': 17487, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:33.075412+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:31:33.075412+0000', 'last_peered': '2024-09-16T10:31:33.075412+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:33.075412+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'6900", 'ondisk_log_start': "42'6900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12989382, 'num_objects': 3173, 'num_object_clones': 0, 'num_object_copies': 6346, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3953, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16967, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 218, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -780}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:37.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:37.332 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'17637", 'reported_seq': 17882, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:33.025902+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:33.025902+0000', 'last_peered': '2024-09-16T10:31:33.025902+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:33.025902+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'7600", 'ondisk_log_start': "42'7600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13350796, 'num_objects': 3263, 'num_object_clones': 0, 'num_object_copies': 6526, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17634, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1059}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:37.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:38.059 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:38.060 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37927/104000 objects degraded (36.468%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":586,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":52000,"data_bytes":212914721,"bytes_used":1757364224,"bytes_avail":288152928256,"bytes_total":289910292480,"degraded_objects":37927,"degraded_total":104000,"degraded_ratio":0.36468269230769229,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":251},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:31:38.085 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:31:38.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:38.654 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:38.655 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:38.682 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-16T10:31:36.728196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":212914721,"num_objects":52000,"num_object_clones":0,"num_object_copies":104000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52000,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254309,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13420,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1716176,"kb_used_data":624628,"kb_used_omap":12,"kb_used_meta":1091507,"kb_avail":281399344,"statfs":{"total":289910292480,"available":288152928256,"internally_reserved":0,"allocated":639619072,"data_stored":638733012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1117703170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,9,6,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8675328,"num_objects":-2118,"num_object_clones":0,"num_object_copies":-4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2118,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.437947"},"pg_stats":[{"pgid":"2.f","version":"48'17242","reported_seq":25709,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.057720+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:33.057720+0000","last_peered":"2024-09-16T10:31:33.057720+0000","last_clean":"2024-09-16T10:31:33.057720+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:33.057720+0000","last_undegraded":"2024-09-16T10:31:33.057720+0000","last_fullsized":"2024-09-16T10:31:33.057720+0000","mapping_epoch":45,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13213696,"num_objects":3226,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17242,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17267","reported_seq":17430,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.126039+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:32.126039+0000","last_peered":"2024-09-16T10:31:32.126039+0000","last_clean":"2024-09-16T10:31:32.126039+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:32.126039+0000","last_undegraded":"2024-09-16T10:31:32.126039+0000","last_fullsized":"2024-09-16T10:31:32.126039+0000","mapping_epoch":34,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13086720,"num_objects":3195,"num_object_clones":0,"num_object_copies":6390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13480,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17336","reported_seq":17591,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.208208+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:32.208208+0000","last_peered":"2024-09-16T10:31:32.208208+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:32.208208+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13627416,"num_objects":3328,"num_object_clones":0,"num_object_copies":6656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13393,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17238","reported_seq":17300,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.143990+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:32.143990+0000","last_peered":"2024-09-16T10:31:32.143990+0000","last_clean":"2024-09-16T10:31:32.143990+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:32.143990+0000","last_undegraded":"2024-09-16T10:31:32.143990+0000","last_fullsized":"2024-09-16T10:31:32.143990+0000","mapping_epoch":34,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13205504,"num_objects":3224,"num_object_clones":0,"num_object_copies":6448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17238,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17263","reported_seq":17426,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.959948+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:32.959948+0000","last_peered":"2024-09-16T10:31:32.959948+0000","last_clean":"2024-09-16T10:31:32.959948+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:32.959948+0000","last_undegraded":"2024-09-16T10:31:32.959948+0000","last_fullsized":"2024-09-16T10:31:32.959948+0000","mapping_epoch":33,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13279232,"num_objects":3242,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13905,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17118","reported_seq":17453,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.059805+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:33.059805+0000","last_peered":"2024-09-16T10:31:33.059805+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:33.059805+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13324288,"num_objects":3253,"num_object_clones":0,"num_object_copies":6506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13743,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'17674","reported_seq":18086,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.957135+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:32.957135+0000","last_peered":"2024-09-16T10:31:32.957135+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:32.957135+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13295616,"num_objects":3246,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13916,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1041}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17198","reported_seq":17379,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:33.071554+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:33.071554+0000","last_peered":"2024-09-16T10:31:33.071554+0000","last_clean":"2024-09-16T10:31:33.071554+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:33.071554+0000","last_undegraded":"2024-09-16T10:31:33.071554+0000","last_fullsized":"2024-09-16T10:31:33.071554+0000","mapping_epoch":33,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13074432,"num_objects":3192,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13001,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17136","reported_seq":25645,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.066101+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:32.066101+0000","last_peered":"2024-09-16T10:31:32.066101+0000","last_clean":"2024-09-16T10:31:32.066101+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:32.066101+0000","last_undegraded":"2024-09-16T10:31:32.066101+0000","last_fullsized":"2024-09-16T10:31:32.066101+0000","mapping_epoch":43,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13620166,"num_objects":3327,"num_object_clones":0,"num_object_copies":6654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17134,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'17702","reported_seq":17981,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.056491+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:33.056491+0000","last_peered":"2024-09-16T10:31:33.056491+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:33.056491+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13673414,"num_objects":3340,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17700,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-997}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17141","reported_seq":17341,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.151544+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:32.151544+0000","last_peered":"2024-09-16T10:31:32.151544+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:32.151544+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13427654,"num_objects":3280,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17139,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-953}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17285","reported_seq":25773,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:32.121682+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:32.121682+0000","last_peered":"2024-09-16T10:31:32.121682+0000","last_clean":"2024-09-16T10:31:32.121682+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:32.121682+0000","last_undegraded":"2024-09-16T10:31:32.121682+0000","last_fullsized":"2024-09-16T10:31:32.121682+0000","mapping_epoch":47,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13480902,"num_objects":3293,"num_object_clones":0,"num_object_copies":6586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17283,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'16879","reported_seq":17133,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:32.873009+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:32.873009+0000","last_peered":"2024-09-16T10:31:32.873009+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:32.873009+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'6800","ondisk_log_start":"42'6800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12923363,"num_objects":3156,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16878,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-919}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'17657","reported_seq":17935,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.015177+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:33.015177+0000","last_peered":"2024-09-16T10:31:33.015177+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:33.015177+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13342121,"num_objects":3260,"num_object_clones":0,"num_object_copies":6520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17654,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'16969","reported_seq":17487,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.075412+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:33.075412+0000","last_peered":"2024-09-16T10:31:33.075412+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:33.075412+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'6900","ondisk_log_start":"42'6900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12989382,"num_objects":3173,"num_object_clones":0,"num_object_copies":6346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16967,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-780}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'17637","reported_seq":17882,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:33.025902+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:33.025902+0000","last_peered":"2024-09-16T10:31:33.025902+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:33.025902+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13350796,"num_objects":3263,"num_object_clones":0,"num_object_copies":6526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17634,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1059}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":212914702,"num_objects":51998,"num_object_clones":0,"num_object_copies":103996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37927,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254307,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13419,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":636932096,"data_stored":636855305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018989,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570784,"kb_used_data":242200,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93801056,"statfs":{"total":96636764160,"available":96052281344,"internally_reserved":0,"allocated":248012800,"data_stored":247734154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":5.4379999999999997},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":0.30199999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.0129999999999999},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.0619999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691933,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515528,"kb_used_data":159036,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93856312,"statfs":{"total":96636764160,"available":96108863488,"internally_reserved":0,"allocated":162852864,"data_stored":162516410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":365031082},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.8849999999999998},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":2.855}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.444},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.2410000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215216,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":629864,"kb_used_data":223392,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93741976,"statfs":{"total":96636764160,"available":95991783424,"internally_reserved":0,"allocated":228753408,"data_stored":228482448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":416214696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.4849999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.76700000000000002},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.948}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228229120,"data_stored":228225507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161247232,"data_stored":161181739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":247455744,"data_stored":247448059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:41.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:42.253 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:42.279 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-16T10:31:40.729130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":203502113,"num_objects":49702,"num_object_clones":0,"num_object_copies":99404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49702,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":256607,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13504,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160943,"ondisk_log_size":160943,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1701792,"kb_used_data":606148,"kb_used_omap":12,"kb_used_meta":1095603,"kb_avail":281413728,"statfs":{"total":289910292480,"available":288167657472,"internally_reserved":0,"allocated":620695552,"data_stored":619809492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1121897474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,5,7,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18087936,"num_objects":-4416,"num_object_clones":0,"num_object_copies":-8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.429278"},"pg_stats":[{"pgid":"2.f","version":"48'17401","reported_seq":25867,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:38.115127+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:38.115127+0000","last_peered":"2024-09-16T10:31:38.115127+0000","last_clean":"2024-09-16T10:31:38.115127+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:38.115127+0000","last_undegraded":"2024-09-16T10:31:38.115127+0000","last_fullsized":"2024-09-16T10:31:38.115127+0000","mapping_epoch":45,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12562432,"num_objects":3067,"num_object_clones":0,"num_object_copies":6134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17401,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17397","reported_seq":17560,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:37.171718+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:37.171718+0000","last_peered":"2024-09-16T10:31:37.171718+0000","last_clean":"2024-09-16T10:31:37.171718+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:37.171718+0000","last_undegraded":"2024-09-16T10:31:37.171718+0000","last_fullsized":"2024-09-16T10:31:37.171718+0000","mapping_epoch":34,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12554240,"num_objects":3065,"num_object_clones":0,"num_object_copies":6130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13610,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17490","reported_seq":17746,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:37.145260+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:37.145260+0000","last_peered":"2024-09-16T10:31:37.145260+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:37.145260+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12996632,"num_objects":3174,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13547,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1129}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17378","reported_seq":17440,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:37.172009+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:37.172009+0000","last_peered":"2024-09-16T10:31:37.172009+0000","last_clean":"2024-09-16T10:31:37.172009+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:37.172009+0000","last_undegraded":"2024-09-16T10:31:37.172009+0000","last_fullsized":"2024-09-16T10:31:37.172009+0000","mapping_epoch":34,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12632064,"num_objects":3084,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17378,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17406","reported_seq":17569,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:37.992035+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:37.992035+0000","last_peered":"2024-09-16T10:31:37.992035+0000","last_clean":"2024-09-16T10:31:37.992035+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:37.992035+0000","last_undegraded":"2024-09-16T10:31:37.992035+0000","last_fullsized":"2024-09-16T10:31:37.992035+0000","mapping_epoch":33,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12693504,"num_objects":3099,"num_object_clones":0,"num_object_copies":6198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14048,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17255","reported_seq":17591,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.029194+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:38.029194+0000","last_peered":"2024-09-16T10:31:38.029194+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:38.029194+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12763136,"num_objects":3116,"num_object_clones":0,"num_object_copies":6232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13880,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'17844","reported_seq":18256,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:37.896410+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:37.896410+0000","last_peered":"2024-09-16T10:31:37.896410+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:37.896410+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12599296,"num_objects":3076,"num_object_clones":0,"num_object_copies":6152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14086,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17335","reported_seq":17515,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:38.049547+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:38.049547+0000","last_peered":"2024-09-16T10:31:38.049547+0000","last_clean":"2024-09-16T10:31:38.049547+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:38.049547+0000","last_undegraded":"2024-09-16T10:31:38.049547+0000","last_fullsized":"2024-09-16T10:31:38.049547+0000","mapping_epoch":33,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12513280,"num_objects":3055,"num_object_clones":0,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13138,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17281","reported_seq":25790,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:37.083775+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:37.083775+0000","last_peered":"2024-09-16T10:31:37.083775+0000","last_clean":"2024-09-16T10:31:37.083775+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:37.083775+0000","last_undegraded":"2024-09-16T10:31:37.083775+0000","last_fullsized":"2024-09-16T10:31:37.083775+0000","mapping_epoch":43,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13026246,"num_objects":3182,"num_object_clones":0,"num_object_copies":6364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17279,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'17838","reported_seq":18117,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.086962+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:38.086962+0000","last_peered":"2024-09-16T10:31:38.086962+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:38.086962+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13116358,"num_objects":3204,"num_object_clones":0,"num_object_copies":6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17836,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1133}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17278","reported_seq":17478,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:37.076302+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:37.076302+0000","last_peered":"2024-09-16T10:31:37.076302+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:37.076302+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12866502,"num_objects":3143,"num_object_clones":0,"num_object_copies":6286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17276,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1090}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17425","reported_seq":25913,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:37.134447+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:37.134447+0000","last_peered":"2024-09-16T10:31:37.134447+0000","last_clean":"2024-09-16T10:31:37.134447+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:37.134447+0000","last_undegraded":"2024-09-16T10:31:37.134447+0000","last_fullsized":"2024-09-16T10:31:37.134447+0000","mapping_epoch":47,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12907462,"num_objects":3153,"num_object_clones":0,"num_object_copies":6306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17423,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17039","reported_seq":17292,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.100255+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:38.100255+0000","last_peered":"2024-09-16T10:31:38.100255+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:38.100255+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12268003,"num_objects":2996,"num_object_clones":0,"num_object_copies":5992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17038,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'17795","reported_seq":18074,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.097269+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:38.097269+0000","last_peered":"2024-09-16T10:31:38.097269+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:38.097269+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12776873,"num_objects":3122,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17792,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17088","reported_seq":17776,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.036599+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:38.036599+0000","last_peered":"2024-09-16T10:31:38.036599+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:38.036599+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12501958,"num_objects":3054,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17086,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'17790","reported_seq":18035,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.051510+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:38.051510+0000","last_peered":"2024-09-16T10:31:38.051510+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:38.051510+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12724108,"num_objects":3110,"num_object_clones":0,"num_object_copies":6220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17787,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":203502094,"num_objects":49700,"num_object_clones":0,"num_object_copies":99400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":256605,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13503,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":617951232,"data_stored":617874441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160940,"ondisk_log_size":160940,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018990,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553964,"kb_used_data":237668,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93817876,"statfs":{"total":96636764160,"available":96069505024,"internally_reserved":0,"allocated":243372032,"data_stored":243093386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":323874480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":3.0369999999999999},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":2.964}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.32300000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691934,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516720,"kb_used_data":152036,"kb_used_omap":4,"kb_used_meta":364667,"kb_avail":93855120,"statfs":{"total":96636764160,"available":96107642880,"internally_reserved":0,"allocated":155684864,"data_stored":155348410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":373419690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,3,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.2070000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":8.7569999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.8959999999999999},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.54}]}]},{"osd":0,"up_from":24,"seq":103079215217,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631108,"kb_used_data":216444,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93740732,"statfs":{"total":96636764160,"available":95990509568,"internally_reserved":0,"allocated":221638656,"data_stored":221367696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424603304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":6.6429999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.54300000000000004},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221118464,"data_stored":221114851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154017792,"data_stored":153952299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":242814976,"data_stored":242807291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:42.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:42.844 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:42.845 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:42.867 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-16T10:31:42.729566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":200012321,"num_objects":48850,"num_object_clones":0,"num_object_copies":97700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":257459,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13504,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1699988,"kb_used_data":594492,"kb_used_omap":12,"kb_used_meta":1105459,"kb_avail":281415532,"statfs":{"total":289910292480,"available":288169504768,"internally_reserved":0,"allocated":608759808,"data_stored":607873748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1131990018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,5,10,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21577728,"num_objects":-5268,"num_object_clones":0,"num_object_copies":-10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5268,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002793"},"pg_stats":[{"pgid":"2.f","version":"48'17401","reported_seq":25867,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:38.115127+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:38.115127+0000","last_peered":"2024-09-16T10:31:38.115127+0000","last_clean":"2024-09-16T10:31:38.115127+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:38.115127+0000","last_undegraded":"2024-09-16T10:31:38.115127+0000","last_fullsized":"2024-09-16T10:31:38.115127+0000","mapping_epoch":45,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12562432,"num_objects":3067,"num_object_clones":0,"num_object_copies":6134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17401,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17536","reported_seq":17697,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.147713+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:42.147713+0000","last_peered":"2024-09-16T10:31:42.147713+0000","last_clean":"2024-09-16T10:31:42.147713+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:42.147713+0000","last_undegraded":"2024-09-16T10:31:42.147713+0000","last_fullsized":"2024-09-16T10:31:42.147713+0000","mapping_epoch":34,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13749,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17629","reported_seq":17885,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:42.143340+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:42.143340+0000","last_peered":"2024-09-16T10:31:42.143340+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:42.143340+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12427288,"num_objects":3035,"num_object_clones":0,"num_object_copies":6070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13686,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1268}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17521","reported_seq":17583,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.147898+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:42.147898+0000","last_peered":"2024-09-16T10:31:42.147898+0000","last_clean":"2024-09-16T10:31:42.147898+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:42.147898+0000","last_undegraded":"2024-09-16T10:31:42.147898+0000","last_fullsized":"2024-09-16T10:31:42.147898+0000","mapping_epoch":34,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12046336,"num_objects":2941,"num_object_clones":0,"num_object_copies":5882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17521,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17406","reported_seq":17569,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:37.992035+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:37.992035+0000","last_peered":"2024-09-16T10:31:37.992035+0000","last_clean":"2024-09-16T10:31:37.992035+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:37.992035+0000","last_undegraded":"2024-09-16T10:31:37.992035+0000","last_fullsized":"2024-09-16T10:31:37.992035+0000","mapping_epoch":33,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12693504,"num_objects":3099,"num_object_clones":0,"num_object_copies":6198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14048,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17255","reported_seq":17591,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.029194+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:38.029194+0000","last_peered":"2024-09-16T10:31:38.029194+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:38.029194+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12763136,"num_objects":3116,"num_object_clones":0,"num_object_copies":6232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13880,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'17844","reported_seq":18256,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:37.896410+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:37.896410+0000","last_peered":"2024-09-16T10:31:37.896410+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:37.896410+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12599296,"num_objects":3076,"num_object_clones":0,"num_object_copies":6152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14086,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17335","reported_seq":17515,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:38.049547+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:38.049547+0000","last_peered":"2024-09-16T10:31:38.049547+0000","last_clean":"2024-09-16T10:31:38.049547+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:38.049547+0000","last_undegraded":"2024-09-16T10:31:38.049547+0000","last_fullsized":"2024-09-16T10:31:38.049547+0000","mapping_epoch":33,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12513280,"num_objects":3055,"num_object_clones":0,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13138,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17407","reported_seq":25916,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.118601+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:42.118601+0000","last_peered":"2024-09-16T10:31:42.118601+0000","last_clean":"2024-09-16T10:31:42.118601+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:42.118601+0000","last_undegraded":"2024-09-16T10:31:42.118601+0000","last_fullsized":"2024-09-16T10:31:42.118601+0000","mapping_epoch":43,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12510150,"num_objects":3056,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17405,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'17838","reported_seq":18117,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.086962+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:38.086962+0000","last_peered":"2024-09-16T10:31:38.086962+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:38.086962+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13116358,"num_objects":3204,"num_object_clones":0,"num_object_copies":6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17836,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1133}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17429","reported_seq":17629,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:42.192103+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:42.192103+0000","last_peered":"2024-09-16T10:31:42.192103+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:42.192103+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12248006,"num_objects":2992,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17427,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1241}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17579","reported_seq":26067,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.179417+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:42.179417+0000","last_peered":"2024-09-16T10:31:42.179417+0000","last_clean":"2024-09-16T10:31:42.179417+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:42.179417+0000","last_undegraded":"2024-09-16T10:31:42.179417+0000","last_fullsized":"2024-09-16T10:31:42.179417+0000","mapping_epoch":47,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12276678,"num_objects":2999,"num_object_clones":0,"num_object_copies":5998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17577,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17039","reported_seq":17292,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.100255+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:38.100255+0000","last_peered":"2024-09-16T10:31:38.100255+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:38.100255+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12268003,"num_objects":2996,"num_object_clones":0,"num_object_copies":5992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17038,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'17795","reported_seq":18074,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.097269+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:38.097269+0000","last_peered":"2024-09-16T10:31:38.097269+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:38.097269+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12776873,"num_objects":3122,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17792,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17088","reported_seq":17776,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.036599+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:38.036599+0000","last_peered":"2024-09-16T10:31:38.036599+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:38.036599+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12501958,"num_objects":3054,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17086,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'17790","reported_seq":18035,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.051510+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:38.051510+0000","last_peered":"2024-09-16T10:31:38.051510+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:38.051510+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12724108,"num_objects":3110,"num_object_clones":0,"num_object_copies":6220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17787,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":200012302,"num_objects":48848,"num_object_clones":0,"num_object_copies":97696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":257457,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13503,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":606031872,"data_stored":605955081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160692,"ondisk_log_size":160692,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018991,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553416,"kb_used_data":233024,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93818424,"statfs":{"total":96636764160,"available":96070066176,"internally_reserved":0,"allocated":238616576,"data_stored":238337930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":2.4470000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.9350000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.6520000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.5750000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691935,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515464,"kb_used_data":145024,"kb_used_omap":4,"kb_used_meta":370427,"kb_avail":93856376,"statfs":{"total":96636764160,"available":96108929024,"internally_reserved":0,"allocated":148504576,"data_stored":148168122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":379317930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.6019999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.86199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.4239999999999999},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.4470000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215217,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631108,"kb_used_data":216444,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93740732,"statfs":{"total":96636764160,"available":95990509568,"internally_reserved":0,"allocated":221638656,"data_stored":221367696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424603304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":6.6429999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.54300000000000004},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221118464,"data_stored":221114851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146784256,"data_stored":146718763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238129152,"data_stored":238121467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:42.868 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:42.869 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'17629", 'reported_seq': 17885, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:42.143340+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:42.143340+0000', 'last_peered': '2024-09-16T10:31:42.143340+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:42.143340+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'7600", 'ondisk_log_start': "42'7600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12427288, 'num_objects': 3035, 'num_object_clones': 0, 'num_object_copies': 6070, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3035, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13686, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1268}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:42.869 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:42.869 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'17255", 'reported_seq': 17591, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:38.029194+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:38.029194+0000', 'last_peered': '2024-09-16T10:31:38.029194+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:38.029194+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'7200", 'ondisk_log_start': "42'7200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12763136, 'num_objects': 3116, 'num_object_clones': 0, 'num_object_copies': 6232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13880, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1018}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:42.869 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:42.869 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'17844", 'reported_seq': 18256, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:37.896410+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:37.896410+0000', 'last_peered': '2024-09-16T10:31:37.896410+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:37.896410+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'7800", 'ondisk_log_start': "42'7800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12599296, 'num_objects': 3076, 'num_object_clones': 0, 'num_object_copies': 6152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14086, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1211}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:42.869 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:42.869 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'17838", 'reported_seq': 18117, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:38.086962+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:38.086962+0000', 'last_peered': '2024-09-16T10:31:38.086962+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:38.086962+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'7800", 'ondisk_log_start': "42'7800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13116358, 'num_objects': 3204, 'num_object_clones': 0, 'num_object_copies': 6408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17836, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1133}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:42.869 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:42.869 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'17429", 'reported_seq': 17629, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:42.192103+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:42.192103+0000', 'last_peered': '2024-09-16T10:31:42.192103+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:42.192103+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'7400", 'ondisk_log_start': "42'7400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12248006, 'num_objects': 2992, 'num_object_clones': 0, 'num_object_copies': 5984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17427, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1241}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:42.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:42.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'17039", 'reported_seq': 17292, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:38.100255+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:38.100255+0000', 'last_peered': '2024-09-16T10:31:38.100255+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:38.100255+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'7000", 'ondisk_log_start': "42'7000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12268003, 'num_objects': 2996, 'num_object_clones': 0, 'num_object_copies': 5992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17038, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:42.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:42.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'17795", 'reported_seq': 18074, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:38.097269+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:38.097269+0000', 'last_peered': '2024-09-16T10:31:38.097269+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:38.097269+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'7700", 'ondisk_log_start': "42'7700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12776873, 'num_objects': 3122, 'num_object_clones': 0, 'num_object_copies': 6244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17792, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1145}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:42.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:42.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'17088", 'reported_seq': 17776, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:38.036599+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:31:38.036599+0000', 'last_peered': '2024-09-16T10:31:38.036599+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:38.036599+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'7000", 'ondisk_log_start': "42'7000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12501958, 'num_objects': 3054, 'num_object_clones': 0, 'num_object_copies': 6108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3869, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17086, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 302, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:42.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:42.870 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'17790", 'reported_seq': 18035, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:38.051510+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:38.051510+0000', 'last_peered': '2024-09-16T10:31:38.051510+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:38.051510+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'7700", 'ondisk_log_start': "42'7700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12724108, 'num_objects': 3110, 'num_object_clones': 0, 'num_object_copies': 6220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17787, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1212}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:42.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:43.524 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:43.524 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37843/97700 objects degraded (38.734%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":592,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":48850,"data_bytes":200012321,"bytes_used":1740787712,"bytes_avail":288169504768,"bytes_total":289910292480,"degraded_objects":37843,"degraded_total":97700,"degraded_ratio":0.38733879222108497,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":217,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":438},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:31:43.543 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:31:43.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:44.110 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:44.110 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:44.143 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-16T10:31:42.729566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":200012321,"num_objects":48850,"num_object_clones":0,"num_object_copies":97700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":257459,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13504,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160695,"ondisk_log_size":160695,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1699988,"kb_used_data":594492,"kb_used_omap":12,"kb_used_meta":1105459,"kb_avail":281415532,"statfs":{"total":289910292480,"available":288169504768,"internally_reserved":0,"allocated":608759808,"data_stored":607873748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1131990018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,5,10,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21577728,"num_objects":-5268,"num_object_clones":0,"num_object_copies":-10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5268,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":217,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002793"},"pg_stats":[{"pgid":"2.f","version":"48'17401","reported_seq":25867,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:38.115127+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:38.115127+0000","last_peered":"2024-09-16T10:31:38.115127+0000","last_clean":"2024-09-16T10:31:38.115127+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:38.115127+0000","last_undegraded":"2024-09-16T10:31:38.115127+0000","last_fullsized":"2024-09-16T10:31:38.115127+0000","mapping_epoch":45,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12562432,"num_objects":3067,"num_object_clones":0,"num_object_copies":6134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17401,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17536","reported_seq":17697,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.147713+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:42.147713+0000","last_peered":"2024-09-16T10:31:42.147713+0000","last_clean":"2024-09-16T10:31:42.147713+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:42.147713+0000","last_undegraded":"2024-09-16T10:31:42.147713+0000","last_fullsized":"2024-09-16T10:31:42.147713+0000","mapping_epoch":34,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13749,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17629","reported_seq":17885,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:42.143340+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:42.143340+0000","last_peered":"2024-09-16T10:31:42.143340+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:42.143340+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12427288,"num_objects":3035,"num_object_clones":0,"num_object_copies":6070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13686,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1268}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17521","reported_seq":17583,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.147898+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:42.147898+0000","last_peered":"2024-09-16T10:31:42.147898+0000","last_clean":"2024-09-16T10:31:42.147898+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:42.147898+0000","last_undegraded":"2024-09-16T10:31:42.147898+0000","last_fullsized":"2024-09-16T10:31:42.147898+0000","mapping_epoch":34,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12046336,"num_objects":2941,"num_object_clones":0,"num_object_copies":5882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17521,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17406","reported_seq":17569,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:37.992035+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:37.992035+0000","last_peered":"2024-09-16T10:31:37.992035+0000","last_clean":"2024-09-16T10:31:37.992035+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:37.992035+0000","last_undegraded":"2024-09-16T10:31:37.992035+0000","last_fullsized":"2024-09-16T10:31:37.992035+0000","mapping_epoch":33,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12693504,"num_objects":3099,"num_object_clones":0,"num_object_copies":6198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14048,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17255","reported_seq":17591,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.029194+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:38.029194+0000","last_peered":"2024-09-16T10:31:38.029194+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:38.029194+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12763136,"num_objects":3116,"num_object_clones":0,"num_object_copies":6232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13880,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1018}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'17844","reported_seq":18256,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:37.896410+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:37.896410+0000","last_peered":"2024-09-16T10:31:37.896410+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:37.896410+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12599296,"num_objects":3076,"num_object_clones":0,"num_object_copies":6152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14086,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17335","reported_seq":17515,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:38.049547+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:38.049547+0000","last_peered":"2024-09-16T10:31:38.049547+0000","last_clean":"2024-09-16T10:31:38.049547+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:38.049547+0000","last_undegraded":"2024-09-16T10:31:38.049547+0000","last_fullsized":"2024-09-16T10:31:38.049547+0000","mapping_epoch":33,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12513280,"num_objects":3055,"num_object_clones":0,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13138,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17407","reported_seq":25916,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.118601+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:42.118601+0000","last_peered":"2024-09-16T10:31:42.118601+0000","last_clean":"2024-09-16T10:31:42.118601+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:42.118601+0000","last_undegraded":"2024-09-16T10:31:42.118601+0000","last_fullsized":"2024-09-16T10:31:42.118601+0000","mapping_epoch":43,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12510150,"num_objects":3056,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17405,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'17838","reported_seq":18117,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.086962+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:38.086962+0000","last_peered":"2024-09-16T10:31:38.086962+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:38.086962+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13116358,"num_objects":3204,"num_object_clones":0,"num_object_copies":6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17836,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1133}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17429","reported_seq":17629,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:42.192103+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:42.192103+0000","last_peered":"2024-09-16T10:31:42.192103+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:42.192103+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12248006,"num_objects":2992,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17427,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1241}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17579","reported_seq":26067,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.179417+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:42.179417+0000","last_peered":"2024-09-16T10:31:42.179417+0000","last_clean":"2024-09-16T10:31:42.179417+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:42.179417+0000","last_undegraded":"2024-09-16T10:31:42.179417+0000","last_fullsized":"2024-09-16T10:31:42.179417+0000","mapping_epoch":47,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12276678,"num_objects":2999,"num_object_clones":0,"num_object_copies":5998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17577,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17039","reported_seq":17292,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.100255+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:38.100255+0000","last_peered":"2024-09-16T10:31:38.100255+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:38.100255+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12268003,"num_objects":2996,"num_object_clones":0,"num_object_copies":5992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17038,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'17795","reported_seq":18074,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.097269+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:38.097269+0000","last_peered":"2024-09-16T10:31:38.097269+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:38.097269+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12776873,"num_objects":3122,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17792,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1145}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17088","reported_seq":17776,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.036599+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:38.036599+0000","last_peered":"2024-09-16T10:31:38.036599+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:38.036599+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7000","ondisk_log_start":"42'7000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12501958,"num_objects":3054,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17086,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'17790","reported_seq":18035,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:38.051510+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:38.051510+0000","last_peered":"2024-09-16T10:31:38.051510+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:38.051510+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12724108,"num_objects":3110,"num_object_clones":0,"num_object_copies":6220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17787,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":200012302,"num_objects":48848,"num_object_clones":0,"num_object_copies":97696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37843,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":257457,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13503,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":606031872,"data_stored":605955081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160692,"ondisk_log_size":160692,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018991,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553416,"kb_used_data":233024,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93818424,"statfs":{"total":96636764160,"available":96070066176,"internally_reserved":0,"allocated":238616576,"data_stored":238337930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":2.4470000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.9350000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.6520000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.5750000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691935,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515464,"kb_used_data":145024,"kb_used_omap":4,"kb_used_meta":370427,"kb_avail":93856376,"statfs":{"total":96636764160,"available":96108929024,"internally_reserved":0,"allocated":148504576,"data_stored":148168122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":379317930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.6019999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.86199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.4239999999999999},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.4470000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215217,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631108,"kb_used_data":216444,"kb_used_omap":4,"kb_used_meta":414651,"kb_avail":93740732,"statfs":{"total":96636764160,"available":95990509568,"internally_reserved":0,"allocated":221638656,"data_stored":221367696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":424603304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":6.6429999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.29799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.54300000000000004},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":221118464,"data_stored":221114851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146784256,"data_stored":146718763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238129152,"data_stored":238121467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:47.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:47.713 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:47.715 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:47.743 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T10:31:46.730517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":193831457,"num_objects":47341,"num_object_clones":0,"num_object_copies":94682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47341,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":258968,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13577,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1714912,"kb_used_data":587272,"kb_used_omap":12,"kb_used_meta":1127603,"kb_avail":281400608,"statfs":{"total":289910292480,"available":288154222592,"internally_reserved":0,"allocated":601366528,"data_stored":600480468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1154665474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,5,12,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19083264,"num_objects":-4659,"num_object_clones":0,"num_object_copies":-9318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4659,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002803"},"pg_stats":[{"pgid":"2.f","version":"48'17559","reported_seq":26027,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:43.206389+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:43.206389+0000","last_peered":"2024-09-16T10:31:43.206389+0000","last_clean":"2024-09-16T10:31:43.206389+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:43.206389+0000","last_undegraded":"2024-09-16T10:31:43.206389+0000","last_fullsized":"2024-09-16T10:31:43.206389+0000","mapping_epoch":45,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11915264,"num_objects":2909,"num_object_clones":0,"num_object_copies":5818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17559,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17536","reported_seq":17697,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.147713+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:42.147713+0000","last_peered":"2024-09-16T10:31:42.147713+0000","last_clean":"2024-09-16T10:31:42.147713+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:42.147713+0000","last_undegraded":"2024-09-16T10:31:42.147713+0000","last_fullsized":"2024-09-16T10:31:42.147713+0000","mapping_epoch":34,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13749,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17629","reported_seq":17885,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:42.143340+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:42.143340+0000","last_peered":"2024-09-16T10:31:42.143340+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:42.143340+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12427288,"num_objects":3035,"num_object_clones":0,"num_object_copies":6070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13686,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1268}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17521","reported_seq":17583,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.147898+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:42.147898+0000","last_peered":"2024-09-16T10:31:42.147898+0000","last_clean":"2024-09-16T10:31:42.147898+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:42.147898+0000","last_undegraded":"2024-09-16T10:31:42.147898+0000","last_fullsized":"2024-09-16T10:31:42.147898+0000","mapping_epoch":34,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12046336,"num_objects":2941,"num_object_clones":0,"num_object_copies":5882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17521,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17564","reported_seq":17727,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:43.112172+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:43.112172+0000","last_peered":"2024-09-16T10:31:43.112172+0000","last_clean":"2024-09-16T10:31:43.112172+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:43.112172+0000","last_undegraded":"2024-09-16T10:31:43.112172+0000","last_fullsized":"2024-09-16T10:31:43.112172+0000","mapping_epoch":33,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12046336,"num_objects":2941,"num_object_clones":0,"num_object_copies":5882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14206,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17406","reported_seq":17740,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.165321+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:43.165321+0000","last_peered":"2024-09-16T10:31:43.165321+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:43.165321+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12144640,"num_objects":2965,"num_object_clones":0,"num_object_copies":5930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14031,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18000","reported_seq":18411,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.032806+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:43.032806+0000","last_peered":"2024-09-16T10:31:43.032806+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:43.032806+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11960320,"num_objects":2920,"num_object_clones":0,"num_object_copies":5840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14242,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17484","reported_seq":17665,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:43.099174+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:43.099174+0000","last_peered":"2024-09-16T10:31:43.099174+0000","last_clean":"2024-09-16T10:31:43.099174+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:43.099174+0000","last_undegraded":"2024-09-16T10:31:43.099174+0000","last_fullsized":"2024-09-16T10:31:43.099174+0000","mapping_epoch":33,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11902976,"num_objects":2906,"num_object_clones":0,"num_object_copies":5812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13287,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17407","reported_seq":25916,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.118601+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:42.118601+0000","last_peered":"2024-09-16T10:31:42.118601+0000","last_clean":"2024-09-16T10:31:42.118601+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:42.118601+0000","last_undegraded":"2024-09-16T10:31:42.118601+0000","last_fullsized":"2024-09-16T10:31:42.118601+0000","mapping_epoch":43,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12510150,"num_objects":3056,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17405,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18004","reported_seq":18279,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.078494+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:43.078494+0000","last_peered":"2024-09-16T10:31:43.078494+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:43.078494+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12436422,"num_objects":3038,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18002,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1299}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17429","reported_seq":17629,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:42.192103+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:42.192103+0000","last_peered":"2024-09-16T10:31:42.192103+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:42.192103+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12248006,"num_objects":2992,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17427,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1241}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17579","reported_seq":26067,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.179417+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:42.179417+0000","last_peered":"2024-09-16T10:31:42.179417+0000","last_clean":"2024-09-16T10:31:42.179417+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:42.179417+0000","last_undegraded":"2024-09-16T10:31:42.179417+0000","last_fullsized":"2024-09-16T10:31:42.179417+0000","mapping_epoch":47,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12276678,"num_objects":2999,"num_object_clones":0,"num_object_copies":5998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17577,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17164","reported_seq":17418,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.122870+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:43.122870+0000","last_peered":"2024-09-16T10:31:43.122870+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:43.122870+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11756003,"num_objects":2871,"num_object_clones":0,"num_object_copies":5742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17163,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'17947","reported_seq":18224,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.126534+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:43.126534+0000","last_peered":"2024-09-16T10:31:43.126534+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:43.126534+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12154281,"num_objects":2970,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17944,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17246","reported_seq":18080,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.167398+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:43.167398+0000","last_peered":"2024-09-16T10:31:43.167398+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:43.167398+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11854790,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17244,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":375,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'17926","reported_seq":18171,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:42.800704+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:42.800704+0000","last_peered":"2024-09-16T10:31:42.800704+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:42.800704+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12167052,"num_objects":2974,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17923,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":193831438,"num_objects":47339,"num_object_clones":0,"num_object_copies":94678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47339,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":258966,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13576,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598638592,"data_stored":598561801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018991,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553416,"kb_used_data":233024,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93818424,"statfs":{"total":96636764160,"available":96070066176,"internally_reserved":0,"allocated":238616576,"data_stored":238337930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":2.4470000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.9350000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.6520000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.5750000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691935,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515464,"kb_used_data":145024,"kb_used_omap":4,"kb_used_meta":370427,"kb_avail":93856376,"statfs":{"total":96636764160,"available":96108929024,"internally_reserved":0,"allocated":148504576,"data_stored":148168122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":379317930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.6019999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.86199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.4239999999999999},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.4470000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215218,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":646032,"kb_used_data":209224,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93725808,"statfs":{"total":96636764160,"available":95975227392,"internally_reserved":0,"allocated":214245376,"data_stored":213974416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":3.169},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.5720000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":213725184,"data_stored":213721571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146784256,"data_stored":146718763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238129152,"data_stored":238121467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:47.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:48.299 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:48.299 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:48.320 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-16T10:31:46.730517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":193831457,"num_objects":47341,"num_object_clones":0,"num_object_copies":94682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47341,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":258968,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13577,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1714912,"kb_used_data":587272,"kb_used_omap":12,"kb_used_meta":1127603,"kb_avail":281400608,"statfs":{"total":289910292480,"available":288154222592,"internally_reserved":0,"allocated":601366528,"data_stored":600480468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1154665474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,2,5,12,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19083264,"num_objects":-4659,"num_object_clones":0,"num_object_copies":-9318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-157,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4659,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002803"},"pg_stats":[{"pgid":"2.f","version":"48'17559","reported_seq":26027,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:43.206389+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:43.206389+0000","last_peered":"2024-09-16T10:31:43.206389+0000","last_clean":"2024-09-16T10:31:43.206389+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:43.206389+0000","last_undegraded":"2024-09-16T10:31:43.206389+0000","last_fullsized":"2024-09-16T10:31:43.206389+0000","mapping_epoch":45,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11915264,"num_objects":2909,"num_object_clones":0,"num_object_copies":5818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17559,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17536","reported_seq":17697,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.147713+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:42.147713+0000","last_peered":"2024-09-16T10:31:42.147713+0000","last_clean":"2024-09-16T10:31:42.147713+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:42.147713+0000","last_undegraded":"2024-09-16T10:31:42.147713+0000","last_fullsized":"2024-09-16T10:31:42.147713+0000","mapping_epoch":34,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11984896,"num_objects":2926,"num_object_clones":0,"num_object_copies":5852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13749,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17629","reported_seq":17885,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:42.143340+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:42.143340+0000","last_peered":"2024-09-16T10:31:42.143340+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:42.143340+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12427288,"num_objects":3035,"num_object_clones":0,"num_object_copies":6070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13686,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1268}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17521","reported_seq":17583,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.147898+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:42.147898+0000","last_peered":"2024-09-16T10:31:42.147898+0000","last_clean":"2024-09-16T10:31:42.147898+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:42.147898+0000","last_undegraded":"2024-09-16T10:31:42.147898+0000","last_fullsized":"2024-09-16T10:31:42.147898+0000","mapping_epoch":34,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12046336,"num_objects":2941,"num_object_clones":0,"num_object_copies":5882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17521,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17564","reported_seq":17727,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:43.112172+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:43.112172+0000","last_peered":"2024-09-16T10:31:43.112172+0000","last_clean":"2024-09-16T10:31:43.112172+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:43.112172+0000","last_undegraded":"2024-09-16T10:31:43.112172+0000","last_fullsized":"2024-09-16T10:31:43.112172+0000","mapping_epoch":33,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12046336,"num_objects":2941,"num_object_clones":0,"num_object_copies":5882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14206,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17406","reported_seq":17740,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.165321+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:43.165321+0000","last_peered":"2024-09-16T10:31:43.165321+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:43.165321+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12144640,"num_objects":2965,"num_object_clones":0,"num_object_copies":5930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14031,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1169}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18000","reported_seq":18411,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.032806+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:43.032806+0000","last_peered":"2024-09-16T10:31:43.032806+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:43.032806+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11960320,"num_objects":2920,"num_object_clones":0,"num_object_copies":5840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14242,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17484","reported_seq":17665,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:43.099174+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:43.099174+0000","last_peered":"2024-09-16T10:31:43.099174+0000","last_clean":"2024-09-16T10:31:43.099174+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:43.099174+0000","last_undegraded":"2024-09-16T10:31:43.099174+0000","last_fullsized":"2024-09-16T10:31:43.099174+0000","mapping_epoch":33,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11902976,"num_objects":2906,"num_object_clones":0,"num_object_copies":5812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13287,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17407","reported_seq":25916,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.118601+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:42.118601+0000","last_peered":"2024-09-16T10:31:42.118601+0000","last_clean":"2024-09-16T10:31:42.118601+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:42.118601+0000","last_undegraded":"2024-09-16T10:31:42.118601+0000","last_fullsized":"2024-09-16T10:31:42.118601+0000","mapping_epoch":43,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12510150,"num_objects":3056,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17405,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18004","reported_seq":18279,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.078494+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:43.078494+0000","last_peered":"2024-09-16T10:31:43.078494+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:43.078494+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12436422,"num_objects":3038,"num_object_clones":0,"num_object_copies":6076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18002,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1299}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17429","reported_seq":17629,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:42.192103+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:42.192103+0000","last_peered":"2024-09-16T10:31:42.192103+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:42.192103+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12248006,"num_objects":2992,"num_object_clones":0,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17427,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1241}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17579","reported_seq":26067,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:42.179417+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:42.179417+0000","last_peered":"2024-09-16T10:31:42.179417+0000","last_clean":"2024-09-16T10:31:42.179417+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:42.179417+0000","last_undegraded":"2024-09-16T10:31:42.179417+0000","last_fullsized":"2024-09-16T10:31:42.179417+0000","mapping_epoch":47,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12276678,"num_objects":2999,"num_object_clones":0,"num_object_copies":5998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17577,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17164","reported_seq":17418,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.122870+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:43.122870+0000","last_peered":"2024-09-16T10:31:43.122870+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:43.122870+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7100","ondisk_log_start":"42'7100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11756003,"num_objects":2871,"num_object_clones":0,"num_object_copies":5742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17163,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1204}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'17947","reported_seq":18224,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.126534+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:43.126534+0000","last_peered":"2024-09-16T10:31:43.126534+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:43.126534+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12154281,"num_objects":2970,"num_object_clones":0,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17944,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17246","reported_seq":18080,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:43.167398+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:43.167398+0000","last_peered":"2024-09-16T10:31:43.167398+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:43.167398+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7200","ondisk_log_start":"42'7200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11854790,"num_objects":2896,"num_object_clones":0,"num_object_copies":5792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17244,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":375,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'17926","reported_seq":18171,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:42.800704+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:42.800704+0000","last_peered":"2024-09-16T10:31:42.800704+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:42.800704+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12167052,"num_objects":2974,"num_object_clones":0,"num_object_copies":5948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17923,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":193831438,"num_objects":47339,"num_object_clones":0,"num_object_copies":94678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47339,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":258966,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13576,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598638592,"data_stored":598561801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160701,"ondisk_log_size":160701,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018991,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553416,"kb_used_data":233024,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93818424,"statfs":{"total":96636764160,"available":96070066176,"internally_reserved":0,"allocated":238616576,"data_stored":238337930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":2.4470000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.9350000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":4.6520000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":3.5750000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691935,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515464,"kb_used_data":145024,"kb_used_omap":4,"kb_used_meta":370427,"kb_avail":93856376,"statfs":{"total":96636764160,"available":96108929024,"internally_reserved":0,"allocated":148504576,"data_stored":148168122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":379317930},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,6,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.6019999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.86199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":2.4239999999999999},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.4470000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215218,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":646032,"kb_used_data":209224,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93725808,"statfs":{"total":96636764160,"available":95975227392,"internally_reserved":0,"allocated":214245376,"data_stored":213974416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,2,5,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":3.169},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.5720000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":213725184,"data_stored":213721571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146784256,"data_stored":146718763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":238129152,"data_stored":238121467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:48.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:48.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'17629", 'reported_seq': 17885, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:42.143340+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:42.143340+0000', 'last_peered': '2024-09-16T10:31:42.143340+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:42.143340+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'7600", 'ondisk_log_start': "42'7600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12427288, 'num_objects': 3035, 'num_object_clones': 0, 'num_object_copies': 6070, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3035, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13686, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1268}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:48.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:48.321 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'17406", 'reported_seq': 17740, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:43.165321+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:43.165321+0000', 'last_peered': '2024-09-16T10:31:43.165321+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:43.165321+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'7400", 'ondisk_log_start': "42'7400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12144640, 'num_objects': 2965, 'num_object_clones': 0, 'num_object_copies': 5930, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14031, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1169}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'18000", 'reported_seq': 18411, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:43.032806+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:43.032806+0000', 'last_peered': '2024-09-16T10:31:43.032806+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:43.032806+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'7900", 'ondisk_log_start': "42'7900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11960320, 'num_objects': 2920, 'num_object_clones': 0, 'num_object_copies': 5840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2920, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14242, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1367}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'18004", 'reported_seq': 18279, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:43.078494+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:43.078494+0000', 'last_peered': '2024-09-16T10:31:43.078494+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:43.078494+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'8000", 'ondisk_log_start': "42'8000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12436422, 'num_objects': 3038, 'num_object_clones': 0, 'num_object_copies': 6076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18002, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1299}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'17429", 'reported_seq': 17629, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:42.192103+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:42.192103+0000', 'last_peered': '2024-09-16T10:31:42.192103+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:42.192103+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'7400", 'ondisk_log_start': "42'7400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12248006, 'num_objects': 2992, 'num_object_clones': 0, 'num_object_copies': 5984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17427, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1241}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'17164", 'reported_seq': 17418, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:43.122870+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:43.122870+0000', 'last_peered': '2024-09-16T10:31:43.122870+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:43.122870+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'7100", 'ondisk_log_start': "42'7100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11756003, 'num_objects': 2871, 'num_object_clones': 0, 'num_object_copies': 5742, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17163, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1204}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'17947", 'reported_seq': 18224, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:43.126534+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:43.126534+0000', 'last_peered': '2024-09-16T10:31:43.126534+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:43.126534+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'7900", 'ondisk_log_start': "42'7900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12154281, 'num_objects': 2970, 'num_object_clones': 0, 'num_object_copies': 5940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2970, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17944, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1297}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:48.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:48.323 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'17246", 'reported_seq': 18080, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:43.167398+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:31:43.167398+0000', 'last_peered': '2024-09-16T10:31:43.167398+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:43.167398+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'7200", 'ondisk_log_start': "42'7200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11854790, 'num_objects': 2896, 'num_object_clones': 0, 'num_object_copies': 5792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3796, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2896, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17244, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 375, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:48.323 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:48.323 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'17926", 'reported_seq': 18171, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:42.800704+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:42.800704+0000', 'last_peered': '2024-09-16T10:31:42.800704+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:42.800704+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'7900", 'ondisk_log_start': "42'7900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12167052, 'num_objects': 2974, 'num_object_clones': 0, 'num_object_copies': 5948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2974, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17923, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:48.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:49.004 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:49.004 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37690/90006 objects degraded (41.875%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":597,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":45003,"data_bytes":184255009,"bytes_used":1774235648,"bytes_avail":288136056832,"bytes_total":289910292480,"degraded_objects":37690,"degraded_total":90006,"degraded_ratio":0.41874986112036977,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":237,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":582},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-16T10:30:26.711632+0000","services":{}},"progress_events":{}} 2024-09-16T10:31:49.023 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:31:49.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:49.586 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:49.589 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:49.612 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-16T10:31:48.730950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":184255009,"num_objects":45003,"num_object_clones":0,"num_object_copies":90006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":261306,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13657,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1732652,"kb_used_data":568904,"kb_used_omap":12,"kb_used_meta":1163699,"kb_avail":281382868,"statfs":{"total":289910292480,"available":288136056832,"internally_reserved":0,"allocated":582557696,"data_stored":581671636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1191627778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,3,10,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28659712,"num_objects":-6997,"num_object_clones":0,"num_object_copies":-13994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6997,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002754"},"pg_stats":[{"pgid":"2.f","version":"48'17711","reported_seq":26177,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:48.215982+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:48.215982+0000","last_peered":"2024-09-16T10:31:48.215982+0000","last_clean":"2024-09-16T10:31:48.215982+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:48.215982+0000","last_undegraded":"2024-09-16T10:31:48.215982+0000","last_fullsized":"2024-09-16T10:31:48.215982+0000","mapping_epoch":45,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11292672,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17711,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17678","reported_seq":17841,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:47.194854+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:47.194854+0000","last_peered":"2024-09-16T10:31:47.194854+0000","last_clean":"2024-09-16T10:31:47.194854+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:47.194854+0000","last_undegraded":"2024-09-16T10:31:47.194854+0000","last_fullsized":"2024-09-16T10:31:47.194854+0000","mapping_epoch":34,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11403264,"num_objects":2784,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13891,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17763","reported_seq":18019,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:47.216008+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:47.216008+0000","last_peered":"2024-09-16T10:31:47.216008+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:47.216008+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11878424,"num_objects":2901,"num_object_clones":0,"num_object_copies":5802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13820,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1402}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17664","reported_seq":17726,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:47.211275+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:47.211275+0000","last_peered":"2024-09-16T10:31:47.211275+0000","last_clean":"2024-09-16T10:31:47.211275+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:47.211275+0000","last_undegraded":"2024-09-16T10:31:47.211275+0000","last_fullsized":"2024-09-16T10:31:47.211275+0000","mapping_epoch":34,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11460608,"num_objects":2798,"num_object_clones":0,"num_object_copies":5596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17664,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17704","reported_seq":17867,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:48.188548+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:48.188548+0000","last_peered":"2024-09-16T10:31:48.188548+0000","last_clean":"2024-09-16T10:31:48.188548+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:48.188548+0000","last_undegraded":"2024-09-16T10:31:48.188548+0000","last_fullsized":"2024-09-16T10:31:48.188548+0000","mapping_epoch":33,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11472896,"num_objects":2801,"num_object_clones":0,"num_object_copies":5602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14346,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17552","reported_seq":17888,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.218387+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:48.218387+0000","last_peered":"2024-09-16T10:31:48.218387+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:48.218387+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11546624,"num_objects":2819,"num_object_clones":0,"num_object_copies":5638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14177,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1315}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18158","reported_seq":18568,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.233024+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:48.233024+0000","last_peered":"2024-09-16T10:31:48.233024+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:48.233024+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11313152,"num_objects":2762,"num_object_clones":0,"num_object_copies":5524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14400,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17634","reported_seq":17815,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:48.236663+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:48.236663+0000","last_peered":"2024-09-16T10:31:48.236663+0000","last_clean":"2024-09-16T10:31:48.236663+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:48.236663+0000","last_undegraded":"2024-09-16T10:31:48.236663+0000","last_fullsized":"2024-09-16T10:31:48.236663+0000","mapping_epoch":33,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11288576,"num_objects":2756,"num_object_clones":0,"num_object_copies":5512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13437,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17571","reported_seq":26080,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:47.256751+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:47.256751+0000","last_peered":"2024-09-16T10:31:47.256751+0000","last_clean":"2024-09-16T10:31:47.256751+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:47.256751+0000","last_undegraded":"2024-09-16T10:31:47.256751+0000","last_fullsized":"2024-09-16T10:31:47.256751+0000","mapping_epoch":43,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11838406,"num_objects":2892,"num_object_clones":0,"num_object_copies":5784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17569,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18160","reported_seq":18439,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.161487+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:48.161487+0000","last_peered":"2024-09-16T10:31:48.161487+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:48.161487+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11797446,"num_objects":2882,"num_object_clones":0,"num_object_copies":5764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18158,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1455}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17586","reported_seq":17786,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:47.136680+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:47.136680+0000","last_peered":"2024-09-16T10:31:47.136680+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:47.136680+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11604934,"num_objects":2835,"num_object_clones":0,"num_object_copies":5670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17584,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1398}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17725","reported_seq":26213,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:47.230048+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:47.230048+0000","last_peered":"2024-09-16T10:31:47.230048+0000","last_clean":"2024-09-16T10:31:47.230048+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:47.230048+0000","last_undegraded":"2024-09-16T10:31:47.230048+0000","last_fullsized":"2024-09-16T10:31:47.230048+0000","mapping_epoch":47,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11678662,"num_objects":2853,"num_object_clones":0,"num_object_copies":5706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17304","reported_seq":17558,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.196668+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:48.196668+0000","last_peered":"2024-09-16T10:31:48.196668+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:48.196668+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11182563,"num_objects":2731,"num_object_clones":0,"num_object_copies":5462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17303,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18060","reported_seq":18339,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:47.922694+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:47.922694+0000","last_peered":"2024-09-16T10:31:47.922694+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:47.922694+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11691433,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18057,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17391","reported_seq":18385,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.258810+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:48.258810+0000","last_peered":"2024-09-16T10:31:48.258810+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:48.258810+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11260870,"num_objects":2751,"num_object_clones":0,"num_object_copies":5502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17389,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":455,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-965}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18078","reported_seq":18323,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.158439+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:48.158439+0000","last_peered":"2024-09-16T10:31:48.158439+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:48.158439+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11544460,"num_objects":2822,"num_object_clones":0,"num_object_copies":5644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18075,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":184254990,"num_objects":45001,"num_object_clones":0,"num_object_copies":90002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45001,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":261304,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13656,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":579555328,"data_stored":579478537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018992,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":553008,"kb_used_data":228520,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93818832,"statfs":{"total":96636764160,"available":96070483968,"internally_reserved":0,"allocated":234004480,"data_stored":233725834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":0.93999999999999995},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":6.9390000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.253},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.97299999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691936,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552736,"kb_used_data":138008,"kb_used_omap":4,"kb_used_meta":414715,"kb_avail":93819104,"statfs":{"total":96636764160,"available":96070762496,"internally_reserved":0,"allocated":141320192,"data_stored":140983738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":424668842},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":9.4879999999999995},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":9.4550000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.7589999999999999},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":3.645}]}]},{"osd":0,"up_from":24,"seq":103079215219,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626908,"kb_used_data":202376,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93744932,"statfs":{"total":96636764160,"available":95994810368,"internally_reserved":0,"allocated":207233024,"data_stored":206962064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":434695848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":3.1600000000000001},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.0899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":206688256,"data_stored":206684643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139534336,"data_stored":139468843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233332736,"data_stored":233325051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:52.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:53.155 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:53.157 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:53.178 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-16T10:31:52.731835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":180810273,"num_objects":44162,"num_object_clones":0,"num_object_copies":88324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":262147,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13657,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160583,"ondisk_log_size":160583,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1732244,"kb_used_data":556192,"kb_used_omap":12,"kb_used_meta":1175987,"kb_avail":281383276,"statfs":{"total":289910292480,"available":288136474624,"internally_reserved":0,"allocated":569540608,"data_stored":568654548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1204210690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,10,8,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22691840,"num_objects":-5540,"num_object_clones":0,"num_object_copies":-11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5540,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002704"},"pg_stats":[{"pgid":"2.f","version":"48'17711","reported_seq":26177,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:48.215982+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:48.215982+0000","last_peered":"2024-09-16T10:31:48.215982+0000","last_clean":"2024-09-16T10:31:48.215982+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:48.215982+0000","last_undegraded":"2024-09-16T10:31:48.215982+0000","last_fullsized":"2024-09-16T10:31:48.215982+0000","mapping_epoch":45,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11292672,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17711,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17808","reported_seq":17968,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.189918+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:52.189918+0000","last_peered":"2024-09-16T10:31:52.189918+0000","last_clean":"2024-09-16T10:31:52.189918+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:52.189918+0000","last_undegraded":"2024-09-16T10:31:52.189918+0000","last_fullsized":"2024-09-16T10:31:52.189918+0000","mapping_epoch":34,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10870784,"num_objects":2654,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14021,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17922","reported_seq":18178,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:52.177724+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:52.177724+0000","last_peered":"2024-09-16T10:31:52.177724+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:52.177724+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11227160,"num_objects":2742,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13979,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17820","reported_seq":17882,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.245971+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:52.245971+0000","last_peered":"2024-09-16T10:31:52.245971+0000","last_clean":"2024-09-16T10:31:52.245971+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:52.245971+0000","last_undegraded":"2024-09-16T10:31:52.245971+0000","last_fullsized":"2024-09-16T10:31:52.245971+0000","mapping_epoch":34,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10821632,"num_objects":2642,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17820,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17704","reported_seq":17867,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:48.188548+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:48.188548+0000","last_peered":"2024-09-16T10:31:48.188548+0000","last_clean":"2024-09-16T10:31:48.188548+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:48.188548+0000","last_undegraded":"2024-09-16T10:31:48.188548+0000","last_fullsized":"2024-09-16T10:31:48.188548+0000","mapping_epoch":33,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11472896,"num_objects":2801,"num_object_clones":0,"num_object_copies":5602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14346,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17552","reported_seq":17888,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.218387+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:48.218387+0000","last_peered":"2024-09-16T10:31:48.218387+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:48.218387+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11546624,"num_objects":2819,"num_object_clones":0,"num_object_copies":5638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14177,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1315}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18158","reported_seq":18568,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.233024+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:48.233024+0000","last_peered":"2024-09-16T10:31:48.233024+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:48.233024+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11313152,"num_objects":2762,"num_object_clones":0,"num_object_copies":5524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14400,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17634","reported_seq":17815,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:48.236663+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:48.236663+0000","last_peered":"2024-09-16T10:31:48.236663+0000","last_clean":"2024-09-16T10:31:48.236663+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:48.236663+0000","last_undegraded":"2024-09-16T10:31:48.236663+0000","last_fullsized":"2024-09-16T10:31:48.236663+0000","mapping_epoch":33,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11288576,"num_objects":2756,"num_object_clones":0,"num_object_copies":5512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13437,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17711","reported_seq":26220,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.306456+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:52.306456+0000","last_peered":"2024-09-16T10:31:52.306456+0000","last_clean":"2024-09-16T10:31:52.306456+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:52.306456+0000","last_undegraded":"2024-09-16T10:31:52.306456+0000","last_fullsized":"2024-09-16T10:31:52.306456+0000","mapping_epoch":43,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11264966,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17709,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18160","reported_seq":18439,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.161487+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:48.161487+0000","last_peered":"2024-09-16T10:31:48.161487+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:48.161487+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11797446,"num_objects":2882,"num_object_clones":0,"num_object_copies":5764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18158,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1455}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17725","reported_seq":17925,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:52.235697+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:52.235697+0000","last_peered":"2024-09-16T10:31:52.235697+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:52.235697+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11035590,"num_objects":2696,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1537}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17842","reported_seq":26330,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.272104+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:52.272104+0000","last_peered":"2024-09-16T10:31:52.272104+0000","last_clean":"2024-09-16T10:31:52.272104+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:52.272104+0000","last_undegraded":"2024-09-16T10:31:52.272104+0000","last_fullsized":"2024-09-16T10:31:52.272104+0000","mapping_epoch":47,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11199430,"num_objects":2736,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17840,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17304","reported_seq":17558,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.196668+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:48.196668+0000","last_peered":"2024-09-16T10:31:48.196668+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:48.196668+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11182563,"num_objects":2731,"num_object_clones":0,"num_object_copies":5462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17303,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18060","reported_seq":18339,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:47.922694+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:47.922694+0000","last_peered":"2024-09-16T10:31:47.922694+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:47.922694+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11691433,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18057,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17391","reported_seq":18385,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.258810+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:48.258810+0000","last_peered":"2024-09-16T10:31:48.258810+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:48.258810+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11260870,"num_objects":2751,"num_object_clones":0,"num_object_copies":5502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17389,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":455,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-965}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18078","reported_seq":18323,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.158439+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:48.158439+0000","last_peered":"2024-09-16T10:31:48.158439+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:48.158439+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11544460,"num_objects":2822,"num_object_clones":0,"num_object_copies":5644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18075,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":180810254,"num_objects":44160,"num_object_clones":0,"num_object_copies":88320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":262145,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13656,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":567861248,"data_stored":567784457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160580,"ondisk_log_size":160580,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018993,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552452,"kb_used_data":223860,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93819388,"statfs":{"total":96636764160,"available":96071053312,"internally_reserved":0,"allocated":229232640,"data_stored":228953994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":6.7290000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.179}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.8140000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":6.7919999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691937,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552884,"kb_used_data":129956,"kb_used_omap":4,"kb_used_meta":422907,"kb_avail":93818956,"statfs":{"total":96636764160,"available":96070610944,"internally_reserved":0,"allocated":133074944,"data_stored":132738490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":433057450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.671},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.744}]}]},{"osd":0,"up_from":24,"seq":103079215219,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626908,"kb_used_data":202376,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93744932,"statfs":{"total":96636764160,"available":95994810368,"internally_reserved":0,"allocated":207233024,"data_stored":206962064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":434695848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":3.1600000000000001},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.0899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":206688256,"data_stored":206684643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132571136,"data_stored":132505643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":228601856,"data_stored":228594171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:53.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:53.743 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:53.746 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:53.769 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-16T10:31:52.731835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":180810273,"num_objects":44162,"num_object_clones":0,"num_object_copies":88324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":262147,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13657,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160583,"ondisk_log_size":160583,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1732244,"kb_used_data":556192,"kb_used_omap":12,"kb_used_meta":1175987,"kb_avail":281383276,"statfs":{"total":289910292480,"available":288136474624,"internally_reserved":0,"allocated":569540608,"data_stored":568654548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1204210690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,10,8,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22691840,"num_objects":-5540,"num_object_clones":0,"num_object_copies":-11080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5540,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002704"},"pg_stats":[{"pgid":"2.f","version":"48'17711","reported_seq":26177,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:48.215982+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:48.215982+0000","last_peered":"2024-09-16T10:31:48.215982+0000","last_clean":"2024-09-16T10:31:48.215982+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:48.215982+0000","last_undegraded":"2024-09-16T10:31:48.215982+0000","last_fullsized":"2024-09-16T10:31:48.215982+0000","mapping_epoch":45,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11292672,"num_objects":2757,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17711,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17808","reported_seq":17968,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.189918+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:52.189918+0000","last_peered":"2024-09-16T10:31:52.189918+0000","last_clean":"2024-09-16T10:31:52.189918+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:52.189918+0000","last_undegraded":"2024-09-16T10:31:52.189918+0000","last_fullsized":"2024-09-16T10:31:52.189918+0000","mapping_epoch":34,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10870784,"num_objects":2654,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14021,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17922","reported_seq":18178,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:52.177724+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:52.177724+0000","last_peered":"2024-09-16T10:31:52.177724+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:52.177724+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11227160,"num_objects":2742,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13979,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17820","reported_seq":17882,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.245971+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:52.245971+0000","last_peered":"2024-09-16T10:31:52.245971+0000","last_clean":"2024-09-16T10:31:52.245971+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:52.245971+0000","last_undegraded":"2024-09-16T10:31:52.245971+0000","last_fullsized":"2024-09-16T10:31:52.245971+0000","mapping_epoch":34,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10821632,"num_objects":2642,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17820,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17704","reported_seq":17867,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:48.188548+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:48.188548+0000","last_peered":"2024-09-16T10:31:48.188548+0000","last_clean":"2024-09-16T10:31:48.188548+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:48.188548+0000","last_undegraded":"2024-09-16T10:31:48.188548+0000","last_fullsized":"2024-09-16T10:31:48.188548+0000","mapping_epoch":33,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11472896,"num_objects":2801,"num_object_clones":0,"num_object_copies":5602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14346,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17552","reported_seq":17888,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.218387+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:48.218387+0000","last_peered":"2024-09-16T10:31:48.218387+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:48.218387+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11546624,"num_objects":2819,"num_object_clones":0,"num_object_copies":5638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14177,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1315}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18158","reported_seq":18568,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.233024+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:48.233024+0000","last_peered":"2024-09-16T10:31:48.233024+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:48.233024+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11313152,"num_objects":2762,"num_object_clones":0,"num_object_copies":5524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14400,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1525}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17634","reported_seq":17815,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:48.236663+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:48.236663+0000","last_peered":"2024-09-16T10:31:48.236663+0000","last_clean":"2024-09-16T10:31:48.236663+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:48.236663+0000","last_undegraded":"2024-09-16T10:31:48.236663+0000","last_fullsized":"2024-09-16T10:31:48.236663+0000","mapping_epoch":33,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11288576,"num_objects":2756,"num_object_clones":0,"num_object_copies":5512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13437,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17711","reported_seq":26220,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.306456+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:52.306456+0000","last_peered":"2024-09-16T10:31:52.306456+0000","last_clean":"2024-09-16T10:31:52.306456+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:52.306456+0000","last_undegraded":"2024-09-16T10:31:52.306456+0000","last_fullsized":"2024-09-16T10:31:52.306456+0000","mapping_epoch":43,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11264966,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17709,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18160","reported_seq":18439,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.161487+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:48.161487+0000","last_peered":"2024-09-16T10:31:48.161487+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:48.161487+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11797446,"num_objects":2882,"num_object_clones":0,"num_object_copies":5764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18158,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1455}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17725","reported_seq":17925,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:52.235697+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:52.235697+0000","last_peered":"2024-09-16T10:31:52.235697+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:52.235697+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11035590,"num_objects":2696,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1537}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17842","reported_seq":26330,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.272104+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:52.272104+0000","last_peered":"2024-09-16T10:31:52.272104+0000","last_clean":"2024-09-16T10:31:52.272104+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:52.272104+0000","last_undegraded":"2024-09-16T10:31:52.272104+0000","last_fullsized":"2024-09-16T10:31:52.272104+0000","mapping_epoch":47,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11199430,"num_objects":2736,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17840,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17304","reported_seq":17558,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.196668+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:48.196668+0000","last_peered":"2024-09-16T10:31:48.196668+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:48.196668+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11182563,"num_objects":2731,"num_object_clones":0,"num_object_copies":5462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17303,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18060","reported_seq":18339,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:47.922694+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:47.922694+0000","last_peered":"2024-09-16T10:31:47.922694+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:47.922694+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11691433,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18057,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17391","reported_seq":18385,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.258810+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:48.258810+0000","last_peered":"2024-09-16T10:31:48.258810+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:48.258810+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7300","ondisk_log_start":"42'7300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11260870,"num_objects":2751,"num_object_clones":0,"num_object_copies":5502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3716,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17389,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":455,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-965}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18078","reported_seq":18323,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:48.158439+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:48.158439+0000","last_peered":"2024-09-16T10:31:48.158439+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:48.158439+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11544460,"num_objects":2822,"num_object_clones":0,"num_object_copies":5644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18075,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1500}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":180810254,"num_objects":44160,"num_object_clones":0,"num_object_copies":88320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44160,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":262145,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13656,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":567861248,"data_stored":567784457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160580,"ondisk_log_size":160580,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018993,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552452,"kb_used_data":223860,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93819388,"statfs":{"total":96636764160,"available":96071053312,"internally_reserved":0,"allocated":229232640,"data_stored":228953994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":6.7290000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.179}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.8140000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":6.7919999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691937,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552884,"kb_used_data":129956,"kb_used_omap":4,"kb_used_meta":422907,"kb_avail":93818956,"statfs":{"total":96636764160,"available":96070610944,"internally_reserved":0,"allocated":133074944,"data_stored":132738490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":433057450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.671},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.744}]}]},{"osd":0,"up_from":24,"seq":103079215219,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":626908,"kb_used_data":202376,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93744932,"statfs":{"total":96636764160,"available":95994810368,"internally_reserved":0,"allocated":207233024,"data_stored":206962064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":434695848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.26300000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":3.1600000000000001},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.0899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":206688256,"data_stored":206684643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132571136,"data_stored":132505643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":228601856,"data_stored":228594171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:53.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:53.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'17922", 'reported_seq': 18178, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:52.177724+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:52.177724+0000', 'last_peered': '2024-09-16T10:31:52.177724+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:52.177724+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'7900", 'ondisk_log_start': "42'7900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11227160, 'num_objects': 2742, 'num_object_clones': 0, 'num_object_copies': 5484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2742, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 13979, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1561}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:53.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:53.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'17552", 'reported_seq': 17888, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:48.218387+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:48.218387+0000', 'last_peered': '2024-09-16T10:31:48.218387+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:48.218387+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'7500", 'ondisk_log_start': "42'7500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11546624, 'num_objects': 2819, 'num_object_clones': 0, 'num_object_copies': 5638, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14177, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1315}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:53.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:53.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'18158", 'reported_seq': 18568, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:48.233024+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:48.233024+0000', 'last_peered': '2024-09-16T10:31:48.233024+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:48.233024+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'8100", 'ondisk_log_start': "42'8100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11313152, 'num_objects': 2762, 'num_object_clones': 0, 'num_object_copies': 5524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14400, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1525}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:53.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:53.771 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'18160", 'reported_seq': 18439, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:48.161487+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:48.161487+0000', 'last_peered': '2024-09-16T10:31:48.161487+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:48.161487+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'8100", 'ondisk_log_start': "42'8100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11797446, 'num_objects': 2882, 'num_object_clones': 0, 'num_object_copies': 5764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18158, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1455}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:53.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:53.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'17725", 'reported_seq': 17925, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:52.235697+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:52.235697+0000', 'last_peered': '2024-09-16T10:31:52.235697+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:52.235697+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'7700", 'ondisk_log_start': "42'7700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11035590, 'num_objects': 2696, 'num_object_clones': 0, 'num_object_copies': 5392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17723, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1537}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:53.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:53.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'17304", 'reported_seq': 17558, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:48.196668+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:48.196668+0000', 'last_peered': '2024-09-16T10:31:48.196668+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:48.196668+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'7300", 'ondisk_log_start': "42'7300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11182563, 'num_objects': 2731, 'num_object_clones': 0, 'num_object_copies': 5462, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2731, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17303, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1344}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:53.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:53.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'18060", 'reported_seq': 18339, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:47.922694+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:47.922694+0000', 'last_peered': '2024-09-16T10:31:47.922694+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:47.922694+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'8000", 'ondisk_log_start': "42'8000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11691433, 'num_objects': 2857, 'num_object_clones': 0, 'num_object_copies': 5714, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18057, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:53.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:53.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'17391", 'reported_seq': 18385, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:48.258810+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:31:48.258810+0000', 'last_peered': '2024-09-16T10:31:48.258810+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:48.258810+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'7300", 'ondisk_log_start': "42'7300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11260870, 'num_objects': 2751, 'num_object_clones': 0, 'num_object_copies': 5502, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3716, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17389, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 455, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -965}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:53.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:53.772 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'18078", 'reported_seq': 18323, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:48.158439+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:48.158439+0000', 'last_peered': '2024-09-16T10:31:48.158439+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:48.158439+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'8000", 'ondisk_log_start': "42'8000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11544460, 'num_objects': 2822, 'num_object_clones': 0, 'num_object_copies': 5644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18075, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1500}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:53.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:54.468 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:54.468 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37690/88324 objects degraded (42.672%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":603,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":44162,"data_bytes":180810273,"bytes_used":1773817856,"bytes_avail":288136474624,"bytes_total":289910292480,"degraded_objects":37690,"degraded_total":88324,"degraded_ratio":0.42672433313708619,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":461},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:31:54.505 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:31:54.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:55.077 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:55.078 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:55.101 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-16T10:31:54.732319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":174899745,"num_objects":42719,"num_object_clones":0,"num_object_copies":85438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42719,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263590,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13733,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160626,"ondisk_log_size":160626,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1733552,"kb_used_data":549320,"kb_used_omap":12,"kb_used_meta":1184179,"kb_avail":281381968,"statfs":{"total":289910292480,"available":288135135232,"internally_reserved":0,"allocated":562503680,"data_stored":561617620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1212599298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,9,14,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25112576,"num_objects":-6131,"num_object_clones":0,"num_object_copies":-12262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002752"},"pg_stats":[{"pgid":"2.f","version":"48'17864","reported_seq":26332,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:53.259571+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:53.259571+0000","last_peered":"2024-09-16T10:31:53.259571+0000","last_clean":"2024-09-16T10:31:53.259571+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:53.259571+0000","last_undegraded":"2024-09-16T10:31:53.259571+0000","last_fullsized":"2024-09-16T10:31:53.259571+0000","mapping_epoch":45,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10665984,"num_objects":2604,"num_object_clones":0,"num_object_copies":5208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17864,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17808","reported_seq":17968,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.189918+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:52.189918+0000","last_peered":"2024-09-16T10:31:52.189918+0000","last_clean":"2024-09-16T10:31:52.189918+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:52.189918+0000","last_undegraded":"2024-09-16T10:31:52.189918+0000","last_fullsized":"2024-09-16T10:31:52.189918+0000","mapping_epoch":34,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10870784,"num_objects":2654,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14021,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17922","reported_seq":18178,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:52.177724+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:52.177724+0000","last_peered":"2024-09-16T10:31:52.177724+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:52.177724+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11227160,"num_objects":2742,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13979,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17820","reported_seq":17882,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.245971+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:52.245971+0000","last_peered":"2024-09-16T10:31:52.245971+0000","last_clean":"2024-09-16T10:31:52.245971+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:52.245971+0000","last_undegraded":"2024-09-16T10:31:52.245971+0000","last_fullsized":"2024-09-16T10:31:52.245971+0000","mapping_epoch":34,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10821632,"num_objects":2642,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17820,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17844","reported_seq":18007,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:53.276382+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:53.276382+0000","last_peered":"2024-09-16T10:31:53.276382+0000","last_clean":"2024-09-16T10:31:53.276382+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:53.276382+0000","last_undegraded":"2024-09-16T10:31:53.276382+0000","last_fullsized":"2024-09-16T10:31:53.276382+0000","mapping_epoch":33,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10899456,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17696","reported_seq":18032,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.228860+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:53.228860+0000","last_peered":"2024-09-16T10:31:53.228860+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:53.228860+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10956800,"num_objects":2675,"num_object_clones":0,"num_object_copies":5350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14321,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1459}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18297","reported_seq":18708,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.245644+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:53.245644+0000","last_peered":"2024-09-16T10:31:53.245644+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:53.245644+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10743808,"num_objects":2623,"num_object_clones":0,"num_object_copies":5246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14539,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17801","reported_seq":17982,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:53.174282+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:53.174282+0000","last_peered":"2024-09-16T10:31:53.174282+0000","last_clean":"2024-09-16T10:31:53.174282+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:53.174282+0000","last_undegraded":"2024-09-16T10:31:53.174282+0000","last_fullsized":"2024-09-16T10:31:53.174282+0000","mapping_epoch":33,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10604544,"num_objects":2589,"num_object_clones":0,"num_object_copies":5178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13604,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17711","reported_seq":26220,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.306456+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:52.306456+0000","last_peered":"2024-09-16T10:31:52.306456+0000","last_clean":"2024-09-16T10:31:52.306456+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:52.306456+0000","last_undegraded":"2024-09-16T10:31:52.306456+0000","last_fullsized":"2024-09-16T10:31:52.306456+0000","mapping_epoch":43,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11264966,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17709,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18297","reported_seq":18576,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.230885+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:53.230885+0000","last_peered":"2024-09-16T10:31:53.230885+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:53.230885+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11236294,"num_objects":2745,"num_object_clones":0,"num_object_copies":5490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18295,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1592}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17725","reported_seq":17925,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:52.235697+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:52.235697+0000","last_peered":"2024-09-16T10:31:52.235697+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:52.235697+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11035590,"num_objects":2696,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1537}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17842","reported_seq":26330,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.272104+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:52.272104+0000","last_peered":"2024-09-16T10:31:52.272104+0000","last_clean":"2024-09-16T10:31:52.272104+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:52.272104+0000","last_undegraded":"2024-09-16T10:31:52.272104+0000","last_fullsized":"2024-09-16T10:31:52.272104+0000","mapping_epoch":47,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11199430,"num_objects":2736,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17840,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17420","reported_seq":17674,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.089318+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:53.089318+0000","last_peered":"2024-09-16T10:31:53.089318+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:53.089318+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10707427,"num_objects":2615,"num_object_clones":0,"num_object_copies":5230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17419,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1460}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18216","reported_seq":18495,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.261079+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:53.261079+0000","last_peered":"2024-09-16T10:31:53.261079+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:53.261079+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11052457,"num_objects":2701,"num_object_clones":0,"num_object_copies":5402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18213,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17518","reported_seq":18664,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.191970+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:53.191970+0000","last_peered":"2024-09-16T10:31:53.191970+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:53.191970+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10740678,"num_objects":2624,"num_object_clones":0,"num_object_copies":5248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17516,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1016}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18242","reported_seq":18487,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.215455+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:53.215455+0000","last_peered":"2024-09-16T10:31:53.215455+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:53.215455+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10872716,"num_objects":2658,"num_object_clones":0,"num_object_copies":5316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18239,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":174899726,"num_objects":42717,"num_object_clones":0,"num_object_copies":85434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42717,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263588,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13732,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":560693248,"data_stored":560616457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160623,"ondisk_log_size":160623,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018993,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552452,"kb_used_data":223860,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93819388,"statfs":{"total":96636764160,"available":96071053312,"internally_reserved":0,"allocated":229232640,"data_stored":228953994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":6.7290000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.179}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.8140000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":6.7919999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691937,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552884,"kb_used_data":129956,"kb_used_omap":4,"kb_used_meta":422907,"kb_avail":93818956,"statfs":{"total":96636764160,"available":96070610944,"internally_reserved":0,"allocated":133074944,"data_stored":132738490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":433057450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.671},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.744}]}]},{"osd":0,"up_from":24,"seq":103079215220,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628216,"kb_used_data":195504,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93743624,"statfs":{"total":96636764160,"available":95993470976,"internally_reserved":0,"allocated":200196096,"data_stored":199925136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":443084456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":5.508},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.312}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.996},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":11.863}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199520256,"data_stored":199516643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132571136,"data_stored":132505643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":228601856,"data_stored":228594171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:58.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:58.664 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:58.665 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:58.690 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-16T10:31:56.732798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":174899745,"num_objects":42719,"num_object_clones":0,"num_object_copies":85438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42719,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263590,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13733,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160626,"ondisk_log_size":160626,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1733552,"kb_used_data":549320,"kb_used_omap":12,"kb_used_meta":1184179,"kb_avail":281381968,"statfs":{"total":289910292480,"available":288135135232,"internally_reserved":0,"allocated":562503680,"data_stored":561617620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1212599298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,9,14,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18931712,"num_objects":-4622,"num_object_clones":0,"num_object_copies":-9244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4622,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002755"},"pg_stats":[{"pgid":"2.f","version":"48'17864","reported_seq":26332,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:53.259571+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:53.259571+0000","last_peered":"2024-09-16T10:31:53.259571+0000","last_clean":"2024-09-16T10:31:53.259571+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:53.259571+0000","last_undegraded":"2024-09-16T10:31:53.259571+0000","last_fullsized":"2024-09-16T10:31:53.259571+0000","mapping_epoch":45,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10665984,"num_objects":2604,"num_object_clones":0,"num_object_copies":5208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17864,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17808","reported_seq":17968,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.189918+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:52.189918+0000","last_peered":"2024-09-16T10:31:52.189918+0000","last_clean":"2024-09-16T10:31:52.189918+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:52.189918+0000","last_undegraded":"2024-09-16T10:31:52.189918+0000","last_fullsized":"2024-09-16T10:31:52.189918+0000","mapping_epoch":34,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10870784,"num_objects":2654,"num_object_clones":0,"num_object_copies":5308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14021,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'17922","reported_seq":18178,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:52.177724+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:52.177724+0000","last_peered":"2024-09-16T10:31:52.177724+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:52.177724+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11227160,"num_objects":2742,"num_object_clones":0,"num_object_copies":5484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13979,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1561}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17820","reported_seq":17882,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.245971+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:52.245971+0000","last_peered":"2024-09-16T10:31:52.245971+0000","last_clean":"2024-09-16T10:31:52.245971+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:52.245971+0000","last_undegraded":"2024-09-16T10:31:52.245971+0000","last_fullsized":"2024-09-16T10:31:52.245971+0000","mapping_epoch":34,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10821632,"num_objects":2642,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17820,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17844","reported_seq":18007,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:53.276382+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:53.276382+0000","last_peered":"2024-09-16T10:31:53.276382+0000","last_clean":"2024-09-16T10:31:53.276382+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:53.276382+0000","last_undegraded":"2024-09-16T10:31:53.276382+0000","last_fullsized":"2024-09-16T10:31:53.276382+0000","mapping_epoch":33,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10899456,"num_objects":2661,"num_object_clones":0,"num_object_copies":5322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17696","reported_seq":18032,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.228860+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:53.228860+0000","last_peered":"2024-09-16T10:31:53.228860+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:53.228860+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10956800,"num_objects":2675,"num_object_clones":0,"num_object_copies":5350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14321,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1459}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18297","reported_seq":18708,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.245644+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:53.245644+0000","last_peered":"2024-09-16T10:31:53.245644+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:53.245644+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10743808,"num_objects":2623,"num_object_clones":0,"num_object_copies":5246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14539,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17801","reported_seq":17982,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:53.174282+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:53.174282+0000","last_peered":"2024-09-16T10:31:53.174282+0000","last_clean":"2024-09-16T10:31:53.174282+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:53.174282+0000","last_undegraded":"2024-09-16T10:31:53.174282+0000","last_fullsized":"2024-09-16T10:31:53.174282+0000","mapping_epoch":33,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10604544,"num_objects":2589,"num_object_clones":0,"num_object_copies":5178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13604,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294331+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:28.294331+0000","last_peered":"2024-09-16T10:31:28.294331+0000","last_clean":"2024-09-16T10:31:28.294331+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:28.294331+0000","last_undegraded":"2024-09-16T10:31:28.294331+0000","last_fullsized":"2024-09-16T10:31:28.294331+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17711","reported_seq":26220,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.306456+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:52.306456+0000","last_peered":"2024-09-16T10:31:52.306456+0000","last_clean":"2024-09-16T10:31:52.306456+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:52.306456+0000","last_undegraded":"2024-09-16T10:31:52.306456+0000","last_fullsized":"2024-09-16T10:31:52.306456+0000","mapping_epoch":43,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11264966,"num_objects":2752,"num_object_clones":0,"num_object_copies":5504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17709,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18297","reported_seq":18576,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.230885+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:53.230885+0000","last_peered":"2024-09-16T10:31:53.230885+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:53.230885+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11236294,"num_objects":2745,"num_object_clones":0,"num_object_copies":5490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18295,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1592}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17725","reported_seq":17925,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:52.235697+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:52.235697+0000","last_peered":"2024-09-16T10:31:52.235697+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:52.235697+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11035590,"num_objects":2696,"num_object_clones":0,"num_object_copies":5392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17723,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1537}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17842","reported_seq":26330,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:52.272104+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:52.272104+0000","last_peered":"2024-09-16T10:31:52.272104+0000","last_clean":"2024-09-16T10:31:52.272104+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:52.272104+0000","last_undegraded":"2024-09-16T10:31:52.272104+0000","last_fullsized":"2024-09-16T10:31:52.272104+0000","mapping_epoch":47,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11199430,"num_objects":2736,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17840,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17420","reported_seq":17674,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.089318+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:53.089318+0000","last_peered":"2024-09-16T10:31:53.089318+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:53.089318+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7400","ondisk_log_start":"42'7400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10707427,"num_objects":2615,"num_object_clones":0,"num_object_copies":5230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17419,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1460}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18216","reported_seq":18495,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.261079+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:53.261079+0000","last_peered":"2024-09-16T10:31:53.261079+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:53.261079+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11052457,"num_objects":2701,"num_object_clones":0,"num_object_copies":5402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18213,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17518","reported_seq":18664,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.191970+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:53.191970+0000","last_peered":"2024-09-16T10:31:53.191970+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:53.191970+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10740678,"num_objects":2624,"num_object_clones":0,"num_object_copies":5248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17516,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1016}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18242","reported_seq":18487,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:53.215455+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:53.215455+0000","last_peered":"2024-09-16T10:31:53.215455+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:53.215455+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10872716,"num_objects":2658,"num_object_clones":0,"num_object_copies":5316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18239,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1664}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":174899726,"num_objects":42717,"num_object_clones":0,"num_object_copies":85434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42717,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263588,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13732,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":560693248,"data_stored":560616457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160623,"ondisk_log_size":160623,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018993,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552452,"kb_used_data":223860,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93819388,"statfs":{"total":96636764160,"available":96071053312,"internally_reserved":0,"allocated":229232640,"data_stored":228953994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":6.7290000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.179}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.8140000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":6.7919999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691937,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552884,"kb_used_data":129956,"kb_used_omap":4,"kb_used_meta":422907,"kb_avail":93818956,"statfs":{"total":96636764160,"available":96070610944,"internally_reserved":0,"allocated":133074944,"data_stored":132738490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":433057450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,4,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.671},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.744}]}]},{"osd":0,"up_from":24,"seq":103079215220,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":628216,"kb_used_data":195504,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93743624,"statfs":{"total":96636764160,"available":95993470976,"internally_reserved":0,"allocated":200196096,"data_stored":199925136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":443084456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,4,7,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":5.508},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.312}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.996},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":11.863}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199520256,"data_stored":199516643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132571136,"data_stored":132505643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":228601856,"data_stored":228594171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:58.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:31:59.246 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:59.247 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:31:59.273 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-16T10:31:58.734504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165224993,"num_objects":40357,"num_object_clones":0,"num_object_copies":80714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40357,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":265952,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13815,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160988,"ondisk_log_size":160988,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1695800,"kb_used_data":532032,"kb_used_omap":12,"kb_used_meta":1163699,"kb_avail":281419720,"statfs":{"total":289910292480,"available":288173793280,"internally_reserved":0,"allocated":544800768,"data_stored":543914708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1191627778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,8,7,6,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28606464,"num_objects":-6984,"num_object_clones":0,"num_object_copies":-13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003987"},"pg_stats":[{"pgid":"2.f","version":"48'18024","reported_seq":26492,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.134528+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:58.134528+0000","last_peered":"2024-09-16T10:31:58.134528+0000","last_clean":"2024-09-16T10:31:58.134528+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:58.134528+0000","last_undegraded":"2024-09-16T10:31:58.134528+0000","last_fullsized":"2024-09-16T10:31:58.134528+0000","mapping_epoch":45,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10010624,"num_objects":2444,"num_object_clones":0,"num_object_copies":4888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18024,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17938","reported_seq":18101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.199042+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:57.199042+0000","last_peered":"2024-09-16T10:31:57.199042+0000","last_clean":"2024-09-16T10:31:57.199042+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:57.199042+0000","last_undegraded":"2024-09-16T10:31:57.199042+0000","last_fullsized":"2024-09-16T10:31:57.199042+0000","mapping_epoch":34,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10338304,"num_objects":2524,"num_object_clones":0,"num_object_copies":5048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14151,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18061","reported_seq":18317,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:57.279722+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:57.279722+0000","last_peered":"2024-09-16T10:31:57.279722+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:57.279722+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10657816,"num_objects":2603,"num_object_clones":0,"num_object_copies":5206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14118,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1700}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17975","reported_seq":18037,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.326969+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:57.326969+0000","last_peered":"2024-09-16T10:31:57.326969+0000","last_clean":"2024-09-16T10:31:57.326969+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:57.326969+0000","last_undegraded":"2024-09-16T10:31:57.326969+0000","last_fullsized":"2024-09-16T10:31:57.326969+0000","mapping_epoch":34,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10186752,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17975,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17999","reported_seq":18162,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.307419+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:58.307419+0000","last_peered":"2024-09-16T10:31:58.307419+0000","last_clean":"2024-09-16T10:31:58.307419+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:58.307419+0000","last_undegraded":"2024-09-16T10:31:58.307419+0000","last_fullsized":"2024-09-16T10:31:58.307419+0000","mapping_epoch":33,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10264576,"num_objects":2506,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14641,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17845","reported_seq":18180,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.242791+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:58.242791+0000","last_peered":"2024-09-16T10:31:58.242791+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:58.242791+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10346496,"num_objects":2526,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14470,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18436","reported_seq":18848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.243646+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:58.243646+0000","last_peered":"2024-09-16T10:31:58.243646+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:58.243646+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'8400","ondisk_log_start":"42'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10174464,"num_objects":2484,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17963","reported_seq":18144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.295993+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:58.295993+0000","last_peered":"2024-09-16T10:31:58.295993+0000","last_clean":"2024-09-16T10:31:58.295993+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:58.295993+0000","last_undegraded":"2024-09-16T10:31:58.295993+0000","last_fullsized":"2024-09-16T10:31:58.295993+0000","mapping_epoch":33,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9940992,"num_objects":2427,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13766,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17854","reported_seq":26363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.314873+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:57.314873+0000","last_peered":"2024-09-16T10:31:57.314873+0000","last_clean":"2024-09-16T10:31:57.314873+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:57.314873+0000","last_undegraded":"2024-09-16T10:31:57.314873+0000","last_fullsized":"2024-09-16T10:31:57.314873+0000","mapping_epoch":43,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679238,"num_objects":2609,"num_object_clones":0,"num_object_copies":5218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17852,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18465","reported_seq":18740,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.323763+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:58.323763+0000","last_peered":"2024-09-16T10:31:58.323763+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:58.323763+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8400","ondisk_log_start":"42'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10548166,"num_objects":2577,"num_object_clones":0,"num_object_copies":5154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18463,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1760}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17858","reported_seq":18058,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:57.271049+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:57.271049+0000","last_peered":"2024-09-16T10:31:57.271049+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:57.271049+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10490822,"num_objects":2563,"num_object_clones":0,"num_object_copies":5126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17856,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17997","reported_seq":26485,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.312110+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:57.312110+0000","last_peered":"2024-09-16T10:31:57.312110+0000","last_clean":"2024-09-16T10:31:57.312110+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:57.312110+0000","last_undegraded":"2024-09-16T10:31:57.312110+0000","last_fullsized":"2024-09-16T10:31:57.312110+0000","mapping_epoch":47,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10564550,"num_objects":2581,"num_object_clones":0,"num_object_copies":5162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17995,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17575","reported_seq":17829,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.283434+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:58.283434+0000","last_peered":"2024-09-16T10:31:58.283434+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:58.283434+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10072547,"num_objects":2460,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17574,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18347","reported_seq":18626,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.275448+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:58.275448+0000","last_peered":"2024-09-16T10:31:58.275448+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:58.275448+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10515881,"num_objects":2570,"num_object_clones":0,"num_object_copies":5140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18344,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1697}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17670","reported_seq":18980,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.262915+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:58.262915+0000","last_peered":"2024-09-16T10:31:58.262915+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:58.262915+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118086,"num_objects":2472,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3558,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17668,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18378","reported_seq":18623,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.286813+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:58.286813+0000","last_peered":"2024-09-16T10:31:58.286813+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:58.286813+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10315660,"num_objects":2522,"num_object_clones":0,"num_object_copies":5044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18375,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1800}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":165224974,"num_objects":40355,"num_object_clones":0,"num_object_copies":80710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40355,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":265950,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13814,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":541577216,"data_stored":541500425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160985,"ondisk_log_size":160985,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018994,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535512,"kb_used_data":219204,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93836328,"statfs":{"total":96636764160,"available":96088399872,"internally_reserved":0,"allocated":224464896,"data_stored":224186250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":323874480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":0.97399999999999998},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":2.9950000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":8.3439999999999994}]}]},{"osd":1,"up_from":40,"seq":171798691938,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535048,"kb_used_data":124412,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93836792,"statfs":{"total":96636764160,"available":96088875008,"internally_reserved":0,"allocated":127397888,"data_stored":127061434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":420474538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.9910000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":2.0179999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.925},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.8240000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215221,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625240,"kb_used_data":188416,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93746600,"statfs":{"total":96636764160,"available":95996518400,"internally_reserved":0,"allocated":192937984,"data_stored":192667024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.4129999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.45600000000000002},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.782}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192217088,"data_stored":192213475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":125501440,"data_stored":125435947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":223858688,"data_stored":223851003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:31:59.274 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:31:59.274 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'18061", 'reported_seq': 18317, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:57.279722+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:31:57.279722+0000', 'last_peered': '2024-09-16T10:31:57.279722+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:31:57.279722+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'8000", 'ondisk_log_start': "42'8000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10657816, 'num_objects': 2603, 'num_object_clones': 0, 'num_object_copies': 5206, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2603, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14118, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1700}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:59.274 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:31:59.274 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'17845", 'reported_seq': 18180, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.242791+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:58.242791+0000', 'last_peered': '2024-09-16T10:31:58.242791+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:58.242791+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'7800", 'ondisk_log_start': "42'7800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10346496, 'num_objects': 2526, 'num_object_clones': 0, 'num_object_copies': 5052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14470, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:31:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'18436", 'reported_seq': 18848, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.243646+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:58.243646+0000', 'last_peered': '2024-09-16T10:31:58.243646+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:58.243646+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'8400", 'ondisk_log_start': "42'8400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10174464, 'num_objects': 2484, 'num_object_clones': 0, 'num_object_copies': 4968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14678, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1803}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:31:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'18465", 'reported_seq': 18740, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.323763+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:58.323763+0000', 'last_peered': '2024-09-16T10:31:58.323763+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:58.323763+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'8400", 'ondisk_log_start': "42'8400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10548166, 'num_objects': 2577, 'num_object_clones': 0, 'num_object_copies': 5154, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18463, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1760}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:31:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'17858", 'reported_seq': 18058, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:57.271049+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:31:57.271049+0000', 'last_peered': '2024-09-16T10:31:57.271049+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:31:57.271049+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'7800", 'ondisk_log_start': "42'7800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10490822, 'num_objects': 2563, 'num_object_clones': 0, 'num_object_copies': 5126, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17856, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1670}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:31:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:31:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'17575", 'reported_seq': 17829, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.283434+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:58.283434+0000', 'last_peered': '2024-09-16T10:31:58.283434+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:58.283434+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'7500", 'ondisk_log_start': "42'7500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10072547, 'num_objects': 2460, 'num_object_clones': 0, 'num_object_copies': 4920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17574, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1615}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:31:59.275 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'18347", 'reported_seq': 18626, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.275448+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:58.275448+0000', 'last_peered': '2024-09-16T10:31:58.275448+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:58.275448+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'8300", 'ondisk_log_start': "42'8300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10515881, 'num_objects': 2570, 'num_object_clones': 0, 'num_object_copies': 5140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18344, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1697}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:31:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'17670", 'reported_seq': 18980, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.262915+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:31:58.262915+0000', 'last_peered': '2024-09-16T10:31:58.262915+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:58.262915+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'7600", 'ondisk_log_start': "42'7600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10118086, 'num_objects': 2472, 'num_object_clones': 0, 'num_object_copies': 4944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3558, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17668, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 613, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:31:59.276 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'18378", 'reported_seq': 18623, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.286813+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:58.286813+0000', 'last_peered': '2024-09-16T10:31:58.286813+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:58.286813+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'8300", 'ondisk_log_start': "42'8300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10315660, 'num_objects': 2522, 'num_object_clones': 0, 'num_object_copies': 5044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18375, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1800}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:31:59.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:31:59.984 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:31:59.984 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37532/80714 objects degraded (46.500%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":608,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":40357,"data_bytes":165224993,"bytes_used":1736499200,"bytes_avail":288173793280,"bytes_total":289910292480,"degraded_objects":37532,"degraded_total":80714,"degraded_ratio":0.4649998761057561,"recovering_objects_per_sec":19,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":238,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":581},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:00.005 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:00.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:00.551 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:00.552 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:00.573 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-16T10:31:58.734504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165224993,"num_objects":40357,"num_object_clones":0,"num_object_copies":80714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40357,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":265952,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13815,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160988,"ondisk_log_size":160988,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1695800,"kb_used_data":532032,"kb_used_omap":12,"kb_used_meta":1163699,"kb_avail":281419720,"statfs":{"total":289910292480,"available":288173793280,"internally_reserved":0,"allocated":544800768,"data_stored":543914708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1191627778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,8,7,6,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28606464,"num_objects":-6984,"num_object_clones":0,"num_object_copies":-13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003987"},"pg_stats":[{"pgid":"2.f","version":"48'18024","reported_seq":26492,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.134528+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:58.134528+0000","last_peered":"2024-09-16T10:31:58.134528+0000","last_clean":"2024-09-16T10:31:58.134528+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:58.134528+0000","last_undegraded":"2024-09-16T10:31:58.134528+0000","last_fullsized":"2024-09-16T10:31:58.134528+0000","mapping_epoch":45,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10010624,"num_objects":2444,"num_object_clones":0,"num_object_copies":4888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18024,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'17938","reported_seq":18101,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.199042+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:31:57.199042+0000","last_peered":"2024-09-16T10:31:57.199042+0000","last_clean":"2024-09-16T10:31:57.199042+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:31:57.199042+0000","last_undegraded":"2024-09-16T10:31:57.199042+0000","last_fullsized":"2024-09-16T10:31:57.199042+0000","mapping_epoch":34,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10338304,"num_objects":2524,"num_object_clones":0,"num_object_copies":5048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14151,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18061","reported_seq":18317,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:57.279722+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:31:57.279722+0000","last_peered":"2024-09-16T10:31:57.279722+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:31:57.279722+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10657816,"num_objects":2603,"num_object_clones":0,"num_object_copies":5206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14118,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1700}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'17975","reported_seq":18037,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.326969+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:31:57.326969+0000","last_peered":"2024-09-16T10:31:57.326969+0000","last_clean":"2024-09-16T10:31:57.326969+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:31:57.326969+0000","last_undegraded":"2024-09-16T10:31:57.326969+0000","last_fullsized":"2024-09-16T10:31:57.326969+0000","mapping_epoch":34,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10186752,"num_objects":2487,"num_object_clones":0,"num_object_copies":4974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17975,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17999","reported_seq":18162,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.307419+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:58.307419+0000","last_peered":"2024-09-16T10:31:58.307419+0000","last_clean":"2024-09-16T10:31:58.307419+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:58.307419+0000","last_undegraded":"2024-09-16T10:31:58.307419+0000","last_fullsized":"2024-09-16T10:31:58.307419+0000","mapping_epoch":33,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10264576,"num_objects":2506,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14641,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17845","reported_seq":18180,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.242791+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:58.242791+0000","last_peered":"2024-09-16T10:31:58.242791+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:58.242791+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10346496,"num_objects":2526,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14470,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18436","reported_seq":18848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.243646+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:58.243646+0000","last_peered":"2024-09-16T10:31:58.243646+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:58.243646+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'8400","ondisk_log_start":"42'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10174464,"num_objects":2484,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17963","reported_seq":18144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.295993+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:58.295993+0000","last_peered":"2024-09-16T10:31:58.295993+0000","last_clean":"2024-09-16T10:31:58.295993+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:58.295993+0000","last_undegraded":"2024-09-16T10:31:58.295993+0000","last_fullsized":"2024-09-16T10:31:58.295993+0000","mapping_epoch":33,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9940992,"num_objects":2427,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13766,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'17854","reported_seq":26363,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.314873+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:31:57.314873+0000","last_peered":"2024-09-16T10:31:57.314873+0000","last_clean":"2024-09-16T10:31:57.314873+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:31:57.314873+0000","last_undegraded":"2024-09-16T10:31:57.314873+0000","last_fullsized":"2024-09-16T10:31:57.314873+0000","mapping_epoch":43,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10679238,"num_objects":2609,"num_object_clones":0,"num_object_copies":5218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17852,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18465","reported_seq":18740,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.323763+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:58.323763+0000","last_peered":"2024-09-16T10:31:58.323763+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:58.323763+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8400","ondisk_log_start":"42'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10548166,"num_objects":2577,"num_object_clones":0,"num_object_copies":5154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18463,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1760}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'17858","reported_seq":18058,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:57.271049+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:31:57.271049+0000","last_peered":"2024-09-16T10:31:57.271049+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:31:57.271049+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10490822,"num_objects":2563,"num_object_clones":0,"num_object_copies":5126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17856,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'17997","reported_seq":26485,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:57.312110+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:31:57.312110+0000","last_peered":"2024-09-16T10:31:57.312110+0000","last_clean":"2024-09-16T10:31:57.312110+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:31:57.312110+0000","last_undegraded":"2024-09-16T10:31:57.312110+0000","last_fullsized":"2024-09-16T10:31:57.312110+0000","mapping_epoch":47,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10564550,"num_objects":2581,"num_object_clones":0,"num_object_copies":5162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17995,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17575","reported_seq":17829,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.283434+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:58.283434+0000","last_peered":"2024-09-16T10:31:58.283434+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:58.283434+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10072547,"num_objects":2460,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17574,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18347","reported_seq":18626,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.275448+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:58.275448+0000","last_peered":"2024-09-16T10:31:58.275448+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:58.275448+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10515881,"num_objects":2570,"num_object_clones":0,"num_object_copies":5140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18344,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1697}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17670","reported_seq":18980,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.262915+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:58.262915+0000","last_peered":"2024-09-16T10:31:58.262915+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:58.262915+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118086,"num_objects":2472,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3558,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17668,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18378","reported_seq":18623,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.286813+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:58.286813+0000","last_peered":"2024-09-16T10:31:58.286813+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:58.286813+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10315660,"num_objects":2522,"num_object_clones":0,"num_object_copies":5044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18375,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1800}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":165224974,"num_objects":40355,"num_object_clones":0,"num_object_copies":80710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40355,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":265950,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13814,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":541577216,"data_stored":541500425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160985,"ondisk_log_size":160985,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018994,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535512,"kb_used_data":219204,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93836328,"statfs":{"total":96636764160,"available":96088399872,"internally_reserved":0,"allocated":224464896,"data_stored":224186250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":323874480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":0.97399999999999998},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":2.9950000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":8.3439999999999994}]}]},{"osd":1,"up_from":40,"seq":171798691938,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535048,"kb_used_data":124412,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93836792,"statfs":{"total":96636764160,"available":96088875008,"internally_reserved":0,"allocated":127397888,"data_stored":127061434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":420474538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":1.9910000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":2.0179999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.925},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.8240000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215221,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625240,"kb_used_data":188416,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93746600,"statfs":{"total":96636764160,"available":95996518400,"internally_reserved":0,"allocated":192937984,"data_stored":192667024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.4129999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.45600000000000002},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.782}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192217088,"data_stored":192213475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":125501440,"data_stored":125435947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":223858688,"data_stored":223851003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:03.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:04.147 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:04.172 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-16T10:32:02.735416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161739297,"num_objects":39506,"num_object_clones":0,"num_object_copies":79012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":266803,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13815,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1695176,"kb_used_data":519132,"kb_used_omap":12,"kb_used_meta":1175987,"kb_avail":281420344,"statfs":{"total":289910292480,"available":288174432256,"internally_reserved":0,"allocated":531591168,"data_stored":530705108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1204210690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,10,7,7,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22515712,"num_objects":-5497,"num_object_clones":0,"num_object_copies":-10994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5497,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003980"},"pg_stats":[{"pgid":"2.f","version":"48'18024","reported_seq":26492,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.134528+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:58.134528+0000","last_peered":"2024-09-16T10:31:58.134528+0000","last_clean":"2024-09-16T10:31:58.134528+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:58.134528+0000","last_undegraded":"2024-09-16T10:31:58.134528+0000","last_fullsized":"2024-09-16T10:31:58.134528+0000","mapping_epoch":45,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10010624,"num_objects":2444,"num_object_clones":0,"num_object_copies":4888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18024,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18085","reported_seq":18248,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.297333+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:02.297333+0000","last_peered":"2024-09-16T10:32:02.297333+0000","last_clean":"2024-09-16T10:32:02.297333+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:02.297333+0000","last_undegraded":"2024-09-16T10:32:02.297333+0000","last_fullsized":"2024-09-16T10:32:02.297333+0000","mapping_epoch":34,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9736192,"num_objects":2377,"num_object_clones":0,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14298,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18187","reported_seq":18443,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:02.322648+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:02.322648+0000","last_peered":"2024-09-16T10:32:02.322648+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:02.322648+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10141720,"num_objects":2477,"num_object_clones":0,"num_object_copies":4954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14244,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18099","reported_seq":18162,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.297737+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:02.297737+0000","last_peered":"2024-09-16T10:32:02.297737+0000","last_clean":"2024-09-16T10:32:02.297737+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:02.297737+0000","last_undegraded":"2024-09-16T10:32:02.297737+0000","last_fullsized":"2024-09-16T10:32:02.297737+0000","mapping_epoch":34,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9678848,"num_objects":2363,"num_object_clones":0,"num_object_copies":4726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18099,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17999","reported_seq":18162,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.307419+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:58.307419+0000","last_peered":"2024-09-16T10:31:58.307419+0000","last_clean":"2024-09-16T10:31:58.307419+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:58.307419+0000","last_undegraded":"2024-09-16T10:31:58.307419+0000","last_fullsized":"2024-09-16T10:31:58.307419+0000","mapping_epoch":33,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10264576,"num_objects":2506,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14641,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17845","reported_seq":18180,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.242791+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:58.242791+0000","last_peered":"2024-09-16T10:31:58.242791+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:58.242791+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10346496,"num_objects":2526,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14470,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18436","reported_seq":18848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.243646+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:58.243646+0000","last_peered":"2024-09-16T10:31:58.243646+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:58.243646+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'8400","ondisk_log_start":"42'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10174464,"num_objects":2484,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17963","reported_seq":18144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.295993+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:58.295993+0000","last_peered":"2024-09-16T10:31:58.295993+0000","last_clean":"2024-09-16T10:31:58.295993+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:58.295993+0000","last_undegraded":"2024-09-16T10:31:58.295993+0000","last_fullsized":"2024-09-16T10:31:58.295993+0000","mapping_epoch":33,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9940992,"num_objects":2427,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13766,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18011","reported_seq":26520,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.419930+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:02.419930+0000","last_peered":"2024-09-16T10:32:02.419930+0000","last_clean":"2024-09-16T10:32:02.419930+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:02.419930+0000","last_undegraded":"2024-09-16T10:32:02.419930+0000","last_fullsized":"2024-09-16T10:32:02.419930+0000","mapping_epoch":43,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10036166,"num_objects":2452,"num_object_clones":0,"num_object_copies":4904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18009,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18465","reported_seq":18740,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.323763+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:58.323763+0000","last_peered":"2024-09-16T10:31:58.323763+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:58.323763+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8400","ondisk_log_start":"42'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10548166,"num_objects":2577,"num_object_clones":0,"num_object_copies":5154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18463,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1760}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18003","reported_seq":18203,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:02.323696+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:02.323696+0000","last_peered":"2024-09-16T10:32:02.323696+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:02.323696+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9896902,"num_objects":2418,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18001,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1815}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18149","reported_seq":26637,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.355321+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:02.355321+0000","last_peered":"2024-09-16T10:32:02.355321+0000","last_clean":"2024-09-16T10:32:02.355321+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:02.355321+0000","last_undegraded":"2024-09-16T10:32:02.355321+0000","last_fullsized":"2024-09-16T10:32:02.355321+0000","mapping_epoch":47,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9941958,"num_objects":2429,"num_object_clones":0,"num_object_copies":4858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18147,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17575","reported_seq":17829,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.283434+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:58.283434+0000","last_peered":"2024-09-16T10:31:58.283434+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:58.283434+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10072547,"num_objects":2460,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17574,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18347","reported_seq":18626,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.275448+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:58.275448+0000","last_peered":"2024-09-16T10:31:58.275448+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:58.275448+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10515881,"num_objects":2570,"num_object_clones":0,"num_object_copies":5140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18344,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1697}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17670","reported_seq":18980,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.262915+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:58.262915+0000","last_peered":"2024-09-16T10:31:58.262915+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:58.262915+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118086,"num_objects":2472,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3558,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17668,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18378","reported_seq":18623,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.286813+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:58.286813+0000","last_peered":"2024-09-16T10:31:58.286813+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:58.286813+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10315660,"num_objects":2522,"num_object_clones":0,"num_object_copies":5044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18375,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1800}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":161739278,"num_objects":39504,"num_object_clones":0,"num_object_copies":79008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":266801,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13814,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":529690624,"data_stored":529613833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160936,"ondisk_log_size":160936,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018995,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535124,"kb_used_data":214732,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93836716,"statfs":{"total":96636764160,"available":96088797184,"internally_reserved":0,"allocated":219885568,"data_stored":219606922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":3.9910000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.796}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.35399999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691939,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534812,"kb_used_data":115984,"kb_used_omap":4,"kb_used_meta":418811,"kb_avail":93837028,"statfs":{"total":96636764160,"available":96089116672,"internally_reserved":0,"allocated":118767616,"data_stored":118431162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":428863146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.27200000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215221,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625240,"kb_used_data":188416,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93746600,"statfs":{"total":96636764160,"available":95996518400,"internally_reserved":0,"allocated":192937984,"data_stored":192667024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.4129999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.45600000000000002},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.782}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192217088,"data_stored":192213475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118337536,"data_stored":118272043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219136000,"data_stored":219128315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:04.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:04.720 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:04.723 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:04.745 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-16T10:32:02.735416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161739297,"num_objects":39506,"num_object_clones":0,"num_object_copies":79012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":266803,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13815,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1695176,"kb_used_data":519132,"kb_used_omap":12,"kb_used_meta":1175987,"kb_avail":281420344,"statfs":{"total":289910292480,"available":288174432256,"internally_reserved":0,"allocated":531591168,"data_stored":530705108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1204210690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,10,7,7,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22515712,"num_objects":-5497,"num_object_clones":0,"num_object_copies":-10994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5497,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003980"},"pg_stats":[{"pgid":"2.f","version":"48'18024","reported_seq":26492,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.134528+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:31:58.134528+0000","last_peered":"2024-09-16T10:31:58.134528+0000","last_clean":"2024-09-16T10:31:58.134528+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:31:58.134528+0000","last_undegraded":"2024-09-16T10:31:58.134528+0000","last_fullsized":"2024-09-16T10:31:58.134528+0000","mapping_epoch":45,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10010624,"num_objects":2444,"num_object_clones":0,"num_object_copies":4888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18024,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18085","reported_seq":18248,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.297333+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:02.297333+0000","last_peered":"2024-09-16T10:32:02.297333+0000","last_clean":"2024-09-16T10:32:02.297333+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:02.297333+0000","last_undegraded":"2024-09-16T10:32:02.297333+0000","last_fullsized":"2024-09-16T10:32:02.297333+0000","mapping_epoch":34,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9736192,"num_objects":2377,"num_object_clones":0,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14298,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18187","reported_seq":18443,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:02.322648+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:02.322648+0000","last_peered":"2024-09-16T10:32:02.322648+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:02.322648+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10141720,"num_objects":2477,"num_object_clones":0,"num_object_copies":4954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14244,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18099","reported_seq":18162,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.297737+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:02.297737+0000","last_peered":"2024-09-16T10:32:02.297737+0000","last_clean":"2024-09-16T10:32:02.297737+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:02.297737+0000","last_undegraded":"2024-09-16T10:32:02.297737+0000","last_fullsized":"2024-09-16T10:32:02.297737+0000","mapping_epoch":34,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9678848,"num_objects":2363,"num_object_clones":0,"num_object_copies":4726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18099,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'17999","reported_seq":18162,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.307419+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:31:58.307419+0000","last_peered":"2024-09-16T10:31:58.307419+0000","last_clean":"2024-09-16T10:31:58.307419+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:31:58.307419+0000","last_undegraded":"2024-09-16T10:31:58.307419+0000","last_fullsized":"2024-09-16T10:31:58.307419+0000","mapping_epoch":33,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10264576,"num_objects":2506,"num_object_clones":0,"num_object_copies":5012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14641,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17845","reported_seq":18180,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.242791+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:31:58.242791+0000","last_peered":"2024-09-16T10:31:58.242791+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:31:58.242791+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10346496,"num_objects":2526,"num_object_clones":0,"num_object_copies":5052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14470,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18436","reported_seq":18848,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.243646+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:31:58.243646+0000","last_peered":"2024-09-16T10:31:58.243646+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:31:58.243646+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'8400","ondisk_log_start":"42'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10174464,"num_objects":2484,"num_object_clones":0,"num_object_copies":4968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14678,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1803}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'17963","reported_seq":18144,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:58.295993+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:31:58.295993+0000","last_peered":"2024-09-16T10:31:58.295993+0000","last_clean":"2024-09-16T10:31:58.295993+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:31:58.295993+0000","last_undegraded":"2024-09-16T10:31:58.295993+0000","last_fullsized":"2024-09-16T10:31:58.295993+0000","mapping_epoch":33,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9940992,"num_objects":2427,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13766,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18011","reported_seq":26520,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.419930+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:02.419930+0000","last_peered":"2024-09-16T10:32:02.419930+0000","last_clean":"2024-09-16T10:32:02.419930+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:02.419930+0000","last_undegraded":"2024-09-16T10:32:02.419930+0000","last_fullsized":"2024-09-16T10:32:02.419930+0000","mapping_epoch":43,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10036166,"num_objects":2452,"num_object_clones":0,"num_object_copies":4904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18009,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18465","reported_seq":18740,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.323763+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:31:58.323763+0000","last_peered":"2024-09-16T10:31:58.323763+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:31:58.323763+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"42'8400","ondisk_log_start":"42'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10548166,"num_objects":2577,"num_object_clones":0,"num_object_copies":5154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18463,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1760}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18003","reported_seq":18203,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:02.323696+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:02.323696+0000","last_peered":"2024-09-16T10:32:02.323696+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:02.323696+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9896902,"num_objects":2418,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18001,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1815}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18149","reported_seq":26637,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.355321+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:02.355321+0000","last_peered":"2024-09-16T10:32:02.355321+0000","last_clean":"2024-09-16T10:32:02.355321+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:02.355321+0000","last_undegraded":"2024-09-16T10:32:02.355321+0000","last_fullsized":"2024-09-16T10:32:02.355321+0000","mapping_epoch":47,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9941958,"num_objects":2429,"num_object_clones":0,"num_object_copies":4858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18147,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17575","reported_seq":17829,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.283434+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:31:58.283434+0000","last_peered":"2024-09-16T10:31:58.283434+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:31:58.283434+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7500","ondisk_log_start":"42'7500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10072547,"num_objects":2460,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17574,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1615}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18347","reported_seq":18626,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.275448+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:31:58.275448+0000","last_peered":"2024-09-16T10:31:58.275448+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:31:58.275448+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10515881,"num_objects":2570,"num_object_clones":0,"num_object_copies":5140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18344,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1697}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17670","reported_seq":18980,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.262915+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:31:58.262915+0000","last_peered":"2024-09-16T10:31:58.262915+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:31:58.262915+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7600","ondisk_log_start":"42'7600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10118086,"num_objects":2472,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3558,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17668,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18378","reported_seq":18623,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:31:58.286813+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:31:58.286813+0000","last_peered":"2024-09-16T10:31:58.286813+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:31:58.286813+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10315660,"num_objects":2522,"num_object_clones":0,"num_object_copies":5044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18375,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1800}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":161739278,"num_objects":39504,"num_object_clones":0,"num_object_copies":79008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39504,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":266801,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13814,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":529690624,"data_stored":529613833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160936,"ondisk_log_size":160936,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018995,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535124,"kb_used_data":214732,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93836716,"statfs":{"total":96636764160,"available":96088797184,"internally_reserved":0,"allocated":219885568,"data_stored":219606922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":3.9910000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.796}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.35399999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691939,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534812,"kb_used_data":115984,"kb_used_omap":4,"kb_used_meta":418811,"kb_avail":93837028,"statfs":{"total":96636764160,"available":96089116672,"internally_reserved":0,"allocated":118767616,"data_stored":118431162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":428863146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.27200000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215221,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":625240,"kb_used_data":188416,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93746600,"statfs":{"total":96636764160,"available":95996518400,"internally_reserved":0,"allocated":192937984,"data_stored":192667024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,3,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":2.4129999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.45600000000000002},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.782}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192217088,"data_stored":192213475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118337536,"data_stored":118272043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219136000,"data_stored":219128315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:04.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:32:04.746 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'18187", 'reported_seq': 18443, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:02.322648+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:02.322648+0000', 'last_peered': '2024-09-16T10:32:02.322648+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:02.322648+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'8100", 'ondisk_log_start': "42'8100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10141720, 'num_objects': 2477, 'num_object_clones': 0, 'num_object_copies': 4954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2477, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14244, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1826}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:04.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:32:04.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'17845", 'reported_seq': 18180, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.242791+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:31:58.242791+0000', 'last_peered': '2024-09-16T10:31:58.242791+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:31:58.242791+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'7800", 'ondisk_log_start': "42'7800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10346496, 'num_objects': 2526, 'num_object_clones': 0, 'num_object_copies': 5052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14470, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:04.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:32:04.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'18436", 'reported_seq': 18848, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.243646+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:31:58.243646+0000', 'last_peered': '2024-09-16T10:31:58.243646+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:31:58.243646+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "42'8400", 'ondisk_log_start': "42'8400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10174464, 'num_objects': 2484, 'num_object_clones': 0, 'num_object_copies': 4968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2484, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14678, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1803}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:04.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:32:04.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'18465", 'reported_seq': 18740, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.323763+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:31:58.323763+0000', 'last_peered': '2024-09-16T10:31:58.323763+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:31:58.323763+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "42'8400", 'ondisk_log_start': "42'8400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10548166, 'num_objects': 2577, 'num_object_clones': 0, 'num_object_copies': 5154, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18463, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1760}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:04.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:32:04.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'18003", 'reported_seq': 18203, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:02.323696+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:02.323696+0000', 'last_peered': '2024-09-16T10:32:02.323696+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:02.323696+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'8000", 'ondisk_log_start': "42'8000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9896902, 'num_objects': 2418, 'num_object_clones': 0, 'num_object_copies': 4836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2418, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18001, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1815}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:04.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:32:04.747 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'17575", 'reported_seq': 17829, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.283434+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:31:58.283434+0000', 'last_peered': '2024-09-16T10:31:58.283434+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:31:58.283434+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'7500", 'ondisk_log_start': "42'7500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10072547, 'num_objects': 2460, 'num_object_clones': 0, 'num_object_copies': 4920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17574, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1615}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:04.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:32:04.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'18347", 'reported_seq': 18626, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.275448+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:31:58.275448+0000', 'last_peered': '2024-09-16T10:31:58.275448+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:31:58.275448+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "42'8300", 'ondisk_log_start': "42'8300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10515881, 'num_objects': 2570, 'num_object_clones': 0, 'num_object_copies': 5140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18344, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1697}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:04.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:32:04.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'17670", 'reported_seq': 18980, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.262915+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:31:58.262915+0000', 'last_peered': '2024-09-16T10:31:58.262915+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:31:58.262915+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'7600", 'ondisk_log_start': "42'7600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10118086, 'num_objects': 2472, 'num_object_clones': 0, 'num_object_copies': 4944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3558, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17668, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 613, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:04.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:32:04.748 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'18378", 'reported_seq': 18623, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:31:58.286813+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:31:58.286813+0000', 'last_peered': '2024-09-16T10:31:58.286813+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:31:58.286813+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "42'8300", 'ondisk_log_start': "42'8300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10315660, 'num_objects': 2522, 'num_object_clones': 0, 'num_object_copies': 5044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18375, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1800}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:04.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:32:05.457 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:05.457 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37444/76058 objects degraded (49.231%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":614,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":38029,"data_bytes":155689505,"bytes_used":1716137984,"bytes_avail":288194154496,"bytes_total":289910292480,"degraded_objects":37444,"degraded_total":76058,"degraded_ratio":0.49230850140682109,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":246,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":510},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:05.482 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:05.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:06.042 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:06.043 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:06.070 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-16T10:32:04.735912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":155689505,"num_objects":38029,"num_object_clones":0,"num_object_copies":76058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268280,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13903,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160816,"ondisk_log_size":160816,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1675916,"kb_used_data":512176,"kb_used_omap":12,"kb_used_meta":1163699,"kb_avail":281439604,"statfs":{"total":289910292480,"available":288194154496,"internally_reserved":0,"allocated":524468224,"data_stored":523582164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1191627778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,8,11,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25120768,"num_objects":-6133,"num_object_clones":0,"num_object_copies":-12266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004076"},"pg_stats":[{"pgid":"2.f","version":"48'18167","reported_seq":26635,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.375538+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:03.375538+0000","last_peered":"2024-09-16T10:32:03.375538+0000","last_clean":"2024-09-16T10:32:03.375538+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:03.375538+0000","last_undegraded":"2024-09-16T10:32:03.375538+0000","last_fullsized":"2024-09-16T10:32:03.375538+0000","mapping_epoch":45,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9424896,"num_objects":2301,"num_object_clones":0,"num_object_copies":4602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18167,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18085","reported_seq":18248,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.297333+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:02.297333+0000","last_peered":"2024-09-16T10:32:02.297333+0000","last_clean":"2024-09-16T10:32:02.297333+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:02.297333+0000","last_undegraded":"2024-09-16T10:32:02.297333+0000","last_fullsized":"2024-09-16T10:32:02.297333+0000","mapping_epoch":34,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9736192,"num_objects":2377,"num_object_clones":0,"num_object_copies":4754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14298,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18187","reported_seq":18443,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:02.322648+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:02.322648+0000","last_peered":"2024-09-16T10:32:02.322648+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:02.322648+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10141720,"num_objects":2477,"num_object_clones":0,"num_object_copies":4954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14244,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1826}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18099","reported_seq":18162,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.297737+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:02.297737+0000","last_peered":"2024-09-16T10:32:02.297737+0000","last_clean":"2024-09-16T10:32:02.297737+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:02.297737+0000","last_undegraded":"2024-09-16T10:32:02.297737+0000","last_fullsized":"2024-09-16T10:32:02.297737+0000","mapping_epoch":34,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9678848,"num_objects":2363,"num_object_clones":0,"num_object_copies":4726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18099,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18168","reported_seq":18331,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.347030+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:03.347030+0000","last_peered":"2024-09-16T10:32:03.347030+0000","last_clean":"2024-09-16T10:32:03.347030+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:03.347030+0000","last_undegraded":"2024-09-16T10:32:03.347030+0000","last_fullsized":"2024-09-16T10:32:03.347030+0000","mapping_epoch":33,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9572352,"num_objects":2337,"num_object_clones":0,"num_object_copies":4674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14810,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'17975","reported_seq":18311,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:03.232658+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:03.232658+0000","last_peered":"2024-09-16T10:32:03.232658+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:03.232658+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9814016,"num_objects":2396,"num_object_clones":0,"num_object_copies":4792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14600,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18591","reported_seq":19001,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:03.355194+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:03.355194+0000","last_peered":"2024-09-16T10:32:03.355194+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:03.355194+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"42'8500","ondisk_log_start":"42'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9539584,"num_objects":2329,"num_object_clones":0,"num_object_copies":4658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14833,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1958}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18107","reported_seq":18288,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:03.299639+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:03.299639+0000","last_peered":"2024-09-16T10:32:03.299639+0000","last_clean":"2024-09-16T10:32:03.299639+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:03.299639+0000","last_undegraded":"2024-09-16T10:32:03.299639+0000","last_fullsized":"2024-09-16T10:32:03.299639+0000","mapping_epoch":33,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9351168,"num_objects":2283,"num_object_clones":0,"num_object_copies":4566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13910,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18011","reported_seq":26520,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.419930+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:02.419930+0000","last_peered":"2024-09-16T10:32:02.419930+0000","last_clean":"2024-09-16T10:32:02.419930+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:02.419930+0000","last_undegraded":"2024-09-16T10:32:02.419930+0000","last_fullsized":"2024-09-16T10:32:02.419930+0000","mapping_epoch":43,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10036166,"num_objects":2452,"num_object_clones":0,"num_object_copies":4904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18009,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18612","reported_seq":18891,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:03.229660+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:03.229660+0000","last_peered":"2024-09-16T10:32:03.229660+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:03.229660+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9946054,"num_objects":2430,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18610,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1907}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18003","reported_seq":18203,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:02.323696+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:02.323696+0000","last_peered":"2024-09-16T10:32:02.323696+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:02.323696+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9896902,"num_objects":2418,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18001,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1815}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18149","reported_seq":26637,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:02.355321+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:02.355321+0000","last_peered":"2024-09-16T10:32:02.355321+0000","last_clean":"2024-09-16T10:32:02.355321+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:02.355321+0000","last_undegraded":"2024-09-16T10:32:02.355321+0000","last_fullsized":"2024-09-16T10:32:02.355321+0000","mapping_epoch":47,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9941958,"num_objects":2429,"num_object_clones":0,"num_object_copies":4858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18147,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17740","reported_seq":17994,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:03.359426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:03.359426+0000","last_peered":"2024-09-16T10:32:03.359426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:03.359426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7700","ondisk_log_start":"42'7700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9396707,"num_objects":2295,"num_object_clones":0,"num_object_copies":4590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17739,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1780}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18480","reported_seq":18758,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:03.416289+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:03.416289+0000","last_peered":"2024-09-16T10:32:03.416289+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:03.416289+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"42'8400","ondisk_log_start":"42'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9971113,"num_objects":2437,"num_object_clones":0,"num_object_copies":4874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18477,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1830}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17822","reported_seq":19308,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:03.312951+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:03.312951+0000","last_peered":"2024-09-16T10:32:03.312951+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:03.312951+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9495494,"num_objects":2320,"num_object_clones":0,"num_object_copies":4640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17820,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":701,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1150}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18517","reported_seq":18762,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:03.285609+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:03.285609+0000","last_peered":"2024-09-16T10:32:03.285609+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:03.285609+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9746316,"num_objects":2383,"num_object_clones":0,"num_object_copies":4766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18514,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":155689486,"num_objects":38027,"num_object_clones":0,"num_object_copies":76054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37444,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":268278,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13902,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":522461184,"data_stored":522384393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160813,"ondisk_log_size":160813,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018995,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535124,"kb_used_data":214732,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93836716,"statfs":{"total":96636764160,"available":96088797184,"internally_reserved":0,"allocated":219885568,"data_stored":219606922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5070000000000001,"5min":2.0800000000000001,"15min":1.359},"min":{"1min":0.20200000000000001,"5min":0.19,"15min":0.184},"max":{"1min":13.74,"5min":13.74,"15min":13.74},"last":3.9910000000000001},{"interface":"front","average":{"1min":2.2290000000000001,"5min":2.0510000000000002,"15min":1.5640000000000001},"min":{"1min":0.29199999999999998,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":13.743,"5min":13.917,"15min":13.917},"last":1.796}]},{"osd":1,"last update":"Mon Sep 16 10:31:05 2024","interfaces":[{"interface":"back","average":{"1min":3.7000000000000002,"5min":3.0960000000000001,"15min":2.9409999999999998},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":8.6869999999999994,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":4.3970000000000002,"5min":3.52,"15min":2.6619999999999999},"min":{"1min":0.23400000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.81,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.35399999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691939,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534812,"kb_used_data":115984,"kb_used_omap":4,"kb_used_meta":418811,"kb_avail":93837028,"statfs":{"total":96636764160,"available":96089116672,"internally_reserved":0,"allocated":118767616,"data_stored":118431162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":428863146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.27200000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215222,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":605980,"kb_used_data":181460,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93765860,"statfs":{"total":96636764160,"available":96016240640,"internally_reserved":0,"allocated":185815040,"data_stored":185544080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":434695848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,7,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":7.8620000000000001},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":0.749}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.63},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":3.1400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184987648,"data_stored":184984035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118337536,"data_stored":118272043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219136000,"data_stored":219128315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:09.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:09.628 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:09.630 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:09.653 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-16T10:32:08.736801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":146199073,"num_objects":35712,"num_object_clones":0,"num_object_copies":71424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270597,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13982,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1675412,"kb_used_data":495292,"kb_used_omap":12,"kb_used_meta":1180083,"kb_avail":281440108,"statfs":{"total":289910292480,"available":288194670592,"internally_reserved":0,"allocated":507179008,"data_stored":506292948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1208404994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,10,0,17,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28700672,"num_objects":-7007,"num_object_clones":0,"num_object_copies":-14014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7007,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004003"},"pg_stats":[{"pgid":"2.f","version":"48'18299","reported_seq":26766,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:08.416622+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:08.416622+0000","last_peered":"2024-09-16T10:32:08.416622+0000","last_clean":"2024-09-16T10:32:08.416622+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:08.416622+0000","last_undegraded":"2024-09-16T10:32:08.416622+0000","last_fullsized":"2024-09-16T10:32:08.416622+0000","mapping_epoch":45,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8884224,"num_objects":2169,"num_object_clones":0,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18299,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18212","reported_seq":18375,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.353497+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:07.353497+0000","last_peered":"2024-09-16T10:32:07.353497+0000","last_clean":"2024-09-16T10:32:07.353497+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:07.353497+0000","last_undegraded":"2024-09-16T10:32:07.353497+0000","last_fullsized":"2024-09-16T10:32:07.353497+0000","mapping_epoch":34,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9216000,"num_objects":2250,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14425,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18326","reported_seq":18583,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:07.316333+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:07.316333+0000","last_peered":"2024-09-16T10:32:07.316333+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:07.316333+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9572376,"num_objects":2338,"num_object_clones":0,"num_object_copies":4676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14383,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1965}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18243","reported_seq":18304,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.364288+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:07.364288+0000","last_peered":"2024-09-16T10:32:07.364288+0000","last_clean":"2024-09-16T10:32:07.364288+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:07.364288+0000","last_undegraded":"2024-09-16T10:32:07.364288+0000","last_fullsized":"2024-09-16T10:32:07.364288+0000","mapping_epoch":34,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9089024,"num_objects":2219,"num_object_clones":0,"num_object_copies":4438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18243,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18301","reported_seq":18464,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:08.418883+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:08.418883+0000","last_peered":"2024-09-16T10:32:08.418883+0000","last_clean":"2024-09-16T10:32:08.418883+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:08.418883+0000","last_undegraded":"2024-09-16T10:32:08.418883+0000","last_fullsized":"2024-09-16T10:32:08.418883+0000","mapping_epoch":33,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9027584,"num_objects":2204,"num_object_clones":0,"num_object_copies":4408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14943,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18135","reported_seq":18471,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.434005+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:08.434005+0000","last_peered":"2024-09-16T10:32:08.434005+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:08.434005+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9158656,"num_objects":2236,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14760,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18738","reported_seq":19150,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.401332+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:08.401332+0000","last_peered":"2024-09-16T10:32:08.401332+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:08.401332+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8937472,"num_objects":2182,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14980,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18271","reported_seq":18451,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:08.435408+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:08.435408+0000","last_peered":"2024-09-16T10:32:08.435408+0000","last_clean":"2024-09-16T10:32:08.435408+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:08.435408+0000","last_undegraded":"2024-09-16T10:32:08.435408+0000","last_fullsized":"2024-09-16T10:32:08.435408+0000","mapping_epoch":33,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8679424,"num_objects":2119,"num_object_clones":0,"num_object_copies":4238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14074,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18167","reported_seq":26676,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.459304+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:07.459304+0000","last_peered":"2024-09-16T10:32:07.459304+0000","last_clean":"2024-09-16T10:32:07.459304+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:07.459304+0000","last_undegraded":"2024-09-16T10:32:07.459304+0000","last_fullsized":"2024-09-16T10:32:07.459304+0000","mapping_epoch":43,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9397190,"num_objects":2296,"num_object_clones":0,"num_object_copies":4592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18165,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18768","reported_seq":19046,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.347849+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:08.347849+0000","last_peered":"2024-09-16T10:32:08.347849+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:08.347849+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9307078,"num_objects":2274,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18766,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2063}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18143","reported_seq":18343,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:07.330460+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:07.330460+0000","last_peered":"2024-09-16T10:32:07.330460+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:07.330460+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9323462,"num_objects":2278,"num_object_clones":0,"num_object_copies":4556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18141,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18291","reported_seq":26778,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.373963+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:07.373963+0000","last_peered":"2024-09-16T10:32:07.373963+0000","last_clean":"2024-09-16T10:32:07.373963+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:07.373963+0000","last_undegraded":"2024-09-16T10:32:07.373963+0000","last_fullsized":"2024-09-16T10:32:07.373963+0000","mapping_epoch":47,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9360326,"num_objects":2287,"num_object_clones":0,"num_object_copies":4574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18289,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17889","reported_seq":18143,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.336425+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:08.336425+0000","last_peered":"2024-09-16T10:32:08.336425+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:08.336425+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8786403,"num_objects":2146,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17888,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1929}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18626","reported_seq":18905,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.395636+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:08.395636+0000","last_peered":"2024-09-16T10:32:08.395636+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:08.395636+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9373097,"num_objects":2291,"num_object_clones":0,"num_object_copies":4582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18623,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17970","reported_seq":19612,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.444237+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:08.444237+0000","last_peered":"2024-09-16T10:32:08.444237+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:08.444237+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8889286,"num_objects":2172,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17968,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18651","reported_seq":18896,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.439945+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:08.439945+0000","last_peered":"2024-09-16T10:32:08.439945+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:08.439945+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9197452,"num_objects":2249,"num_object_clones":0,"num_object_copies":4498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18648,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2073}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":146199054,"num_objects":35710,"num_object_clones":0,"num_object_copies":71420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270595,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13981,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":503582720,"data_stored":503505929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160830,"ondisk_log_size":160830,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018996,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534728,"kb_used_data":210240,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93837112,"statfs":{"total":96636764160,"available":96089202688,"internally_reserved":0,"allocated":215285760,"data_stored":215007114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":8.8119999999999994},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.82599999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691940,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533336,"kb_used_data":110416,"kb_used_omap":4,"kb_used_meta":422907,"kb_avail":93838504,"statfs":{"total":96636764160,"available":96090628096,"internally_reserved":0,"allocated":113065984,"data_stored":112729530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":433057450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.16400000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":5.0499999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.2489999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215223,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607348,"kb_used_data":174636,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93764492,"statfs":{"total":96636764160,"available":96014839808,"internally_reserved":0,"allocated":178827264,"data_stored":178556304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":443084456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":5.0049999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":4.3200000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":5.0439999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":177905664,"data_stored":177902051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111173632,"data_stored":111108139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":214503424,"data_stored":214495739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:09.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:10.190 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:10.191 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:10.203 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-16T10:32:08.736801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":146199073,"num_objects":35712,"num_object_clones":0,"num_object_copies":71424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270597,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13982,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1675412,"kb_used_data":495292,"kb_used_omap":12,"kb_used_meta":1180083,"kb_avail":281440108,"statfs":{"total":289910292480,"available":288194670592,"internally_reserved":0,"allocated":507179008,"data_stored":506292948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1208404994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,10,0,17,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28700672,"num_objects":-7007,"num_object_clones":0,"num_object_copies":-14014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7007,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004003"},"pg_stats":[{"pgid":"2.f","version":"48'18299","reported_seq":26766,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:08.416622+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:08.416622+0000","last_peered":"2024-09-16T10:32:08.416622+0000","last_clean":"2024-09-16T10:32:08.416622+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:08.416622+0000","last_undegraded":"2024-09-16T10:32:08.416622+0000","last_fullsized":"2024-09-16T10:32:08.416622+0000","mapping_epoch":45,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8884224,"num_objects":2169,"num_object_clones":0,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18299,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18212","reported_seq":18375,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.353497+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:07.353497+0000","last_peered":"2024-09-16T10:32:07.353497+0000","last_clean":"2024-09-16T10:32:07.353497+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:07.353497+0000","last_undegraded":"2024-09-16T10:32:07.353497+0000","last_fullsized":"2024-09-16T10:32:07.353497+0000","mapping_epoch":34,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9216000,"num_objects":2250,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14425,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18326","reported_seq":18583,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:07.316333+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:07.316333+0000","last_peered":"2024-09-16T10:32:07.316333+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:07.316333+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9572376,"num_objects":2338,"num_object_clones":0,"num_object_copies":4676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14383,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1965}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18243","reported_seq":18304,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.364288+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:07.364288+0000","last_peered":"2024-09-16T10:32:07.364288+0000","last_clean":"2024-09-16T10:32:07.364288+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:07.364288+0000","last_undegraded":"2024-09-16T10:32:07.364288+0000","last_fullsized":"2024-09-16T10:32:07.364288+0000","mapping_epoch":34,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9089024,"num_objects":2219,"num_object_clones":0,"num_object_copies":4438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18243,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18301","reported_seq":18464,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:08.418883+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:08.418883+0000","last_peered":"2024-09-16T10:32:08.418883+0000","last_clean":"2024-09-16T10:32:08.418883+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:08.418883+0000","last_undegraded":"2024-09-16T10:32:08.418883+0000","last_fullsized":"2024-09-16T10:32:08.418883+0000","mapping_epoch":33,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9027584,"num_objects":2204,"num_object_clones":0,"num_object_copies":4408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14943,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18135","reported_seq":18471,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.434005+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:08.434005+0000","last_peered":"2024-09-16T10:32:08.434005+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:08.434005+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9158656,"num_objects":2236,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14760,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18738","reported_seq":19150,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.401332+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:08.401332+0000","last_peered":"2024-09-16T10:32:08.401332+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:08.401332+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8937472,"num_objects":2182,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14980,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18271","reported_seq":18451,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:08.435408+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:08.435408+0000","last_peered":"2024-09-16T10:32:08.435408+0000","last_clean":"2024-09-16T10:32:08.435408+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:08.435408+0000","last_undegraded":"2024-09-16T10:32:08.435408+0000","last_fullsized":"2024-09-16T10:32:08.435408+0000","mapping_epoch":33,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8679424,"num_objects":2119,"num_object_clones":0,"num_object_copies":4238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14074,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18167","reported_seq":26676,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.459304+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:07.459304+0000","last_peered":"2024-09-16T10:32:07.459304+0000","last_clean":"2024-09-16T10:32:07.459304+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:07.459304+0000","last_undegraded":"2024-09-16T10:32:07.459304+0000","last_fullsized":"2024-09-16T10:32:07.459304+0000","mapping_epoch":43,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9397190,"num_objects":2296,"num_object_clones":0,"num_object_copies":4592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18165,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18768","reported_seq":19046,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.347849+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:08.347849+0000","last_peered":"2024-09-16T10:32:08.347849+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:08.347849+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9307078,"num_objects":2274,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18766,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2063}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18143","reported_seq":18343,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:07.330460+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:07.330460+0000","last_peered":"2024-09-16T10:32:07.330460+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:07.330460+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9323462,"num_objects":2278,"num_object_clones":0,"num_object_copies":4556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18141,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18291","reported_seq":26778,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.373963+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:07.373963+0000","last_peered":"2024-09-16T10:32:07.373963+0000","last_clean":"2024-09-16T10:32:07.373963+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:07.373963+0000","last_undegraded":"2024-09-16T10:32:07.373963+0000","last_fullsized":"2024-09-16T10:32:07.373963+0000","mapping_epoch":47,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9360326,"num_objects":2287,"num_object_clones":0,"num_object_copies":4574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18289,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17889","reported_seq":18143,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.336425+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:08.336425+0000","last_peered":"2024-09-16T10:32:08.336425+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:08.336425+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8786403,"num_objects":2146,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17888,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1929}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18626","reported_seq":18905,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.395636+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:08.395636+0000","last_peered":"2024-09-16T10:32:08.395636+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:08.395636+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9373097,"num_objects":2291,"num_object_clones":0,"num_object_copies":4582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18623,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17970","reported_seq":19612,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.444237+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:08.444237+0000","last_peered":"2024-09-16T10:32:08.444237+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:08.444237+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8889286,"num_objects":2172,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17968,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18651","reported_seq":18896,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.439945+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:08.439945+0000","last_peered":"2024-09-16T10:32:08.439945+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:08.439945+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9197452,"num_objects":2249,"num_object_clones":0,"num_object_copies":4498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18648,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2073}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":146199054,"num_objects":35710,"num_object_clones":0,"num_object_copies":71420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270595,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13981,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":503582720,"data_stored":503505929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160830,"ondisk_log_size":160830,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018996,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534728,"kb_used_data":210240,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93837112,"statfs":{"total":96636764160,"available":96089202688,"internally_reserved":0,"allocated":215285760,"data_stored":215007114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":8.8119999999999994},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.82599999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691940,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533336,"kb_used_data":110416,"kb_used_omap":4,"kb_used_meta":422907,"kb_avail":93838504,"statfs":{"total":96636764160,"available":96090628096,"internally_reserved":0,"allocated":113065984,"data_stored":112729530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":433057450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.16400000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":5.0499999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.2489999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215223,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607348,"kb_used_data":174636,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93764492,"statfs":{"total":96636764160,"available":96014839808,"internally_reserved":0,"allocated":178827264,"data_stored":178556304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":443084456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":5.0049999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":4.3200000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":5.0439999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":177905664,"data_stored":177902051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111173632,"data_stored":111108139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":214503424,"data_stored":214495739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:10.204 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:32:10.204 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'18326", 'reported_seq': 18583, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:07.316333+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:07.316333+0000', 'last_peered': '2024-09-16T10:32:07.316333+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:07.316333+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "42'8300", 'ondisk_log_start': "42'8300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9572376, 'num_objects': 2338, 'num_object_clones': 0, 'num_object_copies': 4676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2338, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14383, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1965}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:10.204 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:32:10.204 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'18135", 'reported_seq': 18471, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:08.434005+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:32:08.434005+0000', 'last_peered': '2024-09-16T10:32:08.434005+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:32:08.434005+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'8100", 'ondisk_log_start': "42'8100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9158656, 'num_objects': 2236, 'num_object_clones': 0, 'num_object_copies': 4472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14760, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1898}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'18738", 'reported_seq': 19150, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:08.401332+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:32:08.401332+0000', 'last_peered': '2024-09-16T10:32:08.401332+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:32:08.401332+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'8700", 'ondisk_log_start': "44'8700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8937472, 'num_objects': 2182, 'num_object_clones': 0, 'num_object_copies': 4364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14980, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'18768", 'reported_seq': 19046, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:08.347849+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:32:08.347849+0000', 'last_peered': '2024-09-16T10:32:08.347849+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:32:08.347849+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'8700", 'ondisk_log_start': "44'8700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9307078, 'num_objects': 2274, 'num_object_clones': 0, 'num_object_copies': 4548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18766, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2063}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'18143", 'reported_seq': 18343, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:07.330460+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:07.330460+0000', 'last_peered': '2024-09-16T10:32:07.330460+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:07.330460+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'8100", 'ondisk_log_start': "42'8100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9323462, 'num_objects': 2278, 'num_object_clones': 0, 'num_object_copies': 4556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18141, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1955}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'17889", 'reported_seq': 18143, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:08.336425+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:32:08.336425+0000', 'last_peered': '2024-09-16T10:32:08.336425+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:32:08.336425+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'7800", 'ondisk_log_start': "42'7800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8786403, 'num_objects': 2146, 'num_object_clones': 0, 'num_object_copies': 4292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17888, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1929}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'18626", 'reported_seq': 18905, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:08.395636+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:32:08.395636+0000', 'last_peered': '2024-09-16T10:32:08.395636+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:32:08.395636+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'8600", 'ondisk_log_start': "44'8600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9373097, 'num_objects': 2291, 'num_object_clones': 0, 'num_object_copies': 4582, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18623, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1976}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:10.205 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:32:10.206 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'17970", 'reported_seq': 19612, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:08.444237+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:32:08.444237+0000', 'last_peered': '2024-09-16T10:32:08.444237+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:32:08.444237+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'7900", 'ondisk_log_start': "42'7900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8889286, 'num_objects': 2172, 'num_object_clones': 0, 'num_object_copies': 4344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3391, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17968, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 780, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1219}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:10.206 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:32:10.206 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'18651", 'reported_seq': 18896, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:08.439945+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:32:08.439945+0000', 'last_peered': '2024-09-16T10:32:08.439945+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:32:08.439945+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'8600", 'ondisk_log_start': "44'8600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9197452, 'num_objects': 2249, 'num_object_clones': 0, 'num_object_copies': 4498, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18648, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2073}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:10.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:32:10.829 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:10.829 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37365/71424 objects degraded (52.314%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":619,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":35712,"data_bytes":146199073,"bytes_used":1715621888,"bytes_avail":288194670592,"bytes_total":289910292480,"degraded_objects":37365,"degraded_total":71424,"degraded_ratio":0.52314348118279574,"recovering_objects_per_sec":20,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":249,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":583},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:10.858 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:10.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:11.421 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:11.447 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-16T10:32:10.737299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":146199073,"num_objects":35712,"num_object_clones":0,"num_object_copies":71424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35712,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270597,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13982,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160833,"ondisk_log_size":160833,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1675412,"kb_used_data":495292,"kb_used_omap":12,"kb_used_meta":1180083,"kb_avail":281440108,"statfs":{"total":289910292480,"available":288194670592,"internally_reserved":0,"allocated":507179008,"data_stored":506292948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1208404994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,10,0,17,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19025920,"num_objects":-4645,"num_object_clones":0,"num_object_copies":-9290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4645,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002794"},"pg_stats":[{"pgid":"2.f","version":"48'18299","reported_seq":26766,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:08.416622+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:08.416622+0000","last_peered":"2024-09-16T10:32:08.416622+0000","last_clean":"2024-09-16T10:32:08.416622+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:08.416622+0000","last_undegraded":"2024-09-16T10:32:08.416622+0000","last_fullsized":"2024-09-16T10:32:08.416622+0000","mapping_epoch":45,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8884224,"num_objects":2169,"num_object_clones":0,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18299,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18212","reported_seq":18375,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.353497+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:07.353497+0000","last_peered":"2024-09-16T10:32:07.353497+0000","last_clean":"2024-09-16T10:32:07.353497+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:07.353497+0000","last_undegraded":"2024-09-16T10:32:07.353497+0000","last_fullsized":"2024-09-16T10:32:07.353497+0000","mapping_epoch":34,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9216000,"num_objects":2250,"num_object_clones":0,"num_object_copies":4500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14425,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18326","reported_seq":18583,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:07.316333+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:07.316333+0000","last_peered":"2024-09-16T10:32:07.316333+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:07.316333+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9572376,"num_objects":2338,"num_object_clones":0,"num_object_copies":4676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14383,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1965}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18243","reported_seq":18304,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.364288+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:07.364288+0000","last_peered":"2024-09-16T10:32:07.364288+0000","last_clean":"2024-09-16T10:32:07.364288+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:07.364288+0000","last_undegraded":"2024-09-16T10:32:07.364288+0000","last_fullsized":"2024-09-16T10:32:07.364288+0000","mapping_epoch":34,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9089024,"num_objects":2219,"num_object_clones":0,"num_object_copies":4438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18243,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18301","reported_seq":18464,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:08.418883+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:08.418883+0000","last_peered":"2024-09-16T10:32:08.418883+0000","last_clean":"2024-09-16T10:32:08.418883+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:08.418883+0000","last_undegraded":"2024-09-16T10:32:08.418883+0000","last_fullsized":"2024-09-16T10:32:08.418883+0000","mapping_epoch":33,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9027584,"num_objects":2204,"num_object_clones":0,"num_object_copies":4408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14943,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18135","reported_seq":18471,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.434005+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:08.434005+0000","last_peered":"2024-09-16T10:32:08.434005+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:08.434005+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9158656,"num_objects":2236,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14760,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18738","reported_seq":19150,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.401332+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:08.401332+0000","last_peered":"2024-09-16T10:32:08.401332+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:08.401332+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8937472,"num_objects":2182,"num_object_clones":0,"num_object_copies":4364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14980,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2105}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18271","reported_seq":18451,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:08.435408+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:08.435408+0000","last_peered":"2024-09-16T10:32:08.435408+0000","last_clean":"2024-09-16T10:32:08.435408+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:08.435408+0000","last_undegraded":"2024-09-16T10:32:08.435408+0000","last_fullsized":"2024-09-16T10:32:08.435408+0000","mapping_epoch":33,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8679424,"num_objects":2119,"num_object_clones":0,"num_object_copies":4238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14074,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18167","reported_seq":26676,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.459304+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:07.459304+0000","last_peered":"2024-09-16T10:32:07.459304+0000","last_clean":"2024-09-16T10:32:07.459304+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:07.459304+0000","last_undegraded":"2024-09-16T10:32:07.459304+0000","last_fullsized":"2024-09-16T10:32:07.459304+0000","mapping_epoch":43,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9397190,"num_objects":2296,"num_object_clones":0,"num_object_copies":4592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18165,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18768","reported_seq":19046,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.347849+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:08.347849+0000","last_peered":"2024-09-16T10:32:08.347849+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:08.347849+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9307078,"num_objects":2274,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18766,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2063}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18143","reported_seq":18343,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:07.330460+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:07.330460+0000","last_peered":"2024-09-16T10:32:07.330460+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:07.330460+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9323462,"num_objects":2278,"num_object_clones":0,"num_object_copies":4556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18141,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18291","reported_seq":26778,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:07.373963+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:07.373963+0000","last_peered":"2024-09-16T10:32:07.373963+0000","last_clean":"2024-09-16T10:32:07.373963+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:07.373963+0000","last_undegraded":"2024-09-16T10:32:07.373963+0000","last_fullsized":"2024-09-16T10:32:07.373963+0000","mapping_epoch":47,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9360326,"num_objects":2287,"num_object_clones":0,"num_object_copies":4574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18289,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'17889","reported_seq":18143,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.336425+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:08.336425+0000","last_peered":"2024-09-16T10:32:08.336425+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:08.336425+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'7800","ondisk_log_start":"42'7800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8786403,"num_objects":2146,"num_object_clones":0,"num_object_copies":4292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17888,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1929}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18626","reported_seq":18905,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.395636+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:08.395636+0000","last_peered":"2024-09-16T10:32:08.395636+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:08.395636+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9373097,"num_objects":2291,"num_object_clones":0,"num_object_copies":4582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18623,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1976}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'17970","reported_seq":19612,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.444237+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:08.444237+0000","last_peered":"2024-09-16T10:32:08.444237+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:08.444237+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'7900","ondisk_log_start":"42'7900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8889286,"num_objects":2172,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17968,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18651","reported_seq":18896,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:08.439945+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:08.439945+0000","last_peered":"2024-09-16T10:32:08.439945+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:08.439945+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9197452,"num_objects":2249,"num_object_clones":0,"num_object_copies":4498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18648,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2073}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":146199054,"num_objects":35710,"num_object_clones":0,"num_object_copies":71420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":270595,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13981,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":503582720,"data_stored":503505929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160830,"ondisk_log_size":160830,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018996,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534728,"kb_used_data":210240,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93837112,"statfs":{"total":96636764160,"available":96089202688,"internally_reserved":0,"allocated":215285760,"data_stored":215007114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.26100000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":8.8119999999999994},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.82599999999999996}]}]},{"osd":1,"up_from":40,"seq":171798691940,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533336,"kb_used_data":110416,"kb_used_omap":4,"kb_used_meta":422907,"kb_avail":93838504,"statfs":{"total":96636764160,"available":96090628096,"internally_reserved":0,"allocated":113065984,"data_stored":112729530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":433057450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.16400000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":5.0499999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":4.2489999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215223,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":607348,"kb_used_data":174636,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93764492,"statfs":{"total":96636764160,"available":96014839808,"internally_reserved":0,"allocated":178827264,"data_stored":178556304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":443084456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,9,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:31:09 2024","interfaces":[{"interface":"back","average":{"1min":2.601,"5min":2.6880000000000002,"15min":2.7559999999999998},"min":{"1min":0.18099999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7690000000000001,"5min":13.795999999999999,"15min":13.795999999999999},"last":5.0049999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":2.6520000000000001,"15min":2.2200000000000002},"min":{"1min":0.24099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.301,"5min":14.301,"15min":14.301},"last":4.3200000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:31:11 2024","interfaces":[{"interface":"back","average":{"1min":2.8039999999999998,"5min":2.8100000000000001,"15min":1.595},"min":{"1min":0.27500000000000002,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":7.6950000000000003,"5min":10.919,"15min":18.093},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.6339999999999999,"5min":2.359,"15min":1.756},"min":{"1min":0.254,"5min":0.214,"15min":0.192},"max":{"1min":10.882,"5min":13.752000000000001,"15min":13.752000000000001},"last":5.0439999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":177905664,"data_stored":177902051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111173632,"data_stored":111108139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":214503424,"data_stored":214495739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:14.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:14.993 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:14.995 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:15.018 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-16T10:32:14.738205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":137040417,"num_objects":33476,"num_object_clones":0,"num_object_copies":66952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272833,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14075,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160869,"ondisk_log_size":160869,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1671096,"kb_used_data":477020,"kb_used_omap":12,"kb_used_meta":1194035,"kb_avail":281444424,"statfs":{"total":289910292480,"available":288199090176,"internally_reserved":0,"allocated":488468480,"data_stored":487582420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1222691842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24698880,"num_objects":-6030,"num_object_clones":0,"num_object_copies":-12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6030,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002789"},"pg_stats":[{"pgid":"2.f","version":"48'18428","reported_seq":26896,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:13.434932+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:13.434932+0000","last_peered":"2024-09-16T10:32:13.434932+0000","last_clean":"2024-09-16T10:32:13.434932+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:13.434932+0000","last_undegraded":"2024-09-16T10:32:13.434932+0000","last_fullsized":"2024-09-16T10:32:13.434932+0000","mapping_epoch":45,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8355840,"num_objects":2040,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18428,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18364","reported_seq":18527,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.353960+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:12.353960+0000","last_peered":"2024-09-16T10:32:12.353960+0000","last_clean":"2024-09-16T10:32:12.353960+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:12.353960+0000","last_undegraded":"2024-09-16T10:32:12.353960+0000","last_fullsized":"2024-09-16T10:32:12.353960+0000","mapping_epoch":34,"log_start":"43'8300","ondisk_log_start":"43'8300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8593408,"num_objects":2098,"num_object_clones":0,"num_object_copies":4196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14577,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18485","reported_seq":18742,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:12.195042+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:12.195042+0000","last_peered":"2024-09-16T10:32:12.195042+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:12.195042+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8921112,"num_objects":2179,"num_object_clones":0,"num_object_copies":4358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14542,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18388","reported_seq":18451,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.382887+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:12.382887+0000","last_peered":"2024-09-16T10:32:12.382887+0000","last_clean":"2024-09-16T10:32:12.382887+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:12.382887+0000","last_undegraded":"2024-09-16T10:32:12.382887+0000","last_fullsized":"2024-09-16T10:32:12.382887+0000","mapping_epoch":34,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8495104,"num_objects":2074,"num_object_clones":0,"num_object_copies":4148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18388,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18443","reported_seq":18606,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:13.482137+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:13.482137+0000","last_peered":"2024-09-16T10:32:13.482137+0000","last_clean":"2024-09-16T10:32:13.482137+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:13.482137+0000","last_undegraded":"2024-09-16T10:32:13.482137+0000","last_fullsized":"2024-09-16T10:32:13.482137+0000","mapping_epoch":33,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8445952,"num_objects":2062,"num_object_clones":0,"num_object_copies":4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15085,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18259","reported_seq":18594,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.465858+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:13.465858+0000","last_peered":"2024-09-16T10:32:13.465858+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:13.465858+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":2112,"num_object_clones":0,"num_object_copies":4224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14884,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18890","reported_seq":19301,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.401169+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:13.401169+0000","last_peered":"2024-09-16T10:32:13.401169+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:13.401169+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8314880,"num_objects":2030,"num_object_clones":0,"num_object_copies":4060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15132,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18413","reported_seq":18594,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:13.381961+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:13.381961+0000","last_peered":"2024-09-16T10:32:13.381961+0000","last_clean":"2024-09-16T10:32:13.381961+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:13.381961+0000","last_undegraded":"2024-09-16T10:32:13.381961+0000","last_fullsized":"2024-09-16T10:32:13.381961+0000","mapping_epoch":33,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8097792,"num_objects":1977,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14216,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18314","reported_seq":26823,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.482604+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:12.482604+0000","last_peered":"2024-09-16T10:32:12.482604+0000","last_clean":"2024-09-16T10:32:12.482604+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:12.482604+0000","last_undegraded":"2024-09-16T10:32:12.482604+0000","last_fullsized":"2024-09-16T10:32:12.482604+0000","mapping_epoch":43,"log_start":"44'8300","ondisk_log_start":"44'8300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8795078,"num_objects":2149,"num_object_clones":0,"num_object_copies":4298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18312,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18905","reported_seq":19184,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.464758+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:13.464758+0000","last_peered":"2024-09-16T10:32:13.464758+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:13.464758+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8745926,"num_objects":2137,"num_object_clones":0,"num_object_copies":4274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18903,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2200}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18274","reported_seq":18474,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:12.365488+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:12.365488+0000","last_peered":"2024-09-16T10:32:12.365488+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:12.365488+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8786886,"num_objects":2147,"num_object_clones":0,"num_object_copies":4294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18272,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2086}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18425","reported_seq":26913,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.352768+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:12.352768+0000","last_peered":"2024-09-16T10:32:12.352768+0000","last_clean":"2024-09-16T10:32:12.352768+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:12.352768+0000","last_undegraded":"2024-09-16T10:32:12.352768+0000","last_fullsized":"2024-09-16T10:32:12.352768+0000","mapping_epoch":47,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8811462,"num_objects":2153,"num_object_clones":0,"num_object_copies":4306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18423,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18028","reported_seq":18282,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.398446+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:13.398446+0000","last_peered":"2024-09-16T10:32:13.398446+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:13.398446+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8217059,"num_objects":2007,"num_object_clones":0,"num_object_copies":4014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18027,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18777","reported_seq":19056,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.453534+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:13.453534+0000","last_peered":"2024-09-16T10:32:13.453534+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:13.453534+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8754601,"num_objects":2140,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18774,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2127}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18098","reported_seq":19928,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.506931+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:13.506931+0000","last_peered":"2024-09-16T10:32:13.506931+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:13.506931+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8364998,"num_objects":2044,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18096,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18775","reported_seq":19020,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.450891+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:13.450891+0000","last_peered":"2024-09-16T10:32:13.450891+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:13.450891+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689548,"num_objects":2125,"num_object_clones":0,"num_object_copies":4250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18772,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2197}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":137040398,"num_objects":33474,"num_object_clones":0,"num_object_copies":66948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272831,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14074,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":485076992,"data_stored":485000201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018997,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534164,"kb_used_data":205580,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93837676,"statfs":{"total":96636764160,"available":96089780224,"internally_reserved":0,"allocated":210513920,"data_stored":210235274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":3.145},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.24199999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":5.5110000000000001},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.6930000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691941,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532380,"kb_used_data":103700,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93839460,"statfs":{"total":96636764160,"available":96091607040,"internally_reserved":0,"allocated":106188800,"data_stored":105852346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":438955690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.17699999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.25700000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215224,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604552,"kb_used_data":167740,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93767288,"statfs":{"total":96636764160,"available":96017702912,"internally_reserved":0,"allocated":171765760,"data_stored":171494800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":6.6340000000000003},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":2.4260000000000002},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":6.6929999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":170999808,"data_stored":170996195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104325120,"data_stored":104259627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":209752064,"data_stored":209744379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:15.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:15.601 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:15.603 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:15.639 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-16T10:32:14.738205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":137040417,"num_objects":33476,"num_object_clones":0,"num_object_copies":66952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272833,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14075,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160869,"ondisk_log_size":160869,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1671096,"kb_used_data":477020,"kb_used_omap":12,"kb_used_meta":1194035,"kb_avail":281444424,"statfs":{"total":289910292480,"available":288199090176,"internally_reserved":0,"allocated":488468480,"data_stored":487582420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1222691842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24698880,"num_objects":-6030,"num_object_clones":0,"num_object_copies":-12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6030,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002789"},"pg_stats":[{"pgid":"2.f","version":"48'18428","reported_seq":26896,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:13.434932+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:13.434932+0000","last_peered":"2024-09-16T10:32:13.434932+0000","last_clean":"2024-09-16T10:32:13.434932+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:13.434932+0000","last_undegraded":"2024-09-16T10:32:13.434932+0000","last_fullsized":"2024-09-16T10:32:13.434932+0000","mapping_epoch":45,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8355840,"num_objects":2040,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18428,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18364","reported_seq":18527,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.353960+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:12.353960+0000","last_peered":"2024-09-16T10:32:12.353960+0000","last_clean":"2024-09-16T10:32:12.353960+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:12.353960+0000","last_undegraded":"2024-09-16T10:32:12.353960+0000","last_fullsized":"2024-09-16T10:32:12.353960+0000","mapping_epoch":34,"log_start":"43'8300","ondisk_log_start":"43'8300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8593408,"num_objects":2098,"num_object_clones":0,"num_object_copies":4196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14577,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18485","reported_seq":18742,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:12.195042+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:12.195042+0000","last_peered":"2024-09-16T10:32:12.195042+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:12.195042+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8921112,"num_objects":2179,"num_object_clones":0,"num_object_copies":4358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14542,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18388","reported_seq":18451,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.382887+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:12.382887+0000","last_peered":"2024-09-16T10:32:12.382887+0000","last_clean":"2024-09-16T10:32:12.382887+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:12.382887+0000","last_undegraded":"2024-09-16T10:32:12.382887+0000","last_fullsized":"2024-09-16T10:32:12.382887+0000","mapping_epoch":34,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8495104,"num_objects":2074,"num_object_clones":0,"num_object_copies":4148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18388,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18443","reported_seq":18606,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:13.482137+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:13.482137+0000","last_peered":"2024-09-16T10:32:13.482137+0000","last_clean":"2024-09-16T10:32:13.482137+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:13.482137+0000","last_undegraded":"2024-09-16T10:32:13.482137+0000","last_fullsized":"2024-09-16T10:32:13.482137+0000","mapping_epoch":33,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8445952,"num_objects":2062,"num_object_clones":0,"num_object_copies":4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15085,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18259","reported_seq":18594,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.465858+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:13.465858+0000","last_peered":"2024-09-16T10:32:13.465858+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:13.465858+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":2112,"num_object_clones":0,"num_object_copies":4224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14884,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18890","reported_seq":19301,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.401169+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:13.401169+0000","last_peered":"2024-09-16T10:32:13.401169+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:13.401169+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8314880,"num_objects":2030,"num_object_clones":0,"num_object_copies":4060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15132,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18413","reported_seq":18594,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:13.381961+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:13.381961+0000","last_peered":"2024-09-16T10:32:13.381961+0000","last_clean":"2024-09-16T10:32:13.381961+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:13.381961+0000","last_undegraded":"2024-09-16T10:32:13.381961+0000","last_fullsized":"2024-09-16T10:32:13.381961+0000","mapping_epoch":33,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8097792,"num_objects":1977,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14216,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18314","reported_seq":26823,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.482604+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:12.482604+0000","last_peered":"2024-09-16T10:32:12.482604+0000","last_clean":"2024-09-16T10:32:12.482604+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:12.482604+0000","last_undegraded":"2024-09-16T10:32:12.482604+0000","last_fullsized":"2024-09-16T10:32:12.482604+0000","mapping_epoch":43,"log_start":"44'8300","ondisk_log_start":"44'8300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8795078,"num_objects":2149,"num_object_clones":0,"num_object_copies":4298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18312,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18905","reported_seq":19184,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.464758+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:13.464758+0000","last_peered":"2024-09-16T10:32:13.464758+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:13.464758+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8745926,"num_objects":2137,"num_object_clones":0,"num_object_copies":4274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18903,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2200}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18274","reported_seq":18474,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:12.365488+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:12.365488+0000","last_peered":"2024-09-16T10:32:12.365488+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:12.365488+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8786886,"num_objects":2147,"num_object_clones":0,"num_object_copies":4294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18272,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2086}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18425","reported_seq":26913,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.352768+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:12.352768+0000","last_peered":"2024-09-16T10:32:12.352768+0000","last_clean":"2024-09-16T10:32:12.352768+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:12.352768+0000","last_undegraded":"2024-09-16T10:32:12.352768+0000","last_fullsized":"2024-09-16T10:32:12.352768+0000","mapping_epoch":47,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8811462,"num_objects":2153,"num_object_clones":0,"num_object_copies":4306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18423,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18028","reported_seq":18282,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.398446+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:13.398446+0000","last_peered":"2024-09-16T10:32:13.398446+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:13.398446+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8217059,"num_objects":2007,"num_object_clones":0,"num_object_copies":4014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18027,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18777","reported_seq":19056,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.453534+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:13.453534+0000","last_peered":"2024-09-16T10:32:13.453534+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:13.453534+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8754601,"num_objects":2140,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18774,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2127}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18098","reported_seq":19928,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.506931+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:13.506931+0000","last_peered":"2024-09-16T10:32:13.506931+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:13.506931+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8364998,"num_objects":2044,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18096,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18775","reported_seq":19020,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.450891+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:13.450891+0000","last_peered":"2024-09-16T10:32:13.450891+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:13.450891+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689548,"num_objects":2125,"num_object_clones":0,"num_object_copies":4250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18772,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2197}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":137040398,"num_objects":33474,"num_object_clones":0,"num_object_copies":66948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272831,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14074,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":485076992,"data_stored":485000201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018997,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534164,"kb_used_data":205580,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93837676,"statfs":{"total":96636764160,"available":96089780224,"internally_reserved":0,"allocated":210513920,"data_stored":210235274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":3.145},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.24199999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":5.5110000000000001},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.6930000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691941,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532380,"kb_used_data":103700,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93839460,"statfs":{"total":96636764160,"available":96091607040,"internally_reserved":0,"allocated":106188800,"data_stored":105852346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":438955690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.17699999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.25700000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215224,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604552,"kb_used_data":167740,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93767288,"statfs":{"total":96636764160,"available":96017702912,"internally_reserved":0,"allocated":171765760,"data_stored":171494800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":6.6340000000000003},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":2.4260000000000002},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":6.6929999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":170999808,"data_stored":170996195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104325120,"data_stored":104259627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":209752064,"data_stored":209744379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:15.641 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:32:15.641 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'18485", 'reported_seq': 18742, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:12.195042+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:12.195042+0000', 'last_peered': '2024-09-16T10:32:12.195042+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:12.195042+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'8400", 'ondisk_log_start': "44'8400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8921112, 'num_objects': 2179, 'num_object_clones': 0, 'num_object_copies': 4358, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2179, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14542, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:15.641 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:32:15.641 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'18259", 'reported_seq': 18594, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:13.465858+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:32:13.465858+0000', 'last_peered': '2024-09-16T10:32:13.465858+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:32:13.465858+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "42'8200", 'ondisk_log_start': "42'8200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8650752, 'num_objects': 2112, 'num_object_clones': 0, 'num_object_copies': 4224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 14884, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2022}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:15.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:32:15.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'18890", 'reported_seq': 19301, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:13.401169+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:32:13.401169+0000', 'last_peered': '2024-09-16T10:32:13.401169+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:32:13.401169+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'8800", 'ondisk_log_start': "44'8800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8314880, 'num_objects': 2030, 'num_object_clones': 0, 'num_object_copies': 4060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15132, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2257}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:15.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:32:15.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'18905", 'reported_seq': 19184, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:13.464758+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:32:13.464758+0000', 'last_peered': '2024-09-16T10:32:13.464758+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:32:13.464758+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'8900", 'ondisk_log_start': "44'8900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8745926, 'num_objects': 2137, 'num_object_clones': 0, 'num_object_copies': 4274, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18903, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2200}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:15.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:32:15.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'18274", 'reported_seq': 18474, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:12.365488+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:12.365488+0000', 'last_peered': '2024-09-16T10:32:12.365488+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:12.365488+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "42'8200", 'ondisk_log_start': "42'8200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8786886, 'num_objects': 2147, 'num_object_clones': 0, 'num_object_copies': 4294, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2147, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18272, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2086}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:15.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:32:15.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'18028", 'reported_seq': 18282, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:13.398446+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:32:13.398446+0000', 'last_peered': '2024-09-16T10:32:13.398446+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:32:13.398446+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'8000", 'ondisk_log_start': "42'8000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8217059, 'num_objects': 2007, 'num_object_clones': 0, 'num_object_copies': 4014, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18027, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2068}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:15.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:32:15.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'18777", 'reported_seq': 19056, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:13.453534+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:32:13.453534+0000', 'last_peered': '2024-09-16T10:32:13.453534+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:32:13.453534+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'8700", 'ondisk_log_start': "44'8700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8754601, 'num_objects': 2140, 'num_object_clones': 0, 'num_object_copies': 4280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18774, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2127}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:15.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:32:15.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'18098", 'reported_seq': 19928, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:13.506931+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:32:13.506931+0000', 'last_peered': '2024-09-16T10:32:13.506931+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:32:13.506931+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "42'8000", 'ondisk_log_start': "42'8000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8364998, 'num_objects': 2044, 'num_object_clones': 0, 'num_object_copies': 4088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3298, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18096, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 873, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1254}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:15.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:32:15.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'18775", 'reported_seq': 19020, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:13.450891+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:32:13.450891+0000', 'last_peered': '2024-09-16T10:32:13.450891+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:32:13.450891+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'8700", 'ondisk_log_start': "44'8700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8689548, 'num_objects': 2125, 'num_object_clones': 0, 'num_object_copies': 4250, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18772, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2197}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:15.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:32:16.363 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:16.363 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37272/66952 objects degraded (55.670%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":625,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":33476,"data_bytes":137040417,"bytes_used":1711202304,"bytes_avail":288199090176,"bytes_total":289910292480,"degraded_objects":37272,"degraded_total":66952,"degraded_ratio":0.55669733540446886,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":502},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:16.382 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:16.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:16.931 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:16.933 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:16.966 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-16T10:32:16.738676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":137040417,"num_objects":33476,"num_object_clones":0,"num_object_copies":66952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33476,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272833,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14075,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160869,"ondisk_log_size":160869,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1671096,"kb_used_data":477020,"kb_used_omap":12,"kb_used_meta":1194035,"kb_avail":281444424,"statfs":{"total":289910292480,"available":288199090176,"internally_reserved":0,"allocated":488468480,"data_stored":487582420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1222691842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,3,6,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18649088,"num_objects":-4553,"num_object_clones":0,"num_object_copies":-9106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002764"},"pg_stats":[{"pgid":"2.f","version":"48'18428","reported_seq":26896,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:13.434932+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:13.434932+0000","last_peered":"2024-09-16T10:32:13.434932+0000","last_clean":"2024-09-16T10:32:13.434932+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:13.434932+0000","last_undegraded":"2024-09-16T10:32:13.434932+0000","last_fullsized":"2024-09-16T10:32:13.434932+0000","mapping_epoch":45,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8355840,"num_objects":2040,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18428,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18364","reported_seq":18527,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.353960+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:12.353960+0000","last_peered":"2024-09-16T10:32:12.353960+0000","last_clean":"2024-09-16T10:32:12.353960+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:12.353960+0000","last_undegraded":"2024-09-16T10:32:12.353960+0000","last_fullsized":"2024-09-16T10:32:12.353960+0000","mapping_epoch":34,"log_start":"43'8300","ondisk_log_start":"43'8300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8593408,"num_objects":2098,"num_object_clones":0,"num_object_copies":4196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14577,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18485","reported_seq":18742,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:12.195042+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:12.195042+0000","last_peered":"2024-09-16T10:32:12.195042+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:12.195042+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8921112,"num_objects":2179,"num_object_clones":0,"num_object_copies":4358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14542,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2124}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18388","reported_seq":18451,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.382887+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:12.382887+0000","last_peered":"2024-09-16T10:32:12.382887+0000","last_clean":"2024-09-16T10:32:12.382887+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:12.382887+0000","last_undegraded":"2024-09-16T10:32:12.382887+0000","last_fullsized":"2024-09-16T10:32:12.382887+0000","mapping_epoch":34,"log_start":"42'8300","ondisk_log_start":"42'8300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8495104,"num_objects":2074,"num_object_clones":0,"num_object_copies":4148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18388,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18443","reported_seq":18606,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:13.482137+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:13.482137+0000","last_peered":"2024-09-16T10:32:13.482137+0000","last_clean":"2024-09-16T10:32:13.482137+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:13.482137+0000","last_undegraded":"2024-09-16T10:32:13.482137+0000","last_fullsized":"2024-09-16T10:32:13.482137+0000","mapping_epoch":33,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8445952,"num_objects":2062,"num_object_clones":0,"num_object_copies":4124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15085,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18259","reported_seq":18594,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.465858+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:13.465858+0000","last_peered":"2024-09-16T10:32:13.465858+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:13.465858+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8650752,"num_objects":2112,"num_object_clones":0,"num_object_copies":4224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14884,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2022}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'18890","reported_seq":19301,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.401169+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:13.401169+0000","last_peered":"2024-09-16T10:32:13.401169+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:13.401169+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8314880,"num_objects":2030,"num_object_clones":0,"num_object_copies":4060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15132,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18413","reported_seq":18594,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:13.381961+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:13.381961+0000","last_peered":"2024-09-16T10:32:13.381961+0000","last_clean":"2024-09-16T10:32:13.381961+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:13.381961+0000","last_undegraded":"2024-09-16T10:32:13.381961+0000","last_fullsized":"2024-09-16T10:32:13.381961+0000","mapping_epoch":33,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8097792,"num_objects":1977,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14216,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18314","reported_seq":26823,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.482604+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:12.482604+0000","last_peered":"2024-09-16T10:32:12.482604+0000","last_clean":"2024-09-16T10:32:12.482604+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:12.482604+0000","last_undegraded":"2024-09-16T10:32:12.482604+0000","last_fullsized":"2024-09-16T10:32:12.482604+0000","mapping_epoch":43,"log_start":"44'8300","ondisk_log_start":"44'8300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8795078,"num_objects":2149,"num_object_clones":0,"num_object_copies":4298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18312,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'18905","reported_seq":19184,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.464758+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:13.464758+0000","last_peered":"2024-09-16T10:32:13.464758+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:13.464758+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8745926,"num_objects":2137,"num_object_clones":0,"num_object_copies":4274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18903,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2200}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18274","reported_seq":18474,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:12.365488+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:12.365488+0000","last_peered":"2024-09-16T10:32:12.365488+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:12.365488+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"42'8200","ondisk_log_start":"42'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8786886,"num_objects":2147,"num_object_clones":0,"num_object_copies":4294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18272,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2086}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":74,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.299694+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:28.299694+0000","last_peered":"2024-09-16T10:31:28.299694+0000","last_clean":"2024-09-16T10:31:28.299694+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:28.299694+0000","last_undegraded":"2024-09-16T10:31:28.299694+0000","last_fullsized":"2024-09-16T10:31:28.299694+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18425","reported_seq":26913,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:12.352768+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:12.352768+0000","last_peered":"2024-09-16T10:32:12.352768+0000","last_clean":"2024-09-16T10:32:12.352768+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:12.352768+0000","last_undegraded":"2024-09-16T10:32:12.352768+0000","last_fullsized":"2024-09-16T10:32:12.352768+0000","mapping_epoch":47,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8811462,"num_objects":2153,"num_object_clones":0,"num_object_copies":4306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18423,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18028","reported_seq":18282,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.398446+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:13.398446+0000","last_peered":"2024-09-16T10:32:13.398446+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:13.398446+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8217059,"num_objects":2007,"num_object_clones":0,"num_object_copies":4014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18027,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2068}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294475+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:28.294475+0000","last_peered":"2024-09-16T10:31:28.294475+0000","last_clean":"2024-09-16T10:31:28.294475+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:28.294475+0000","last_undegraded":"2024-09-16T10:31:28.294475+0000","last_fullsized":"2024-09-16T10:31:28.294475+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18777","reported_seq":19056,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.453534+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:13.453534+0000","last_peered":"2024-09-16T10:32:13.453534+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:13.453534+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8754601,"num_objects":2140,"num_object_clones":0,"num_object_copies":4280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18774,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2127}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18098","reported_seq":19928,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.506931+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:13.506931+0000","last_peered":"2024-09-16T10:32:13.506931+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:13.506931+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"42'8000","ondisk_log_start":"42'8000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8364998,"num_objects":2044,"num_object_clones":0,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18096,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":873,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1254}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":60,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.294490+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:28.294490+0000","last_peered":"2024-09-16T10:31:28.294490+0000","last_clean":"2024-09-16T10:31:28.294490+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:28.294490+0000","last_undegraded":"2024-09-16T10:31:28.294490+0000","last_fullsized":"2024-09-16T10:31:28.294490+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18775","reported_seq":19020,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:13.450891+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:13.450891+0000","last_peered":"2024-09-16T10:32:13.450891+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:13.450891+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8689548,"num_objects":2125,"num_object_clones":0,"num_object_copies":4250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18772,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2197}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":137040398,"num_objects":33474,"num_object_clones":0,"num_object_copies":66948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":272831,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14074,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":485076992,"data_stored":485000201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160866,"ondisk_log_size":160866,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018997,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534164,"kb_used_data":205580,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93837676,"statfs":{"total":96636764160,"available":96089780224,"internally_reserved":0,"allocated":210513920,"data_stored":210235274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":3.145},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.24199999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":5.5110000000000001},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.6930000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691941,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532380,"kb_used_data":103700,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93839460,"statfs":{"total":96636764160,"available":96091607040,"internally_reserved":0,"allocated":106188800,"data_stored":105852346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":438955690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":0.17699999999999999},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.25700000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215224,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":604552,"kb_used_data":167740,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93767288,"statfs":{"total":96636764160,"available":96017702912,"internally_reserved":0,"allocated":171765760,"data_stored":171494800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":6.6340000000000003},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":2.4260000000000002},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":6.6929999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":170999808,"data_stored":170996195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":104325120,"data_stored":104259627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":209752064,"data_stored":209744379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:19.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:21.033 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:21.034 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:21.068 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-16T10:32:20.739636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127558177,"num_objects":31161,"num_object_clones":0,"num_object_copies":62322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":275148,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14161,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160584,"ondisk_log_size":160584,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1635036,"kb_used_data":457320,"kb_used_omap":12,"kb_used_meta":1177651,"kb_avail":281480484,"statfs":{"total":289910292480,"available":288236015616,"internally_reserved":0,"allocated":468295680,"data_stored":467409620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1205914626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,9,8,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18640896,"num_objects":-4551,"num_object_clones":0,"num_object_copies":-9102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4551,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002834"},"pg_stats":[{"pgid":"2.f","version":"48'18571","reported_seq":27039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:18.559659+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:18.559659+0000","last_peered":"2024-09-16T10:32:18.559659+0000","last_clean":"2024-09-16T10:32:18.559659+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:18.559659+0000","last_undegraded":"2024-09-16T10:32:18.559659+0000","last_fullsized":"2024-09-16T10:32:18.559659+0000","mapping_epoch":45,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7770112,"num_objects":1897,"num_object_clones":0,"num_object_copies":3794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18571,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18504","reported_seq":18667,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:17.333606+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:17.333606+0000","last_peered":"2024-09-16T10:32:17.333606+0000","last_clean":"2024-09-16T10:32:17.333606+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:17.333606+0000","last_undegraded":"2024-09-16T10:32:17.333606+0000","last_fullsized":"2024-09-16T10:32:17.333606+0000","mapping_epoch":34,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8019968,"num_objects":1958,"num_object_clones":0,"num_object_copies":3916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14717,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18642","reported_seq":18898,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:17.440372+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:17.440372+0000","last_peered":"2024-09-16T10:32:17.440372+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:17.440372+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8278040,"num_objects":2022,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14699,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18541","reported_seq":18603,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:17.419705+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:17.419705+0000","last_peered":"2024-09-16T10:32:17.419705+0000","last_clean":"2024-09-16T10:32:17.419705+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:17.419705+0000","last_undegraded":"2024-09-16T10:32:17.419705+0000","last_fullsized":"2024-09-16T10:32:17.419705+0000","mapping_epoch":34,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7868416,"num_objects":1921,"num_object_clones":0,"num_object_copies":3842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18541,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18606","reported_seq":18769,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:18.461532+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:18.461532+0000","last_peered":"2024-09-16T10:32:18.461532+0000","last_clean":"2024-09-16T10:32:18.461532+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:18.461532+0000","last_undegraded":"2024-09-16T10:32:18.461532+0000","last_fullsized":"2024-09-16T10:32:18.461532+0000","mapping_epoch":33,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7778304,"num_objects":1899,"num_object_clones":0,"num_object_copies":3798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15248,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18412","reported_seq":18748,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.457112+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:18.457112+0000","last_peered":"2024-09-16T10:32:18.457112+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:18.457112+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8024064,"num_objects":1959,"num_object_clones":0,"num_object_copies":3918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15037,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19029","reported_seq":19441,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.428236+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:18.428236+0000","last_peered":"2024-09-16T10:32:18.428236+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:18.428236+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7745536,"num_objects":1891,"num_object_clones":0,"num_object_copies":3782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15271,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18554","reported_seq":18735,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:18.538400+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:18.538400+0000","last_peered":"2024-09-16T10:32:18.538400+0000","last_clean":"2024-09-16T10:32:18.538400+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:18.538400+0000","last_undegraded":"2024-09-16T10:32:18.538400+0000","last_fullsized":"2024-09-16T10:32:18.538400+0000","mapping_epoch":33,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":1836,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14357,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18459","reported_seq":26968,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:17.452682+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:17.452682+0000","last_peered":"2024-09-16T10:32:17.452682+0000","last_clean":"2024-09-16T10:32:17.452682+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:17.452682+0000","last_undegraded":"2024-09-16T10:32:17.452682+0000","last_fullsized":"2024-09-16T10:32:17.452682+0000","mapping_epoch":43,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8201158,"num_objects":2004,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18457,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19065","reported_seq":19340,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.538886+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:18.538886+0000","last_peered":"2024-09-16T10:32:18.538886+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:18.538886+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8090566,"num_objects":1977,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19063,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18416","reported_seq":18616,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:17.274340+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:17.274340+0000","last_peered":"2024-09-16T10:32:17.274340+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:17.274340+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8205254,"num_objects":2005,"num_object_clones":0,"num_object_copies":4010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18414,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18561","reported_seq":27049,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:17.336894+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:17.336894+0000","last_peered":"2024-09-16T10:32:17.336894+0000","last_clean":"2024-09-16T10:32:17.336894+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:17.336894+0000","last_undegraded":"2024-09-16T10:32:17.336894+0000","last_fullsized":"2024-09-16T10:32:17.336894+0000","mapping_epoch":47,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8254406,"num_objects":2017,"num_object_clones":0,"num_object_copies":4034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18559,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18158","reported_seq":18412,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.429452+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:18.429452+0000","last_peered":"2024-09-16T10:32:18.429452+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:18.429452+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7684579,"num_objects":1877,"num_object_clones":0,"num_object_copies":3754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18157,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18922","reported_seq":19201,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.509100+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:18.509100+0000","last_peered":"2024-09-16T10:32:18.509100+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:18.509100+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8160681,"num_objects":1995,"num_object_clones":0,"num_object_copies":3990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18919,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18223","reported_seq":20223,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.550278+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:18.550278+0000","last_peered":"2024-09-16T10:32:18.550278+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:18.550278+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8200","ondisk_log_start":"44'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7852998,"num_objects":1919,"num_object_clones":0,"num_object_copies":3838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18221,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18918","reported_seq":19163,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.507110+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:18.507110+0000","last_peered":"2024-09-16T10:32:18.507110+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:18.507110+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8103820,"num_objects":1982,"num_object_clones":0,"num_object_copies":3964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18915,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":127558158,"num_objects":31159,"num_object_clones":0,"num_object_copies":62318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":275146,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14160,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466124800,"data_stored":466048009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160581,"ondisk_log_size":160581,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018998,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517212,"kb_used_data":200916,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93854628,"statfs":{"total":96636764160,"available":96107139072,"internally_reserved":0,"allocated":205737984,"data_stored":205459338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":323874480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":4.1280000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":4.4790000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.224},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":4.0860000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691942,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532400,"kb_used_data":95512,"kb_used_omap":4,"kb_used_meta":436859,"kb_avail":93839440,"statfs":{"total":96636764160,"available":96091586560,"internally_reserved":0,"allocated":97804288,"data_stored":97467834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":447344298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.1930000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.282},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.98499999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215225,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585424,"kb_used_data":160892,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93786416,"statfs":{"total":96636764160,"available":96037289984,"internally_reserved":0,"allocated":164753408,"data_stored":164482448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":434695848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":3.492},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":12.691000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.78400000000000003},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":3.7269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163921920,"data_stored":163918307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":97316864,"data_stored":97251371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204886016,"data_stored":204878331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:21.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:21.601 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:21.602 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:21.632 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-16T10:32:20.739636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127558177,"num_objects":31161,"num_object_clones":0,"num_object_copies":62322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":275148,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14161,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160584,"ondisk_log_size":160584,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1635036,"kb_used_data":457320,"kb_used_omap":12,"kb_used_meta":1177651,"kb_avail":281480484,"statfs":{"total":289910292480,"available":288236015616,"internally_reserved":0,"allocated":468295680,"data_stored":467409620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1205914626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,9,8,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18640896,"num_objects":-4551,"num_object_clones":0,"num_object_copies":-9102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4551,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002834"},"pg_stats":[{"pgid":"2.f","version":"48'18571","reported_seq":27039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:18.559659+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:18.559659+0000","last_peered":"2024-09-16T10:32:18.559659+0000","last_clean":"2024-09-16T10:32:18.559659+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:18.559659+0000","last_undegraded":"2024-09-16T10:32:18.559659+0000","last_fullsized":"2024-09-16T10:32:18.559659+0000","mapping_epoch":45,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7770112,"num_objects":1897,"num_object_clones":0,"num_object_copies":3794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18571,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18504","reported_seq":18667,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:17.333606+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:17.333606+0000","last_peered":"2024-09-16T10:32:17.333606+0000","last_clean":"2024-09-16T10:32:17.333606+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:17.333606+0000","last_undegraded":"2024-09-16T10:32:17.333606+0000","last_fullsized":"2024-09-16T10:32:17.333606+0000","mapping_epoch":34,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8019968,"num_objects":1958,"num_object_clones":0,"num_object_copies":3916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14717,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18642","reported_seq":18898,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:17.440372+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:17.440372+0000","last_peered":"2024-09-16T10:32:17.440372+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:17.440372+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8278040,"num_objects":2022,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14699,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18541","reported_seq":18603,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:17.419705+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:17.419705+0000","last_peered":"2024-09-16T10:32:17.419705+0000","last_clean":"2024-09-16T10:32:17.419705+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:17.419705+0000","last_undegraded":"2024-09-16T10:32:17.419705+0000","last_fullsized":"2024-09-16T10:32:17.419705+0000","mapping_epoch":34,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7868416,"num_objects":1921,"num_object_clones":0,"num_object_copies":3842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18541,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18606","reported_seq":18769,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:18.461532+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:18.461532+0000","last_peered":"2024-09-16T10:32:18.461532+0000","last_clean":"2024-09-16T10:32:18.461532+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:18.461532+0000","last_undegraded":"2024-09-16T10:32:18.461532+0000","last_fullsized":"2024-09-16T10:32:18.461532+0000","mapping_epoch":33,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7778304,"num_objects":1899,"num_object_clones":0,"num_object_copies":3798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15248,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18412","reported_seq":18748,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.457112+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:18.457112+0000","last_peered":"2024-09-16T10:32:18.457112+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:18.457112+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8024064,"num_objects":1959,"num_object_clones":0,"num_object_copies":3918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15037,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19029","reported_seq":19441,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.428236+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:18.428236+0000","last_peered":"2024-09-16T10:32:18.428236+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:18.428236+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7745536,"num_objects":1891,"num_object_clones":0,"num_object_copies":3782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15271,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18554","reported_seq":18735,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:18.538400+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:18.538400+0000","last_peered":"2024-09-16T10:32:18.538400+0000","last_clean":"2024-09-16T10:32:18.538400+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:18.538400+0000","last_undegraded":"2024-09-16T10:32:18.538400+0000","last_fullsized":"2024-09-16T10:32:18.538400+0000","mapping_epoch":33,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":1836,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14357,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18459","reported_seq":26968,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:17.452682+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:17.452682+0000","last_peered":"2024-09-16T10:32:17.452682+0000","last_clean":"2024-09-16T10:32:17.452682+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:17.452682+0000","last_undegraded":"2024-09-16T10:32:17.452682+0000","last_fullsized":"2024-09-16T10:32:17.452682+0000","mapping_epoch":43,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8201158,"num_objects":2004,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18457,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19065","reported_seq":19340,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.538886+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:18.538886+0000","last_peered":"2024-09-16T10:32:18.538886+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:18.538886+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8090566,"num_objects":1977,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19063,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18416","reported_seq":18616,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:17.274340+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:17.274340+0000","last_peered":"2024-09-16T10:32:17.274340+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:17.274340+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8205254,"num_objects":2005,"num_object_clones":0,"num_object_copies":4010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18414,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18561","reported_seq":27049,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:17.336894+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:17.336894+0000","last_peered":"2024-09-16T10:32:17.336894+0000","last_clean":"2024-09-16T10:32:17.336894+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:17.336894+0000","last_undegraded":"2024-09-16T10:32:17.336894+0000","last_fullsized":"2024-09-16T10:32:17.336894+0000","mapping_epoch":47,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8254406,"num_objects":2017,"num_object_clones":0,"num_object_copies":4034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18559,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18158","reported_seq":18412,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.429452+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:18.429452+0000","last_peered":"2024-09-16T10:32:18.429452+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:18.429452+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7684579,"num_objects":1877,"num_object_clones":0,"num_object_copies":3754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18157,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18922","reported_seq":19201,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.509100+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:18.509100+0000","last_peered":"2024-09-16T10:32:18.509100+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:18.509100+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8160681,"num_objects":1995,"num_object_clones":0,"num_object_copies":3990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18919,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18223","reported_seq":20223,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.550278+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:18.550278+0000","last_peered":"2024-09-16T10:32:18.550278+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:18.550278+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8200","ondisk_log_start":"44'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7852998,"num_objects":1919,"num_object_clones":0,"num_object_copies":3838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18221,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18918","reported_seq":19163,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.507110+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:18.507110+0000","last_peered":"2024-09-16T10:32:18.507110+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:18.507110+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8103820,"num_objects":1982,"num_object_clones":0,"num_object_copies":3964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18915,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":127558158,"num_objects":31159,"num_object_clones":0,"num_object_copies":62318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":275146,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14160,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466124800,"data_stored":466048009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160581,"ondisk_log_size":160581,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018998,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517212,"kb_used_data":200916,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93854628,"statfs":{"total":96636764160,"available":96107139072,"internally_reserved":0,"allocated":205737984,"data_stored":205459338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":323874480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":4.1280000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":4.4790000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":0.224},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":4.0860000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691942,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532400,"kb_used_data":95512,"kb_used_omap":4,"kb_used_meta":436859,"kb_avail":93839440,"statfs":{"total":96636764160,"available":96091586560,"internally_reserved":0,"allocated":97804288,"data_stored":97467834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":447344298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,6,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":2.806,"5min":3.2029999999999998,"15min":3.4820000000000002},"min":{"1min":0.21299999999999999,"5min":0.184,"15min":0.17799999999999999},"max":{"1min":11.084,"5min":17.675000000000001,"15min":17.675000000000001},"last":2.1930000000000001},{"interface":"front","average":{"1min":2.5699999999999998,"5min":2.8860000000000001,"15min":2.6789999999999998},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.077,"5min":15.837999999999999,"15min":16.850999999999999},"last":0.64800000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:31:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":2.6699999999999999,"15min":2.4590000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":7.8239999999999998,"5min":18.638999999999999,"15min":18.638999999999999},"last":1.282},{"interface":"front","average":{"1min":1.51,"5min":2.8679999999999999,"15min":2.855},"min":{"1min":0.23899999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.9480000000000004,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.98499999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215225,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585424,"kb_used_data":160892,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93786416,"statfs":{"total":96636764160,"available":96037289984,"internally_reserved":0,"allocated":164753408,"data_stored":164482448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":434695848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":3.492},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":12.691000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.78400000000000003},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":3.7269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163921920,"data_stored":163918307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":97316864,"data_stored":97251371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204886016,"data_stored":204878331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:21.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:32:21.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'18642", 'reported_seq': 18898, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:17.440372+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:17.440372+0000', 'last_peered': '2024-09-16T10:32:17.440372+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:17.440372+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'8600", 'ondisk_log_start': "44'8600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8278040, 'num_objects': 2022, 'num_object_clones': 0, 'num_object_copies': 4044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2022, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14699, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:21.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:32:21.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'18412", 'reported_seq': 18748, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:18.457112+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:32:18.457112+0000', 'last_peered': '2024-09-16T10:32:18.457112+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:32:18.457112+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'8400", 'ondisk_log_start': "44'8400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8024064, 'num_objects': 1959, 'num_object_clones': 0, 'num_object_copies': 3918, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1959, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15037, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2175}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:21.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:32:21.634 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'19029", 'reported_seq': 19441, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:18.428236+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:32:18.428236+0000', 'last_peered': '2024-09-16T10:32:18.428236+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:32:18.428236+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'9000", 'ondisk_log_start': "44'9000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7745536, 'num_objects': 1891, 'num_object_clones': 0, 'num_object_copies': 3782, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15271, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:21.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:32:21.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'19065", 'reported_seq': 19340, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:18.538886+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:32:18.538886+0000', 'last_peered': '2024-09-16T10:32:18.538886+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:32:18.538886+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'9000", 'ondisk_log_start': "44'9000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8090566, 'num_objects': 1977, 'num_object_clones': 0, 'num_object_copies': 3954, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19063, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2360}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:21.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:32:21.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'18416", 'reported_seq': 18616, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:17.274340+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:17.274340+0000', 'last_peered': '2024-09-16T10:32:17.274340+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:17.274340+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'8400", 'ondisk_log_start': "44'8400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8205254, 'num_objects': 2005, 'num_object_clones': 0, 'num_object_copies': 4010, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18414, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2228}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:21.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:32:21.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'18158", 'reported_seq': 18412, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:18.429452+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:32:18.429452+0000', 'last_peered': '2024-09-16T10:32:18.429452+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:32:18.429452+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "42'8100", 'ondisk_log_start': "42'8100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7684579, 'num_objects': 1877, 'num_object_clones': 0, 'num_object_copies': 3754, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18157, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:21.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:32:21.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'18922", 'reported_seq': 19201, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:18.509100+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:32:18.509100+0000', 'last_peered': '2024-09-16T10:32:18.509100+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:32:18.509100+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'8900", 'ondisk_log_start': "44'8900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8160681, 'num_objects': 1995, 'num_object_clones': 0, 'num_object_copies': 3990, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18919, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:21.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:32:21.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'18223", 'reported_seq': 20223, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:18.550278+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:32:18.550278+0000', 'last_peered': '2024-09-16T10:32:18.550278+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:32:18.550278+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'8200", 'ondisk_log_start': "44'8200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7852998, 'num_objects': 1919, 'num_object_clones': 0, 'num_object_copies': 3838, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3212, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18221, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 959, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:21.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:32:21.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'18918", 'reported_seq': 19163, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:18.507110+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:32:18.507110+0000', 'last_peered': '2024-09-16T10:32:18.507110+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:32:18.507110+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'8900", 'ondisk_log_start': "44'8900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8103820, 'num_objects': 1982, 'num_object_clones': 0, 'num_object_copies': 3964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18915, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:21.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:32:22.299 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:22.299 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37186/62322 objects degraded (59.668%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":631,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":31161,"data_bytes":127558177,"bytes_used":1674276864,"bytes_avail":288236015616,"bytes_total":289910292480,"degraded_objects":37186,"degraded_total":62322,"degraded_ratio":0.59667533134366679,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":379},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:22.326 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:22.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:22.859 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:22.862 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:22.886 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-09-16T10:32:22.740085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":123818529,"num_objects":30248,"num_object_clones":0,"num_object_copies":60496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30248,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276061,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14161,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160697,"ondisk_log_size":160697,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1615060,"kb_used_data":445536,"kb_used_omap":12,"kb_used_meta":1169459,"kb_avail":281500460,"statfs":{"total":289910292480,"available":288256471040,"internally_reserved":0,"allocated":456228864,"data_stored":455342804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1197526018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,9,7,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22380544,"num_objects":-5464,"num_object_clones":0,"num_object_copies":-10928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002785"},"pg_stats":[{"pgid":"2.f","version":"48'18571","reported_seq":27039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:18.559659+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:18.559659+0000","last_peered":"2024-09-16T10:32:18.559659+0000","last_clean":"2024-09-16T10:32:18.559659+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:18.559659+0000","last_undegraded":"2024-09-16T10:32:18.559659+0000","last_fullsized":"2024-09-16T10:32:18.559659+0000","mapping_epoch":45,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7770112,"num_objects":1897,"num_object_clones":0,"num_object_copies":3794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18571,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18653","reported_seq":18815,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.455567+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:22.455567+0000","last_peered":"2024-09-16T10:32:22.455567+0000","last_clean":"2024-09-16T10:32:22.455567+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:22.455567+0000","last_undegraded":"2024-09-16T10:32:22.455567+0000","last_fullsized":"2024-09-16T10:32:22.455567+0000","mapping_epoch":34,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7409664,"num_objects":1809,"num_object_clones":0,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14866,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18793","reported_seq":19050,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:22.361105+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:22.361105+0000","last_peered":"2024-09-16T10:32:22.361105+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:22.361105+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7659544,"num_objects":1871,"num_object_clones":0,"num_object_copies":3742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14850,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2432}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18704","reported_seq":18767,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.467414+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:22.467414+0000","last_peered":"2024-09-16T10:32:22.467414+0000","last_clean":"2024-09-16T10:32:22.467414+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:22.467414+0000","last_undegraded":"2024-09-16T10:32:22.467414+0000","last_fullsized":"2024-09-16T10:32:22.467414+0000","mapping_epoch":34,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7200768,"num_objects":1758,"num_object_clones":0,"num_object_copies":3516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18704,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18606","reported_seq":18769,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:18.461532+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:18.461532+0000","last_peered":"2024-09-16T10:32:18.461532+0000","last_clean":"2024-09-16T10:32:18.461532+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:18.461532+0000","last_undegraded":"2024-09-16T10:32:18.461532+0000","last_fullsized":"2024-09-16T10:32:18.461532+0000","mapping_epoch":33,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7778304,"num_objects":1899,"num_object_clones":0,"num_object_copies":3798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15248,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18412","reported_seq":18748,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.457112+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:18.457112+0000","last_peered":"2024-09-16T10:32:18.457112+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:18.457112+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8024064,"num_objects":1959,"num_object_clones":0,"num_object_copies":3918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15037,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19029","reported_seq":19441,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.428236+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:18.428236+0000","last_peered":"2024-09-16T10:32:18.428236+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:18.428236+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7745536,"num_objects":1891,"num_object_clones":0,"num_object_copies":3782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15271,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18554","reported_seq":18735,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:18.538400+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:18.538400+0000","last_peered":"2024-09-16T10:32:18.538400+0000","last_clean":"2024-09-16T10:32:18.538400+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:18.538400+0000","last_undegraded":"2024-09-16T10:32:18.538400+0000","last_fullsized":"2024-09-16T10:32:18.538400+0000","mapping_epoch":33,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7520256,"num_objects":1836,"num_object_clones":0,"num_object_copies":3672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14357,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18616","reported_seq":27126,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.622943+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:22.622943+0000","last_peered":"2024-09-16T10:32:22.622943+0000","last_clean":"2024-09-16T10:32:22.622943+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:22.622943+0000","last_undegraded":"2024-09-16T10:32:22.622943+0000","last_fullsized":"2024-09-16T10:32:22.622943+0000","mapping_epoch":43,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7558086,"num_objects":1847,"num_object_clones":0,"num_object_copies":3694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18614,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19065","reported_seq":19340,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.538886+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:18.538886+0000","last_peered":"2024-09-16T10:32:18.538886+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:18.538886+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8090566,"num_objects":1977,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19063,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18579","reported_seq":18779,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:22.453585+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:22.453585+0000","last_peered":"2024-09-16T10:32:22.453585+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:22.453585+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7537606,"num_objects":1842,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18577,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18691","reported_seq":27179,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.413134+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:22.413134+0000","last_peered":"2024-09-16T10:32:22.413134+0000","last_clean":"2024-09-16T10:32:22.413134+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:22.413134+0000","last_undegraded":"2024-09-16T10:32:22.413134+0000","last_fullsized":"2024-09-16T10:32:22.413134+0000","mapping_epoch":47,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7721926,"num_objects":1887,"num_object_clones":0,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18689,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18158","reported_seq":18412,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.429452+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:18.429452+0000","last_peered":"2024-09-16T10:32:18.429452+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:18.429452+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"42'8100","ondisk_log_start":"42'8100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7684579,"num_objects":1877,"num_object_clones":0,"num_object_copies":3754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18157,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'18922","reported_seq":19201,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.509100+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:18.509100+0000","last_peered":"2024-09-16T10:32:18.509100+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:18.509100+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8160681,"num_objects":1995,"num_object_clones":0,"num_object_copies":3990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18919,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18223","reported_seq":20223,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.550278+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:18.550278+0000","last_peered":"2024-09-16T10:32:18.550278+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:18.550278+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8200","ondisk_log_start":"44'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7852998,"num_objects":1919,"num_object_clones":0,"num_object_copies":3838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18221,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":959,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1293}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'18918","reported_seq":19163,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:18.507110+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:18.507110+0000","last_peered":"2024-09-16T10:32:18.507110+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:18.507110+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8103820,"num_objects":1982,"num_object_clones":0,"num_object_copies":3964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18915,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2340}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":123818510,"num_objects":30246,"num_object_clones":0,"num_object_copies":60492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30246,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":276059,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14160,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454164480,"data_stored":454087689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160694,"ondisk_log_size":160694,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018999,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516540,"kb_used_data":196144,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93855300,"statfs":{"total":96636764160,"available":96107827200,"internally_reserved":0,"allocated":200851456,"data_stored":200572810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":11.718999999999999},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":2.4860000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":16.199999999999999},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":11.766999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691943,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513096,"kb_used_data":88500,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93858744,"statfs":{"total":96636764160,"available":96111353856,"internally_reserved":0,"allocated":90624000,"data_stored":90287546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":434761386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":3.5710000000000002},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.28100000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.2880000000000003},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":6.0069999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215225,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585424,"kb_used_data":160892,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93786416,"statfs":{"total":96636764160,"available":96037289984,"internally_reserved":0,"allocated":164753408,"data_stored":164482448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":434695848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":3.492},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":12.691000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.78400000000000003},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":3.7269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":163921920,"data_stored":163918307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90193920,"data_stored":90128427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200048640,"data_stored":200040955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:25.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:26.453 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:26.457 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:26.481 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-16T10:32:24.740555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118047265,"num_objects":28839,"num_object_clones":0,"num_object_copies":57678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277470,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14235,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1616472,"kb_used_data":438764,"kb_used_omap":12,"kb_used_meta":1177651,"kb_avail":281499048,"statfs":{"total":289910292480,"available":288255025152,"internally_reserved":0,"allocated":449294336,"data_stored":448408276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1205914626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,9,13,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24596480,"num_objects":-6005,"num_object_clones":0,"num_object_copies":-12010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6005,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002821"},"pg_stats":[{"pgid":"2.f","version":"48'18719","reported_seq":27187,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:23.572223+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:23.572223+0000","last_peered":"2024-09-16T10:32:23.572223+0000","last_clean":"2024-09-16T10:32:23.572223+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:23.572223+0000","last_undegraded":"2024-09-16T10:32:23.572223+0000","last_fullsized":"2024-09-16T10:32:23.572223+0000","mapping_epoch":45,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7163904,"num_objects":1749,"num_object_clones":0,"num_object_copies":3498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18719,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18653","reported_seq":18815,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.455567+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:22.455567+0000","last_peered":"2024-09-16T10:32:22.455567+0000","last_clean":"2024-09-16T10:32:22.455567+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:22.455567+0000","last_undegraded":"2024-09-16T10:32:22.455567+0000","last_fullsized":"2024-09-16T10:32:22.455567+0000","mapping_epoch":34,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7409664,"num_objects":1809,"num_object_clones":0,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14866,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18793","reported_seq":19050,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:22.361105+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:22.361105+0000","last_peered":"2024-09-16T10:32:22.361105+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:22.361105+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7659544,"num_objects":1871,"num_object_clones":0,"num_object_copies":3742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14850,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2432}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18704","reported_seq":18767,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.467414+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:22.467414+0000","last_peered":"2024-09-16T10:32:22.467414+0000","last_clean":"2024-09-16T10:32:22.467414+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:22.467414+0000","last_undegraded":"2024-09-16T10:32:22.467414+0000","last_fullsized":"2024-09-16T10:32:22.467414+0000","mapping_epoch":34,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7200768,"num_objects":1758,"num_object_clones":0,"num_object_copies":3516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18704,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18735","reported_seq":18898,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:23.589758+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:23.589758+0000","last_peered":"2024-09-16T10:32:23.589758+0000","last_clean":"2024-09-16T10:32:23.589758+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:23.589758+0000","last_undegraded":"2024-09-16T10:32:23.589758+0000","last_fullsized":"2024-09-16T10:32:23.589758+0000","mapping_epoch":33,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7249920,"num_objects":1770,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15377,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18555","reported_seq":18891,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.539657+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:23.539657+0000","last_peered":"2024-09-16T10:32:23.539657+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:23.539657+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7438336,"num_objects":1816,"num_object_clones":0,"num_object_copies":3632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15180,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19155","reported_seq":19567,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.471360+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:23.471360+0000","last_peered":"2024-09-16T10:32:23.471360+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:23.471360+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7229440,"num_objects":1765,"num_object_clones":0,"num_object_copies":3530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15397,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18687","reported_seq":18868,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:23.590015+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:23.590015+0000","last_peered":"2024-09-16T10:32:23.590015+0000","last_clean":"2024-09-16T10:32:23.590015+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:23.590015+0000","last_undegraded":"2024-09-16T10:32:23.590015+0000","last_fullsized":"2024-09-16T10:32:23.590015+0000","mapping_epoch":33,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6975488,"num_objects":1703,"num_object_clones":0,"num_object_copies":3406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14490,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18616","reported_seq":27126,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.622943+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:22.622943+0000","last_peered":"2024-09-16T10:32:22.622943+0000","last_clean":"2024-09-16T10:32:22.622943+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:22.622943+0000","last_undegraded":"2024-09-16T10:32:22.622943+0000","last_fullsized":"2024-09-16T10:32:22.622943+0000","mapping_epoch":43,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7558086,"num_objects":1847,"num_object_clones":0,"num_object_copies":3694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18614,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19211","reported_seq":19490,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.465806+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:23.465806+0000","last_peered":"2024-09-16T10:32:23.465806+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:23.465806+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7492550,"num_objects":1831,"num_object_clones":0,"num_object_copies":3662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19209,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18579","reported_seq":18779,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:22.453585+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:22.453585+0000","last_peered":"2024-09-16T10:32:22.453585+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:22.453585+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7537606,"num_objects":1842,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18577,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18691","reported_seq":27179,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.413134+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:22.413134+0000","last_peered":"2024-09-16T10:32:22.413134+0000","last_clean":"2024-09-16T10:32:22.413134+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:22.413134+0000","last_undegraded":"2024-09-16T10:32:22.413134+0000","last_fullsized":"2024-09-16T10:32:22.413134+0000","mapping_epoch":47,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7721926,"num_objects":1887,"num_object_clones":0,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18689,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18280","reported_seq":18534,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.572976+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:23.572976+0000","last_peered":"2024-09-16T10:32:23.572976+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:23.572976+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8200","ondisk_log_start":"44'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7184867,"num_objects":1755,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18279,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19060","reported_seq":19339,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.573378+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:23.573378+0000","last_peered":"2024-09-16T10:32:23.573378+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:23.573378+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7595433,"num_objects":1857,"num_object_clones":0,"num_object_copies":3714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19057,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18383","reported_seq":20530,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.557997+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:23.557997+0000","last_peered":"2024-09-16T10:32:23.557997+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:23.557997+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8300","ondisk_log_start":"44'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7197638,"num_objects":1759,"num_object_clones":0,"num_object_copies":3518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18381,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1033,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19082","reported_seq":19326,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.575225+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:23.575225+0000","last_peered":"2024-09-16T10:32:23.575225+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:23.575225+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7432076,"num_objects":1818,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19079,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":118047246,"num_objects":28837,"num_object_clones":0,"num_object_copies":57674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28837,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277468,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14234,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":447193088,"data_stored":447116297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160903,"ondisk_log_size":160903,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018999,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516540,"kb_used_data":196144,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93855300,"statfs":{"total":96636764160,"available":96107827200,"internally_reserved":0,"allocated":200851456,"data_stored":200572810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":11.718999999999999},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":2.4860000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":16.199999999999999},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":11.766999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691943,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513096,"kb_used_data":88500,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93858744,"statfs":{"total":96636764160,"available":96111353856,"internally_reserved":0,"allocated":90624000,"data_stored":90287546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":434761386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":3.5710000000000002},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.28100000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.2880000000000003},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":6.0069999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215226,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586836,"kb_used_data":154120,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93785004,"statfs":{"total":96636764160,"available":96035844096,"internally_reserved":0,"allocated":157818880,"data_stored":157547920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":443084456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.60999999999999999},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":8.2579999999999991}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":156950528,"data_stored":156946915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90193920,"data_stored":90128427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200048640,"data_stored":200040955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:26.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:27.030 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:27.051 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-16T10:32:26.741012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118047265,"num_objects":28839,"num_object_clones":0,"num_object_copies":57678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277470,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14235,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1616472,"kb_used_data":438764,"kb_used_omap":12,"kb_used_meta":1177651,"kb_avail":281499048,"statfs":{"total":289910292480,"available":288255025152,"internally_reserved":0,"allocated":449294336,"data_stored":448408276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1205914626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,9,13,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18993152,"num_objects":-4637,"num_object_clones":0,"num_object_copies":-9274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002806"},"pg_stats":[{"pgid":"2.f","version":"48'18719","reported_seq":27187,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:23.572223+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:23.572223+0000","last_peered":"2024-09-16T10:32:23.572223+0000","last_clean":"2024-09-16T10:32:23.572223+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:23.572223+0000","last_undegraded":"2024-09-16T10:32:23.572223+0000","last_fullsized":"2024-09-16T10:32:23.572223+0000","mapping_epoch":45,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7163904,"num_objects":1749,"num_object_clones":0,"num_object_copies":3498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18719,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18653","reported_seq":18815,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.455567+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:22.455567+0000","last_peered":"2024-09-16T10:32:22.455567+0000","last_clean":"2024-09-16T10:32:22.455567+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:22.455567+0000","last_undegraded":"2024-09-16T10:32:22.455567+0000","last_fullsized":"2024-09-16T10:32:22.455567+0000","mapping_epoch":34,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7409664,"num_objects":1809,"num_object_clones":0,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14866,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18793","reported_seq":19050,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:22.361105+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:22.361105+0000","last_peered":"2024-09-16T10:32:22.361105+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:22.361105+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7659544,"num_objects":1871,"num_object_clones":0,"num_object_copies":3742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14850,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2432}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18704","reported_seq":18767,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.467414+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:22.467414+0000","last_peered":"2024-09-16T10:32:22.467414+0000","last_clean":"2024-09-16T10:32:22.467414+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:22.467414+0000","last_undegraded":"2024-09-16T10:32:22.467414+0000","last_fullsized":"2024-09-16T10:32:22.467414+0000","mapping_epoch":34,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7200768,"num_objects":1758,"num_object_clones":0,"num_object_copies":3516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18704,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18735","reported_seq":18898,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:23.589758+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:23.589758+0000","last_peered":"2024-09-16T10:32:23.589758+0000","last_clean":"2024-09-16T10:32:23.589758+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:23.589758+0000","last_undegraded":"2024-09-16T10:32:23.589758+0000","last_fullsized":"2024-09-16T10:32:23.589758+0000","mapping_epoch":33,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7249920,"num_objects":1770,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15377,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18555","reported_seq":18891,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.539657+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:23.539657+0000","last_peered":"2024-09-16T10:32:23.539657+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:23.539657+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7438336,"num_objects":1816,"num_object_clones":0,"num_object_copies":3632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15180,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19155","reported_seq":19567,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.471360+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:23.471360+0000","last_peered":"2024-09-16T10:32:23.471360+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:23.471360+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7229440,"num_objects":1765,"num_object_clones":0,"num_object_copies":3530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15397,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18687","reported_seq":18868,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:23.590015+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:23.590015+0000","last_peered":"2024-09-16T10:32:23.590015+0000","last_clean":"2024-09-16T10:32:23.590015+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:23.590015+0000","last_undegraded":"2024-09-16T10:32:23.590015+0000","last_fullsized":"2024-09-16T10:32:23.590015+0000","mapping_epoch":33,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6975488,"num_objects":1703,"num_object_clones":0,"num_object_copies":3406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14490,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18616","reported_seq":27126,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.622943+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:22.622943+0000","last_peered":"2024-09-16T10:32:22.622943+0000","last_clean":"2024-09-16T10:32:22.622943+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:22.622943+0000","last_undegraded":"2024-09-16T10:32:22.622943+0000","last_fullsized":"2024-09-16T10:32:22.622943+0000","mapping_epoch":43,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7558086,"num_objects":1847,"num_object_clones":0,"num_object_copies":3694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18614,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19211","reported_seq":19490,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.465806+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:23.465806+0000","last_peered":"2024-09-16T10:32:23.465806+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:23.465806+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7492550,"num_objects":1831,"num_object_clones":0,"num_object_copies":3662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19209,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18579","reported_seq":18779,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:22.453585+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:22.453585+0000","last_peered":"2024-09-16T10:32:22.453585+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:22.453585+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7537606,"num_objects":1842,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18577,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18691","reported_seq":27179,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.413134+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:22.413134+0000","last_peered":"2024-09-16T10:32:22.413134+0000","last_clean":"2024-09-16T10:32:22.413134+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:22.413134+0000","last_undegraded":"2024-09-16T10:32:22.413134+0000","last_fullsized":"2024-09-16T10:32:22.413134+0000","mapping_epoch":47,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7721926,"num_objects":1887,"num_object_clones":0,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18689,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18280","reported_seq":18534,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.572976+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:23.572976+0000","last_peered":"2024-09-16T10:32:23.572976+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:23.572976+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8200","ondisk_log_start":"44'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7184867,"num_objects":1755,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18279,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19060","reported_seq":19339,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.573378+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:23.573378+0000","last_peered":"2024-09-16T10:32:23.573378+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:23.573378+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7595433,"num_objects":1857,"num_object_clones":0,"num_object_copies":3714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19057,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18383","reported_seq":20530,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.557997+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:23.557997+0000","last_peered":"2024-09-16T10:32:23.557997+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:23.557997+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8300","ondisk_log_start":"44'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7197638,"num_objects":1759,"num_object_clones":0,"num_object_copies":3518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18381,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1033,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19082","reported_seq":19326,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.575225+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:23.575225+0000","last_peered":"2024-09-16T10:32:23.575225+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:23.575225+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7432076,"num_objects":1818,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19079,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":118047246,"num_objects":28837,"num_object_clones":0,"num_object_copies":57674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28837,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277468,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14234,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":447193088,"data_stored":447116297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160903,"ondisk_log_size":160903,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018999,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516540,"kb_used_data":196144,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93855300,"statfs":{"total":96636764160,"available":96107827200,"internally_reserved":0,"allocated":200851456,"data_stored":200572810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":11.718999999999999},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":2.4860000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":16.199999999999999},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":11.766999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691943,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513096,"kb_used_data":88500,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93858744,"statfs":{"total":96636764160,"available":96111353856,"internally_reserved":0,"allocated":90624000,"data_stored":90287546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":434761386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":3.5710000000000002},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.28100000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.2880000000000003},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":6.0069999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215226,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586836,"kb_used_data":154120,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93785004,"statfs":{"total":96636764160,"available":96035844096,"internally_reserved":0,"allocated":157818880,"data_stored":157547920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":443084456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.60999999999999999},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":8.2579999999999991}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":156950528,"data_stored":156946915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90193920,"data_stored":90128427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200048640,"data_stored":200040955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:27.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:32:27.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'18793", 'reported_seq': 19050, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:22.361105+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:22.361105+0000', 'last_peered': '2024-09-16T10:32:22.361105+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:22.361105+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'8700", 'ondisk_log_start': "44'8700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7659544, 'num_objects': 1871, 'num_object_clones': 0, 'num_object_copies': 3742, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1871, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 14850, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2432}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:27.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:32:27.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'18555", 'reported_seq': 18891, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:23.539657+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:32:23.539657+0000', 'last_peered': '2024-09-16T10:32:23.539657+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:32:23.539657+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'8500", 'ondisk_log_start': "44'8500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7438336, 'num_objects': 1816, 'num_object_clones': 0, 'num_object_copies': 3632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1816, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15180, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:27.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:32:27.052 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'19155", 'reported_seq': 19567, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:23.471360+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:32:23.471360+0000', 'last_peered': '2024-09-16T10:32:23.471360+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:32:23.471360+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'9100", 'ondisk_log_start': "44'9100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7229440, 'num_objects': 1765, 'num_object_clones': 0, 'num_object_copies': 3530, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15397, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'19211", 'reported_seq': 19490, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:23.465806+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:32:23.465806+0000', 'last_peered': '2024-09-16T10:32:23.465806+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:32:23.465806+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'9200", 'ondisk_log_start': "44'9200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7492550, 'num_objects': 1831, 'num_object_clones': 0, 'num_object_copies': 3662, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19209, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2506}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'18579", 'reported_seq': 18779, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:22.453585+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:22.453585+0000', 'last_peered': '2024-09-16T10:32:22.453585+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:22.453585+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'8500", 'ondisk_log_start': "44'8500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7537606, 'num_objects': 1842, 'num_object_clones': 0, 'num_object_copies': 3684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18577, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2391}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'18280", 'reported_seq': 18534, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:23.572976+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:32:23.572976+0000', 'last_peered': '2024-09-16T10:32:23.572976+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:32:23.572976+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'8200", 'ondisk_log_start': "44'8200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7184867, 'num_objects': 1755, 'num_object_clones': 0, 'num_object_copies': 3510, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18279, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2320}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'19060", 'reported_seq': 19339, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:23.573378+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:32:23.573378+0000', 'last_peered': '2024-09-16T10:32:23.573378+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:32:23.573378+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'9000", 'ondisk_log_start': "44'9000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7595433, 'num_objects': 1857, 'num_object_clones': 0, 'num_object_copies': 3714, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19057, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'18383", 'reported_seq': 20530, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:23.557997+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:32:23.557997+0000', 'last_peered': '2024-09-16T10:32:23.557997+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:32:23.557997+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'8300", 'ondisk_log_start': "44'8300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7197638, 'num_objects': 1759, 'num_object_clones': 0, 'num_object_copies': 3518, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3138, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18381, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1033, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:27.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:32:27.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'19082", 'reported_seq': 19326, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:23.575225+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:32:23.575225+0000', 'last_peered': '2024-09-16T10:32:23.575225+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:32:23.575225+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'9000", 'ondisk_log_start': "44'9000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7432076, 'num_objects': 1818, 'num_object_clones': 0, 'num_object_copies': 3636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19079, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2504}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:27.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:32:27.737 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:27.737 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37112/57678 objects degraded (64.343%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":636,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":28839,"data_bytes":118047265,"bytes_used":1655267328,"bytes_avail":288255025152,"bytes_total":289910292480,"degraded_objects":37112,"degraded_total":57678,"degraded_ratio":0.64343423835777935,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":386},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:27.767 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:27.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:28.342 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:28.344 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:28.370 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-16T10:32:26.741012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118047265,"num_objects":28839,"num_object_clones":0,"num_object_copies":57678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277470,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14235,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160906,"ondisk_log_size":160906,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1616472,"kb_used_data":438764,"kb_used_omap":12,"kb_used_meta":1177651,"kb_avail":281499048,"statfs":{"total":289910292480,"available":288255025152,"internally_reserved":0,"allocated":449294336,"data_stored":448408276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1205914626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,9,13,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18993152,"num_objects":-4637,"num_object_clones":0,"num_object_copies":-9274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-160,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002806"},"pg_stats":[{"pgid":"2.f","version":"48'18719","reported_seq":27187,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:23.572223+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:23.572223+0000","last_peered":"2024-09-16T10:32:23.572223+0000","last_clean":"2024-09-16T10:32:23.572223+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:23.572223+0000","last_undegraded":"2024-09-16T10:32:23.572223+0000","last_fullsized":"2024-09-16T10:32:23.572223+0000","mapping_epoch":45,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7163904,"num_objects":1749,"num_object_clones":0,"num_object_copies":3498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18719,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18653","reported_seq":18815,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.455567+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:22.455567+0000","last_peered":"2024-09-16T10:32:22.455567+0000","last_clean":"2024-09-16T10:32:22.455567+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:22.455567+0000","last_undegraded":"2024-09-16T10:32:22.455567+0000","last_fullsized":"2024-09-16T10:32:22.455567+0000","mapping_epoch":34,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7409664,"num_objects":1809,"num_object_clones":0,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14866,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18793","reported_seq":19050,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:22.361105+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:22.361105+0000","last_peered":"2024-09-16T10:32:22.361105+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:22.361105+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7659544,"num_objects":1871,"num_object_clones":0,"num_object_copies":3742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14850,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2432}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18704","reported_seq":18767,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.467414+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:22.467414+0000","last_peered":"2024-09-16T10:32:22.467414+0000","last_clean":"2024-09-16T10:32:22.467414+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:22.467414+0000","last_undegraded":"2024-09-16T10:32:22.467414+0000","last_fullsized":"2024-09-16T10:32:22.467414+0000","mapping_epoch":34,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7200768,"num_objects":1758,"num_object_clones":0,"num_object_copies":3516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18704,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18735","reported_seq":18898,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:23.589758+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:23.589758+0000","last_peered":"2024-09-16T10:32:23.589758+0000","last_clean":"2024-09-16T10:32:23.589758+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:23.589758+0000","last_undegraded":"2024-09-16T10:32:23.589758+0000","last_fullsized":"2024-09-16T10:32:23.589758+0000","mapping_epoch":33,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7249920,"num_objects":1770,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15377,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18555","reported_seq":18891,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.539657+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:23.539657+0000","last_peered":"2024-09-16T10:32:23.539657+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:23.539657+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7438336,"num_objects":1816,"num_object_clones":0,"num_object_copies":3632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15180,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19155","reported_seq":19567,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.471360+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:23.471360+0000","last_peered":"2024-09-16T10:32:23.471360+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:23.471360+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7229440,"num_objects":1765,"num_object_clones":0,"num_object_copies":3530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15397,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18687","reported_seq":18868,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:23.590015+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:23.590015+0000","last_peered":"2024-09-16T10:32:23.590015+0000","last_clean":"2024-09-16T10:32:23.590015+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:23.590015+0000","last_undegraded":"2024-09-16T10:32:23.590015+0000","last_fullsized":"2024-09-16T10:32:23.590015+0000","mapping_epoch":33,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6975488,"num_objects":1703,"num_object_clones":0,"num_object_copies":3406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14490,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18616","reported_seq":27126,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.622943+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:22.622943+0000","last_peered":"2024-09-16T10:32:22.622943+0000","last_clean":"2024-09-16T10:32:22.622943+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:22.622943+0000","last_undegraded":"2024-09-16T10:32:22.622943+0000","last_fullsized":"2024-09-16T10:32:22.622943+0000","mapping_epoch":43,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7558086,"num_objects":1847,"num_object_clones":0,"num_object_copies":3694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18614,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19211","reported_seq":19490,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.465806+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:23.465806+0000","last_peered":"2024-09-16T10:32:23.465806+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:23.465806+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7492550,"num_objects":1831,"num_object_clones":0,"num_object_copies":3662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19209,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2506}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18579","reported_seq":18779,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:22.453585+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:22.453585+0000","last_peered":"2024-09-16T10:32:22.453585+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:22.453585+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7537606,"num_objects":1842,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18577,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2391}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18691","reported_seq":27179,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:22.413134+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:22.413134+0000","last_peered":"2024-09-16T10:32:22.413134+0000","last_clean":"2024-09-16T10:32:22.413134+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:22.413134+0000","last_undegraded":"2024-09-16T10:32:22.413134+0000","last_fullsized":"2024-09-16T10:32:22.413134+0000","mapping_epoch":47,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7721926,"num_objects":1887,"num_object_clones":0,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18689,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18280","reported_seq":18534,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.572976+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:23.572976+0000","last_peered":"2024-09-16T10:32:23.572976+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:23.572976+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8200","ondisk_log_start":"44'8200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7184867,"num_objects":1755,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18279,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2320}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19060","reported_seq":19339,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.573378+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:23.573378+0000","last_peered":"2024-09-16T10:32:23.573378+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:23.573378+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7595433,"num_objects":1857,"num_object_clones":0,"num_object_copies":3714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19057,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2410}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18383","reported_seq":20530,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.557997+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:23.557997+0000","last_peered":"2024-09-16T10:32:23.557997+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:23.557997+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8300","ondisk_log_start":"44'8300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7197638,"num_objects":1759,"num_object_clones":0,"num_object_copies":3518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18381,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1033,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1379}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19082","reported_seq":19326,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:23.575225+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:23.575225+0000","last_peered":"2024-09-16T10:32:23.575225+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:23.575225+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7432076,"num_objects":1818,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19079,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":118047246,"num_objects":28837,"num_object_clones":0,"num_object_copies":57674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28837,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":277468,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14234,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":447193088,"data_stored":447116297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160903,"ondisk_log_size":160903,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849018999,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516540,"kb_used_data":196144,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93855300,"statfs":{"total":96636764160,"available":96107827200,"internally_reserved":0,"allocated":200851456,"data_stored":200572810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":328068784},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":11.718999999999999},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":2.4860000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":16.199999999999999},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":11.766999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691943,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513096,"kb_used_data":88500,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93858744,"statfs":{"total":96636764160,"available":96111353856,"internally_reserved":0,"allocated":90624000,"data_stored":90287546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":434761386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":3.5710000000000002},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.28100000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.2880000000000003},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":6.0069999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215226,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586836,"kb_used_data":154120,"kb_used_omap":4,"kb_used_meta":432699,"kb_avail":93785004,"statfs":{"total":96636764160,"available":96035844096,"internally_reserved":0,"allocated":157818880,"data_stored":157547920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":443084456},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,4,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.60999999999999999},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":8.2579999999999991}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":156950528,"data_stored":156946915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90193920,"data_stored":90128427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200048640,"data_stored":200040955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:31.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:31.955 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:31.958 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:31.983 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-16T10:32:30.741934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108614177,"num_objects":26536,"num_object_clones":0,"num_object_copies":53072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26536,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":279773,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14314,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160709,"ondisk_log_size":160709,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1614376,"kb_used_data":420292,"kb_used_omap":12,"kb_used_meta":1194035,"kb_avail":281501144,"statfs":{"total":289910292480,"available":288257171456,"internally_reserved":0,"allocated":430379008,"data_stored":429492948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1222691842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,8,6,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18944000,"num_objects":-4625,"num_object_clones":0,"num_object_copies":-9250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002817"},"pg_stats":[{"pgid":"2.f","version":"48'18861","reported_seq":27329,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:28.562137+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:28.562137+0000","last_peered":"2024-09-16T10:32:28.562137+0000","last_clean":"2024-09-16T10:32:28.562137+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:28.562137+0000","last_undegraded":"2024-09-16T10:32:28.562137+0000","last_fullsized":"2024-09-16T10:32:28.562137+0000","mapping_epoch":45,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6582272,"num_objects":1607,"num_object_clones":0,"num_object_copies":3214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18861,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18790","reported_seq":18953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:27.407917+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:27.407917+0000","last_peered":"2024-09-16T10:32:27.407917+0000","last_clean":"2024-09-16T10:32:27.407917+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:27.407917+0000","last_undegraded":"2024-09-16T10:32:27.407917+0000","last_fullsized":"2024-09-16T10:32:27.407917+0000","mapping_epoch":34,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6848512,"num_objects":1672,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15003,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18944","reported_seq":19201,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:27.375433+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:27.375433+0000","last_peered":"2024-09-16T10:32:27.375433+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:27.375433+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7041048,"num_objects":1720,"num_object_clones":0,"num_object_copies":3440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15001,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2583}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18846","reported_seq":18909,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:27.326876+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:27.326876+0000","last_peered":"2024-09-16T10:32:27.326876+0000","last_clean":"2024-09-16T10:32:27.326876+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:27.326876+0000","last_undegraded":"2024-09-16T10:32:27.326876+0000","last_fullsized":"2024-09-16T10:32:27.326876+0000","mapping_epoch":34,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":1616,"num_object_clones":0,"num_object_copies":3232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18846,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18872","reported_seq":19035,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:28.603873+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:28.603873+0000","last_peered":"2024-09-16T10:32:28.603873+0000","last_clean":"2024-09-16T10:32:28.603873+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:28.603873+0000","last_undegraded":"2024-09-16T10:32:28.603873+0000","last_fullsized":"2024-09-16T10:32:28.603873+0000","mapping_epoch":33,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6688768,"num_objects":1633,"num_object_clones":0,"num_object_copies":3266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15514,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18681","reported_seq":19017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.496235+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:28.496235+0000","last_peered":"2024-09-16T10:32:28.496235+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:28.496235+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6922240,"num_objects":1690,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15306,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19312","reported_seq":19724,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.524981+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:28.524981+0000","last_peered":"2024-09-16T10:32:28.524981+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:28.524981+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6586368,"num_objects":1608,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15554,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2679}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18835","reported_seq":19016,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:28.558888+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:28.558888+0000","last_peered":"2024-09-16T10:32:28.558888+0000","last_clean":"2024-09-16T10:32:28.558888+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:28.558888+0000","last_undegraded":"2024-09-16T10:32:28.558888+0000","last_fullsized":"2024-09-16T10:32:28.558888+0000","mapping_epoch":33,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6369280,"num_objects":1555,"num_object_clones":0,"num_object_copies":3110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14638,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18782","reported_seq":27292,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:27.638140+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:27.638140+0000","last_peered":"2024-09-16T10:32:27.638140+0000","last_clean":"2024-09-16T10:32:27.638140+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:27.638140+0000","last_undegraded":"2024-09-16T10:32:27.638140+0000","last_fullsized":"2024-09-16T10:32:27.638140+0000","mapping_epoch":43,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6878150,"num_objects":1681,"num_object_clones":0,"num_object_copies":3362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18780,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19360","reported_seq":19631,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.614764+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:28.614764+0000","last_peered":"2024-09-16T10:32:28.614764+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:28.614764+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6882246,"num_objects":1682,"num_object_clones":0,"num_object_copies":3364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19358,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2655}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18743","reported_seq":18943,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:27.373326+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:27.373326+0000","last_peered":"2024-09-16T10:32:27.373326+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:27.373326+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6865862,"num_objects":1678,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18741,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18822","reported_seq":27310,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:27.387444+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:27.387444+0000","last_peered":"2024-09-16T10:32:27.387444+0000","last_clean":"2024-09-16T10:32:27.387444+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:27.387444+0000","last_undegraded":"2024-09-16T10:32:27.387444+0000","last_fullsized":"2024-09-16T10:32:27.387444+0000","mapping_epoch":47,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7185350,"num_objects":1756,"num_object_clones":0,"num_object_copies":3512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18820,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18409","reported_seq":18663,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.623954+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:28.623954+0000","last_peered":"2024-09-16T10:32:28.623954+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:28.623954+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6656483,"num_objects":1626,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18408,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19219","reported_seq":19497,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.596815+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:28.596815+0000","last_peered":"2024-09-16T10:32:28.596815+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:28.596815+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6944169,"num_objects":1698,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19216,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18521","reported_seq":20829,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.642270+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:28.642270+0000","last_peered":"2024-09-16T10:32:28.642270+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:28.642270+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6632390,"num_objects":1621,"num_object_clones":0,"num_object_copies":3242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18519,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1438}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19209","reported_seq":19454,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.602734+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:28.602734+0000","last_peered":"2024-09-16T10:32:28.602734+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:28.602734+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6911884,"num_objects":1691,"num_object_clones":0,"num_object_copies":3382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19206,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":108614158,"num_objects":26534,"num_object_clones":0,"num_object_copies":53068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26534,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":279771,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14313,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":428179456,"data_stored":428102665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160706,"ondisk_log_size":160706,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019000,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515892,"kb_used_data":191400,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93855948,"statfs":{"total":96636764160,"available":96108490752,"internally_reserved":0,"allocated":195993600,"data_stored":195714954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":12.265000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.0490000000000004},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.40100000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691944,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514208,"kb_used_data":81428,"kb_used_omap":4,"kb_used_meta":432763,"kb_avail":93857632,"statfs":{"total":96636764160,"available":96110215168,"internally_reserved":0,"allocated":83382272,"data_stored":83045818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":443149994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":4.2409999999999997},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":4.2060000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.93999999999999995}]}]},{"osd":0,"up_from":24,"seq":103079215227,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584276,"kb_used_data":147464,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93787564,"statfs":{"total":96636764160,"available":96038465536,"internally_reserved":0,"allocated":151003136,"data_stored":150732176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,0,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":2.448}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":13.457000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":150011904,"data_stored":150008291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82907136,"data_stored":82841643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195260416,"data_stored":195252731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:31.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:32.526 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:32.527 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:32.559 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-16T10:32:30.741934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108614177,"num_objects":26536,"num_object_clones":0,"num_object_copies":53072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26536,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":279773,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14314,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160709,"ondisk_log_size":160709,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1614376,"kb_used_data":420292,"kb_used_omap":12,"kb_used_meta":1194035,"kb_avail":281501144,"statfs":{"total":289910292480,"available":288257171456,"internally_reserved":0,"allocated":430379008,"data_stored":429492948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1222691842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,8,6,6,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18944000,"num_objects":-4625,"num_object_clones":0,"num_object_copies":-9250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002817"},"pg_stats":[{"pgid":"2.f","version":"48'18861","reported_seq":27329,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:28.562137+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:28.562137+0000","last_peered":"2024-09-16T10:32:28.562137+0000","last_clean":"2024-09-16T10:32:28.562137+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:28.562137+0000","last_undegraded":"2024-09-16T10:32:28.562137+0000","last_fullsized":"2024-09-16T10:32:28.562137+0000","mapping_epoch":45,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6582272,"num_objects":1607,"num_object_clones":0,"num_object_copies":3214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18861,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18790","reported_seq":18953,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:27.407917+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:27.407917+0000","last_peered":"2024-09-16T10:32:27.407917+0000","last_clean":"2024-09-16T10:32:27.407917+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:27.407917+0000","last_undegraded":"2024-09-16T10:32:27.407917+0000","last_fullsized":"2024-09-16T10:32:27.407917+0000","mapping_epoch":34,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6848512,"num_objects":1672,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15003,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'18944","reported_seq":19201,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:27.375433+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:27.375433+0000","last_peered":"2024-09-16T10:32:27.375433+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:27.375433+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7041048,"num_objects":1720,"num_object_clones":0,"num_object_copies":3440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15001,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2583}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18846","reported_seq":18909,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:27.326876+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:27.326876+0000","last_peered":"2024-09-16T10:32:27.326876+0000","last_clean":"2024-09-16T10:32:27.326876+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:27.326876+0000","last_undegraded":"2024-09-16T10:32:27.326876+0000","last_fullsized":"2024-09-16T10:32:27.326876+0000","mapping_epoch":34,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":1616,"num_object_clones":0,"num_object_copies":3232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18846,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18872","reported_seq":19035,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:28.603873+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:28.603873+0000","last_peered":"2024-09-16T10:32:28.603873+0000","last_clean":"2024-09-16T10:32:28.603873+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:28.603873+0000","last_undegraded":"2024-09-16T10:32:28.603873+0000","last_fullsized":"2024-09-16T10:32:28.603873+0000","mapping_epoch":33,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6688768,"num_objects":1633,"num_object_clones":0,"num_object_copies":3266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15514,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18681","reported_seq":19017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.496235+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:28.496235+0000","last_peered":"2024-09-16T10:32:28.496235+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:28.496235+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6922240,"num_objects":1690,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15306,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19312","reported_seq":19724,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.524981+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:28.524981+0000","last_peered":"2024-09-16T10:32:28.524981+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:28.524981+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6586368,"num_objects":1608,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15554,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2679}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18835","reported_seq":19016,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:28.558888+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:28.558888+0000","last_peered":"2024-09-16T10:32:28.558888+0000","last_clean":"2024-09-16T10:32:28.558888+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:28.558888+0000","last_undegraded":"2024-09-16T10:32:28.558888+0000","last_fullsized":"2024-09-16T10:32:28.558888+0000","mapping_epoch":33,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6369280,"num_objects":1555,"num_object_clones":0,"num_object_copies":3110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14638,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18782","reported_seq":27292,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:27.638140+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:27.638140+0000","last_peered":"2024-09-16T10:32:27.638140+0000","last_clean":"2024-09-16T10:32:27.638140+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:27.638140+0000","last_undegraded":"2024-09-16T10:32:27.638140+0000","last_fullsized":"2024-09-16T10:32:27.638140+0000","mapping_epoch":43,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6878150,"num_objects":1681,"num_object_clones":0,"num_object_copies":3362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18780,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19360","reported_seq":19631,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.614764+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:28.614764+0000","last_peered":"2024-09-16T10:32:28.614764+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:28.614764+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6882246,"num_objects":1682,"num_object_clones":0,"num_object_copies":3364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19358,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2655}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18743","reported_seq":18943,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:27.373326+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:27.373326+0000","last_peered":"2024-09-16T10:32:27.373326+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:27.373326+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6865862,"num_objects":1678,"num_object_clones":0,"num_object_copies":3356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18741,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2555}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18822","reported_seq":27310,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:27.387444+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:27.387444+0000","last_peered":"2024-09-16T10:32:27.387444+0000","last_clean":"2024-09-16T10:32:27.387444+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:27.387444+0000","last_undegraded":"2024-09-16T10:32:27.387444+0000","last_fullsized":"2024-09-16T10:32:27.387444+0000","mapping_epoch":47,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7185350,"num_objects":1756,"num_object_clones":0,"num_object_copies":3512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18820,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18409","reported_seq":18663,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.623954+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:28.623954+0000","last_peered":"2024-09-16T10:32:28.623954+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:28.623954+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6656483,"num_objects":1626,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18408,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19219","reported_seq":19497,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.596815+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:28.596815+0000","last_peered":"2024-09-16T10:32:28.596815+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:28.596815+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6944169,"num_objects":1698,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19216,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18521","reported_seq":20829,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.642270+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:28.642270+0000","last_peered":"2024-09-16T10:32:28.642270+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:28.642270+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6632390,"num_objects":1621,"num_object_clones":0,"num_object_copies":3242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18519,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1438}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19209","reported_seq":19454,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.602734+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:28.602734+0000","last_peered":"2024-09-16T10:32:28.602734+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:28.602734+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6911884,"num_objects":1691,"num_object_clones":0,"num_object_copies":3382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19206,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":108614158,"num_objects":26534,"num_object_clones":0,"num_object_copies":53068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26534,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":279771,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14313,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":428179456,"data_stored":428102665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160706,"ondisk_log_size":160706,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019000,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515892,"kb_used_data":191400,"kb_used_omap":4,"kb_used_meta":324475,"kb_avail":93855948,"statfs":{"total":96636764160,"available":96108490752,"internally_reserved":0,"allocated":195993600,"data_stored":195714954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":332263088},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":12.265000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.0490000000000004},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.40100000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691944,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514208,"kb_used_data":81428,"kb_used_omap":4,"kb_used_meta":432763,"kb_avail":93857632,"statfs":{"total":96636764160,"available":96110215168,"internally_reserved":0,"allocated":83382272,"data_stored":83045818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":443149994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":4.2409999999999997},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":4.2060000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.93999999999999995}]}]},{"osd":0,"up_from":24,"seq":103079215227,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584276,"kb_used_data":147464,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93787564,"statfs":{"total":96636764160,"available":96038465536,"internally_reserved":0,"allocated":151003136,"data_stored":150732176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,0,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":2.448}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":13.457000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":150011904,"data_stored":150008291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82907136,"data_stored":82841643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":195260416,"data_stored":195252731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:32.560 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:32:32.560 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'18944", 'reported_seq': 19201, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:27.375433+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:27.375433+0000', 'last_peered': '2024-09-16T10:32:27.375433+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:27.375433+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'8900", 'ondisk_log_start': "44'8900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7041048, 'num_objects': 1720, 'num_object_clones': 0, 'num_object_copies': 3440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1720, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15001, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2583}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'18681", 'reported_seq': 19017, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:28.496235+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:32:28.496235+0000', 'last_peered': '2024-09-16T10:32:28.496235+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:32:28.496235+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'8600", 'ondisk_log_start': "44'8600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6922240, 'num_objects': 1690, 'num_object_clones': 0, 'num_object_copies': 3380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1690, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15306, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'19312", 'reported_seq': 19724, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:28.524981+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:32:28.524981+0000', 'last_peered': '2024-09-16T10:32:28.524981+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:32:28.524981+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'9300", 'ondisk_log_start': "44'9300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6586368, 'num_objects': 1608, 'num_object_clones': 0, 'num_object_copies': 3216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15554, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2679}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'19360", 'reported_seq': 19631, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:28.614764+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:32:28.614764+0000', 'last_peered': '2024-09-16T10:32:28.614764+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:32:28.614764+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'9300", 'ondisk_log_start': "44'9300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6882246, 'num_objects': 1682, 'num_object_clones': 0, 'num_object_copies': 3364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19358, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2655}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'18743", 'reported_seq': 18943, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:27.373326+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:27.373326+0000', 'last_peered': '2024-09-16T10:32:27.373326+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:27.373326+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'8700", 'ondisk_log_start': "44'8700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6865862, 'num_objects': 1678, 'num_object_clones': 0, 'num_object_copies': 3356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18741, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2555}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'18409", 'reported_seq': 18663, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:28.623954+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:32:28.623954+0000', 'last_peered': '2024-09-16T10:32:28.623954+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:32:28.623954+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'8400", 'ondisk_log_start': "44'8400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6656483, 'num_objects': 1626, 'num_object_clones': 0, 'num_object_copies': 3252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18408, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:32.561 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:32:32.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'19219", 'reported_seq': 19497, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:28.596815+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:32:28.596815+0000', 'last_peered': '2024-09-16T10:32:28.596815+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:32:28.596815+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'9200", 'ondisk_log_start': "44'9200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6944169, 'num_objects': 1698, 'num_object_clones': 0, 'num_object_copies': 3396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19216, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2569}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:32.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:32:32.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'18521", 'reported_seq': 20829, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:28.642270+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:32:28.642270+0000', 'last_peered': '2024-09-16T10:32:28.642270+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:32:28.642270+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'8500", 'ondisk_log_start': "44'8500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6632390, 'num_objects': 1621, 'num_object_clones': 0, 'num_object_copies': 3242, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3059, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18519, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1112, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1438}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:32.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:32:32.562 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'19209", 'reported_seq': 19454, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:28.602734+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:32:28.602734+0000', 'last_peered': '2024-09-16T10:32:28.602734+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:32:28.602734+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'9200", 'ondisk_log_start': "44'9200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6911884, 'num_objects': 1691, 'num_object_clones': 0, 'num_object_copies': 3382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1691, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19206, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2631}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:32.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:32:33.251 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:33.251 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 37033/51588 objects degraded (71.786%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":642,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":25794,"data_bytes":105574945,"bytes_used":1652658176,"bytes_avail":288257634304,"bytes_total":289910292480,"degraded_objects":37033,"degraded_total":51588,"degraded_ratio":0.71786074280840506,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":447},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:33.277 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:33.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:33.826 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:33.826 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:33.854 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-16T10:32:32.742375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":105574945,"num_objects":25794,"num_object_clones":0,"num_object_copies":51588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25794,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":280515,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14314,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160851,"ondisk_log_size":160851,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1613924,"kb_used_data":415748,"kb_used_omap":12,"kb_used_meta":1198131,"kb_avail":281501596,"statfs":{"total":289910292480,"available":288257634304,"internally_reserved":0,"allocated":425725952,"data_stored":424839892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1226886146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,10,5,12,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21983232,"num_objects":-5367,"num_object_clones":0,"num_object_copies":-10734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5367,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002738"},"pg_stats":[{"pgid":"2.f","version":"48'18861","reported_seq":27329,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:28.562137+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:28.562137+0000","last_peered":"2024-09-16T10:32:28.562137+0000","last_clean":"2024-09-16T10:32:28.562137+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:28.562137+0000","last_undegraded":"2024-09-16T10:32:28.562137+0000","last_fullsized":"2024-09-16T10:32:28.562137+0000","mapping_epoch":45,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6582272,"num_objects":1607,"num_object_clones":0,"num_object_copies":3214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18861,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18939","reported_seq":19102,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.413840+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:32.413840+0000","last_peered":"2024-09-16T10:32:32.413840+0000","last_clean":"2024-09-16T10:32:32.413840+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:32.413840+0000","last_undegraded":"2024-09-16T10:32:32.413840+0000","last_fullsized":"2024-09-16T10:32:32.413840+0000","mapping_epoch":34,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6238208,"num_objects":1523,"num_object_clones":0,"num_object_copies":3046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15152,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19086","reported_seq":19342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:32.486258+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:32.486258+0000","last_peered":"2024-09-16T10:32:32.486258+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:32.486258+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6459416,"num_objects":1578,"num_object_clones":0,"num_object_copies":3156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15143,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2725}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18996","reported_seq":19059,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.471949+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:32.471949+0000","last_peered":"2024-09-16T10:32:32.471949+0000","last_clean":"2024-09-16T10:32:32.471949+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:32.471949+0000","last_undegraded":"2024-09-16T10:32:32.471949+0000","last_fullsized":"2024-09-16T10:32:32.471949+0000","mapping_epoch":34,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":1466,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18996,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'18872","reported_seq":19035,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:28.603873+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:28.603873+0000","last_peered":"2024-09-16T10:32:28.603873+0000","last_clean":"2024-09-16T10:32:28.603873+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:28.603873+0000","last_undegraded":"2024-09-16T10:32:28.603873+0000","last_fullsized":"2024-09-16T10:32:28.603873+0000","mapping_epoch":33,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6688768,"num_objects":1633,"num_object_clones":0,"num_object_copies":3266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15514,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18681","reported_seq":19017,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.496235+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:28.496235+0000","last_peered":"2024-09-16T10:32:28.496235+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:28.496235+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6922240,"num_objects":1690,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15306,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19312","reported_seq":19724,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.524981+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:28.524981+0000","last_peered":"2024-09-16T10:32:28.524981+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:28.524981+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6586368,"num_objects":1608,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15554,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2679}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18835","reported_seq":19016,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:28.558888+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:28.558888+0000","last_peered":"2024-09-16T10:32:28.558888+0000","last_clean":"2024-09-16T10:32:28.558888+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:28.558888+0000","last_undegraded":"2024-09-16T10:32:28.558888+0000","last_fullsized":"2024-09-16T10:32:28.558888+0000","mapping_epoch":33,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6369280,"num_objects":1555,"num_object_clones":0,"num_object_copies":3110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14638,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18782","reported_seq":27292,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:27.638140+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:27.638140+0000","last_peered":"2024-09-16T10:32:27.638140+0000","last_clean":"2024-09-16T10:32:27.638140+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:27.638140+0000","last_undegraded":"2024-09-16T10:32:27.638140+0000","last_fullsized":"2024-09-16T10:32:27.638140+0000","mapping_epoch":43,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6878150,"num_objects":1681,"num_object_clones":0,"num_object_copies":3362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18780,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19360","reported_seq":19631,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.614764+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:28.614764+0000","last_peered":"2024-09-16T10:32:28.614764+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:28.614764+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6882246,"num_objects":1682,"num_object_clones":0,"num_object_copies":3364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19358,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2655}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18883","reported_seq":19083,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:32.436574+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:32.436574+0000","last_peered":"2024-09-16T10:32:32.436574+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:32.436574+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6292422,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18881,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2695}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18983","reported_seq":27471,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.457306+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:32.457306+0000","last_peered":"2024-09-16T10:32:32.457306+0000","last_clean":"2024-09-16T10:32:32.457306+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:32.457306+0000","last_undegraded":"2024-09-16T10:32:32.457306+0000","last_fullsized":"2024-09-16T10:32:32.457306+0000","mapping_epoch":47,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6525894,"num_objects":1595,"num_object_clones":0,"num_object_copies":3190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18981,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18409","reported_seq":18663,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.623954+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:28.623954+0000","last_peered":"2024-09-16T10:32:28.623954+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:28.623954+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8400","ondisk_log_start":"44'8400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6656483,"num_objects":1626,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18408,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2449}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19219","reported_seq":19497,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.596815+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:28.596815+0000","last_peered":"2024-09-16T10:32:28.596815+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:28.596815+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6944169,"num_objects":1698,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19216,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2569}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18521","reported_seq":20829,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.642270+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:28.642270+0000","last_peered":"2024-09-16T10:32:28.642270+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:28.642270+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6632390,"num_objects":1621,"num_object_clones":0,"num_object_copies":3242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3059,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18519,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1438}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19209","reported_seq":19454,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:28.602734+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:28.602734+0000","last_peered":"2024-09-16T10:32:28.602734+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:28.602734+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6911884,"num_objects":1691,"num_object_clones":0,"num_object_copies":3382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19206,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2631}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":105574926,"num_objects":25792,"num_object_clones":0,"num_object_copies":51584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37033,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25792,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":280513,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14313,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":423419904,"data_stored":423343113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019001,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515440,"kb_used_data":186856,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93856400,"statfs":{"total":96636764160,"available":96108953600,"internally_reserved":0,"allocated":191340544,"data_stored":191061898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.90600000000000003}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.103},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":1.113}]}]},{"osd":1,"up_from":40,"seq":171798691944,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":514208,"kb_used_data":81428,"kb_used_omap":4,"kb_used_meta":432763,"kb_avail":93857632,"statfs":{"total":96636764160,"available":96110215168,"internally_reserved":0,"allocated":83382272,"data_stored":83045818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":443149994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":4.2409999999999997},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":4.2060000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.93999999999999995}]}]},{"osd":0,"up_from":24,"seq":103079215227,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584276,"kb_used_data":147464,"kb_used_omap":4,"kb_used_meta":436795,"kb_avail":93787564,"statfs":{"total":96636764160,"available":96038465536,"internally_reserved":0,"allocated":151003136,"data_stored":150732176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":447278760},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,0,5,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":2.448}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":13.457000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":150011904,"data_stored":150008291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":82907136,"data_stored":82841643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190500864,"data_stored":190493179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:36.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:37.412 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:37.415 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:37.436 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-16T10:32:36.744321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99119649,"num_objects":24218,"num_object_clones":0,"num_object_copies":48436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282091,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14397,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1591704,"kb_used_data":401720,"kb_used_omap":12,"kb_used_meta":1189939,"kb_avail":281523816,"statfs":{"total":289910292480,"available":288280387584,"internally_reserved":0,"allocated":411361280,"data_stored":410475220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1218497538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,6,10,11,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18927616,"num_objects":-4621,"num_object_clones":0,"num_object_copies":-9242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003766"},"pg_stats":[{"pgid":"2.f","version":"48'19022","reported_seq":27490,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:33.697440+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:33.697440+0000","last_peered":"2024-09-16T10:32:33.697440+0000","last_clean":"2024-09-16T10:32:33.697440+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:33.697440+0000","last_undegraded":"2024-09-16T10:32:33.697440+0000","last_fullsized":"2024-09-16T10:32:33.697440+0000","mapping_epoch":45,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5922816,"num_objects":1446,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19022,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18939","reported_seq":19102,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.413840+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:32.413840+0000","last_peered":"2024-09-16T10:32:32.413840+0000","last_clean":"2024-09-16T10:32:32.413840+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:32.413840+0000","last_undegraded":"2024-09-16T10:32:32.413840+0000","last_fullsized":"2024-09-16T10:32:32.413840+0000","mapping_epoch":34,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6238208,"num_objects":1523,"num_object_clones":0,"num_object_copies":3046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15152,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19086","reported_seq":19342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:32.486258+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:32.486258+0000","last_peered":"2024-09-16T10:32:32.486258+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:32.486258+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6459416,"num_objects":1578,"num_object_clones":0,"num_object_copies":3156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15143,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2725}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18996","reported_seq":19059,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.471949+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:32.471949+0000","last_peered":"2024-09-16T10:32:32.471949+0000","last_clean":"2024-09-16T10:32:32.471949+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:32.471949+0000","last_undegraded":"2024-09-16T10:32:32.471949+0000","last_fullsized":"2024-09-16T10:32:32.471949+0000","mapping_epoch":34,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":1466,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18996,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19003","reported_seq":19166,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:33.571892+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:33.571892+0000","last_peered":"2024-09-16T10:32:33.571892+0000","last_clean":"2024-09-16T10:32:33.571892+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:33.571892+0000","last_undegraded":"2024-09-16T10:32:33.571892+0000","last_fullsized":"2024-09-16T10:32:33.571892+0000","mapping_epoch":33,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6152192,"num_objects":1502,"num_object_clones":0,"num_object_copies":3004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15645,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18821","reported_seq":19157,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.613221+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:33.613221+0000","last_peered":"2024-09-16T10:32:33.613221+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:33.613221+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6348800,"num_objects":1550,"num_object_clones":0,"num_object_copies":3100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15446,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19454","reported_seq":19866,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.643045+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:33.643045+0000","last_peered":"2024-09-16T10:32:33.643045+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:33.643045+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":1466,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15696,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18953","reported_seq":19134,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:33.591823+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:33.591823+0000","last_peered":"2024-09-16T10:32:33.591823+0000","last_clean":"2024-09-16T10:32:33.591823+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:33.591823+0000","last_undegraded":"2024-09-16T10:32:33.591823+0000","last_fullsized":"2024-09-16T10:32:33.591823+0000","mapping_epoch":33,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5885952,"num_objects":1437,"num_object_clones":0,"num_object_copies":2874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14756,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18937","reported_seq":27447,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.671608+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:32.671608+0000","last_peered":"2024-09-16T10:32:32.671608+0000","last_clean":"2024-09-16T10:32:32.671608+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:32.671608+0000","last_undegraded":"2024-09-16T10:32:32.671608+0000","last_fullsized":"2024-09-16T10:32:32.671608+0000","mapping_epoch":43,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6243270,"num_objects":1526,"num_object_clones":0,"num_object_copies":3052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18935,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19504","reported_seq":19783,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.604921+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:33.604921+0000","last_peered":"2024-09-16T10:32:33.604921+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:33.604921+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6292422,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19502,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2799}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18883","reported_seq":19083,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:32.436574+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:32.436574+0000","last_peered":"2024-09-16T10:32:32.436574+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:32.436574+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6292422,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18881,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2695}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18983","reported_seq":27471,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.457306+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:32.457306+0000","last_peered":"2024-09-16T10:32:32.457306+0000","last_clean":"2024-09-16T10:32:32.457306+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:32.457306+0000","last_undegraded":"2024-09-16T10:32:32.457306+0000","last_fullsized":"2024-09-16T10:32:32.457306+0000","mapping_epoch":47,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6525894,"num_objects":1595,"num_object_clones":0,"num_object_copies":3190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18981,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18568","reported_seq":18822,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.610825+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:33.610825+0000","last_peered":"2024-09-16T10:32:33.610825+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:33.610825+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6005219,"num_objects":1467,"num_object_clones":0,"num_object_copies":2934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18567,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19360","reported_seq":19640,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.686377+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:33.686377+0000","last_peered":"2024-09-16T10:32:33.686377+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:33.686377+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6366633,"num_objects":1557,"num_object_clones":0,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19357,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18660","reported_seq":21133,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.717397+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:33.717397+0000","last_peered":"2024-09-16T10:32:33.717397+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:33.717397+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6063046,"num_objects":1482,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18658,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19355","reported_seq":19600,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.637284+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:33.637284+0000","last_peered":"2024-09-16T10:32:33.637284+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:33.637284+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6313868,"num_objects":1545,"num_object_clones":0,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19352,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2777}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":99119630,"num_objects":24216,"num_object_clones":0,"num_object_copies":48432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282089,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14396,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409026560,"data_stored":408949769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019001,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515440,"kb_used_data":186856,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93856400,"statfs":{"total":96636764160,"available":96108953600,"internally_reserved":0,"allocated":191340544,"data_stored":191061898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.90600000000000003}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.103},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":1.113}]}]},{"osd":1,"up_from":40,"seq":171798691945,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511264,"kb_used_data":74384,"kb_used_omap":4,"kb_used_meta":436859,"kb_avail":93860576,"statfs":{"total":96636764160,"available":96113229824,"internally_reserved":0,"allocated":76169216,"data_stored":75832762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":447344298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":6.9130000000000003},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":6.9429999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.93999999999999995}]}]},{"osd":0,"up_from":24,"seq":103079215228,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565000,"kb_used_data":140480,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93806840,"statfs":{"total":96636764160,"available":96058204160,"internally_reserved":0,"allocated":143851520,"data_stored":143580560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":434695848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":9.2699999999999996},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":9.2910000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":1.5900000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.073}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":142905344,"data_stored":142901731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75620352,"data_stored":75554859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190500864,"data_stored":190493179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:37.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:38.028 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:38.028 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:38.051 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-16T10:32:36.744321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":99119649,"num_objects":24218,"num_object_clones":0,"num_object_copies":48436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282091,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14397,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1591704,"kb_used_data":401720,"kb_used_omap":12,"kb_used_meta":1189939,"kb_avail":281523816,"statfs":{"total":289910292480,"available":288280387584,"internally_reserved":0,"allocated":411361280,"data_stored":410475220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1218497538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,6,10,11,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18927616,"num_objects":-4621,"num_object_clones":0,"num_object_copies":-9242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003766"},"pg_stats":[{"pgid":"2.f","version":"48'19022","reported_seq":27490,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:33.697440+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:33.697440+0000","last_peered":"2024-09-16T10:32:33.697440+0000","last_clean":"2024-09-16T10:32:33.697440+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:33.697440+0000","last_undegraded":"2024-09-16T10:32:33.697440+0000","last_fullsized":"2024-09-16T10:32:33.697440+0000","mapping_epoch":45,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5922816,"num_objects":1446,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19022,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'18939","reported_seq":19102,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.413840+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:32.413840+0000","last_peered":"2024-09-16T10:32:32.413840+0000","last_clean":"2024-09-16T10:32:32.413840+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:32.413840+0000","last_undegraded":"2024-09-16T10:32:32.413840+0000","last_fullsized":"2024-09-16T10:32:32.413840+0000","mapping_epoch":34,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6238208,"num_objects":1523,"num_object_clones":0,"num_object_copies":3046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15152,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19086","reported_seq":19342,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:32.486258+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:32.486258+0000","last_peered":"2024-09-16T10:32:32.486258+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:32.486258+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6459416,"num_objects":1578,"num_object_clones":0,"num_object_copies":3156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15143,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2725}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'18996","reported_seq":19059,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.471949+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:32.471949+0000","last_peered":"2024-09-16T10:32:32.471949+0000","last_clean":"2024-09-16T10:32:32.471949+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:32.471949+0000","last_undegraded":"2024-09-16T10:32:32.471949+0000","last_fullsized":"2024-09-16T10:32:32.471949+0000","mapping_epoch":34,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":1466,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18996,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19003","reported_seq":19166,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:33.571892+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:33.571892+0000","last_peered":"2024-09-16T10:32:33.571892+0000","last_clean":"2024-09-16T10:32:33.571892+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:33.571892+0000","last_undegraded":"2024-09-16T10:32:33.571892+0000","last_fullsized":"2024-09-16T10:32:33.571892+0000","mapping_epoch":33,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6152192,"num_objects":1502,"num_object_clones":0,"num_object_copies":3004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15645,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18821","reported_seq":19157,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.613221+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:33.613221+0000","last_peered":"2024-09-16T10:32:33.613221+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:33.613221+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6348800,"num_objects":1550,"num_object_clones":0,"num_object_copies":3100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15446,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19454","reported_seq":19866,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.643045+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:33.643045+0000","last_peered":"2024-09-16T10:32:33.643045+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:33.643045+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":1466,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15696,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18953","reported_seq":19134,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:33.591823+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:33.591823+0000","last_peered":"2024-09-16T10:32:33.591823+0000","last_clean":"2024-09-16T10:32:33.591823+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:33.591823+0000","last_undegraded":"2024-09-16T10:32:33.591823+0000","last_fullsized":"2024-09-16T10:32:33.591823+0000","mapping_epoch":33,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5885952,"num_objects":1437,"num_object_clones":0,"num_object_copies":2874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14756,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'18937","reported_seq":27447,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.671608+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:32.671608+0000","last_peered":"2024-09-16T10:32:32.671608+0000","last_clean":"2024-09-16T10:32:32.671608+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:32.671608+0000","last_undegraded":"2024-09-16T10:32:32.671608+0000","last_fullsized":"2024-09-16T10:32:32.671608+0000","mapping_epoch":43,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6243270,"num_objects":1526,"num_object_clones":0,"num_object_copies":3052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18935,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19504","reported_seq":19783,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.604921+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:33.604921+0000","last_peered":"2024-09-16T10:32:33.604921+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:33.604921+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6292422,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19502,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2799}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'18883","reported_seq":19083,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:32.436574+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:32.436574+0000","last_peered":"2024-09-16T10:32:32.436574+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:32.436574+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6292422,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18881,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2695}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'18983","reported_seq":27471,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:32.457306+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:32.457306+0000","last_peered":"2024-09-16T10:32:32.457306+0000","last_clean":"2024-09-16T10:32:32.457306+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:32.457306+0000","last_undegraded":"2024-09-16T10:32:32.457306+0000","last_fullsized":"2024-09-16T10:32:32.457306+0000","mapping_epoch":47,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6525894,"num_objects":1595,"num_object_clones":0,"num_object_copies":3190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18981,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18568","reported_seq":18822,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.610825+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:33.610825+0000","last_peered":"2024-09-16T10:32:33.610825+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:33.610825+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6005219,"num_objects":1467,"num_object_clones":0,"num_object_copies":2934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18567,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19360","reported_seq":19640,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.686377+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:33.686377+0000","last_peered":"2024-09-16T10:32:33.686377+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:33.686377+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6366633,"num_objects":1557,"num_object_clones":0,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19357,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18660","reported_seq":21133,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.717397+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:33.717397+0000","last_peered":"2024-09-16T10:32:33.717397+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:33.717397+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6063046,"num_objects":1482,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18658,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.295217+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:28.295217+0000","last_peered":"2024-09-16T10:31:28.295217+0000","last_clean":"2024-09-16T10:31:28.295217+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:28.295217+0000","last_undegraded":"2024-09-16T10:31:28.295217+0000","last_fullsized":"2024-09-16T10:31:28.295217+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19355","reported_seq":19600,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.637284+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:33.637284+0000","last_peered":"2024-09-16T10:32:33.637284+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:33.637284+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6313868,"num_objects":1545,"num_object_clones":0,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19352,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2777}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":99119630,"num_objects":24216,"num_object_clones":0,"num_object_copies":48432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24216,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282089,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14396,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":409026560,"data_stored":408949769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160824,"ondisk_log_size":160824,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019001,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515440,"kb_used_data":186856,"kb_used_omap":4,"kb_used_meta":328571,"kb_avail":93856400,"statfs":{"total":96636764160,"available":96108953600,"internally_reserved":0,"allocated":191340544,"data_stored":191061898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":336457392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.90600000000000003}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.103},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":1.113}]}]},{"osd":1,"up_from":40,"seq":171798691945,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":511264,"kb_used_data":74384,"kb_used_omap":4,"kb_used_meta":436859,"kb_avail":93860576,"statfs":{"total":96636764160,"available":96113229824,"internally_reserved":0,"allocated":76169216,"data_stored":75832762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":447344298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":6.9130000000000003},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":6.9429999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.97699999999999998},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.93999999999999995}]}]},{"osd":0,"up_from":24,"seq":103079215228,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565000,"kb_used_data":140480,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93806840,"statfs":{"total":96636764160,"available":96058204160,"internally_reserved":0,"allocated":143851520,"data_stored":143580560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":434695848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":9.2699999999999996},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":9.2910000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":1.5900000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.073}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":142905344,"data_stored":142901731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75620352,"data_stored":75554859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190500864,"data_stored":190493179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:38.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:32:38.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'19086", 'reported_seq': 19342, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:32.486258+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:32.486258+0000', 'last_peered': '2024-09-16T10:32:32.486258+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:32.486258+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'9000", 'ondisk_log_start': "44'9000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6459416, 'num_objects': 1578, 'num_object_clones': 0, 'num_object_copies': 3156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1578, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15143, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2725}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:38.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:32:38.053 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'18821", 'reported_seq': 19157, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:33.613221+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:32:33.613221+0000', 'last_peered': '2024-09-16T10:32:33.613221+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:32:33.613221+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'8800", 'ondisk_log_start': "44'8800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6348800, 'num_objects': 1550, 'num_object_clones': 0, 'num_object_copies': 3100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15446, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2584}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:38.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:32:38.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'19454", 'reported_seq': 19866, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:33.643045+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:32:33.643045+0000', 'last_peered': '2024-09-16T10:32:33.643045+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:32:33.643045+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'9400", 'ondisk_log_start': "44'9400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6004736, 'num_objects': 1466, 'num_object_clones': 0, 'num_object_copies': 2932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15696, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2821}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:38.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:32:38.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'19504", 'reported_seq': 19783, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:33.604921+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:32:33.604921+0000', 'last_peered': '2024-09-16T10:32:33.604921+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:32:33.604921+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'9500", 'ondisk_log_start': "44'9500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6292422, 'num_objects': 1538, 'num_object_clones': 0, 'num_object_copies': 3076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19502, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2799}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:38.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:32:38.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'18883", 'reported_seq': 19083, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:32.436574+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:32.436574+0000', 'last_peered': '2024-09-16T10:32:32.436574+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:32.436574+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'8800", 'ondisk_log_start': "44'8800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6292422, 'num_objects': 1538, 'num_object_clones': 0, 'num_object_copies': 3076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18881, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2695}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:38.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:32:38.054 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'18568", 'reported_seq': 18822, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:33.610825+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:32:33.610825+0000', 'last_peered': '2024-09-16T10:32:33.610825+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:32:33.610825+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'8500", 'ondisk_log_start': "44'8500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6005219, 'num_objects': 1467, 'num_object_clones': 0, 'num_object_copies': 2934, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1467, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18567, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:38.055 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:32:38.055 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'19360", 'reported_seq': 19640, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:33.686377+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:32:33.686377+0000', 'last_peered': '2024-09-16T10:32:33.686377+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:32:33.686377+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'9300", 'ondisk_log_start': "44'9300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6366633, 'num_objects': 1557, 'num_object_clones': 0, 'num_object_copies': 3114, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19357, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2710}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:38.055 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:32:38.055 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'18660", 'reported_seq': 21133, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:33.717397+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:32:33.717397+0000', 'last_peered': '2024-09-16T10:32:33.717397+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:32:33.717397+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'8600", 'ondisk_log_start': "44'8600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6063046, 'num_objects': 1482, 'num_object_clones': 0, 'num_object_copies': 2964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2976, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18658, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1195, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1494}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:38.055 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:32:38.055 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'19355", 'reported_seq': 19600, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:33.637284+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:32:33.637284+0000', 'last_peered': '2024-09-16T10:32:33.637284+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:32:33.637284+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'9300", 'ondisk_log_start': "44'9300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6313868, 'num_objects': 1545, 'num_object_clones': 0, 'num_object_copies': 3090, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19352, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2777}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:38.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:32:38.739 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:38.740 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36950/48436 objects degraded (76.286%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":647,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":24218,"data_bytes":99119649,"bytes_used":1629904896,"bytes_avail":288280387584,"bytes_total":289910292480,"degraded_objects":36950,"degraded_total":48436,"degraded_ratio":0.76286233380130486,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":384},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:38.763 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:38.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:39.324 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:39.324 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:39.353 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-16T10:32:38.744754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95449633,"num_objects":23322,"num_object_clones":0,"num_object_copies":46644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23322,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282987,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14397,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1559608,"kb_used_data":390104,"kb_used_omap":12,"kb_used_meta":1169459,"kb_avail":281555912,"statfs":{"total":289910292480,"available":288313253888,"internally_reserved":0,"allocated":399466496,"data_stored":398580436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1197526018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,9,7,5,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22597632,"num_objects":-5517,"num_object_clones":0,"num_object_copies":-11034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003742"},"pg_stats":[{"pgid":"2.f","version":"48'19022","reported_seq":27490,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:33.697440+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:33.697440+0000","last_peered":"2024-09-16T10:32:33.697440+0000","last_clean":"2024-09-16T10:32:33.697440+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:33.697440+0000","last_undegraded":"2024-09-16T10:32:33.697440+0000","last_fullsized":"2024-09-16T10:32:33.697440+0000","mapping_epoch":45,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5922816,"num_objects":1446,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19022,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19072","reported_seq":19235,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:37.457184+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:37.457184+0000","last_peered":"2024-09-16T10:32:37.457184+0000","last_clean":"2024-09-16T10:32:37.457184+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:37.457184+0000","last_undegraded":"2024-09-16T10:32:37.457184+0000","last_fullsized":"2024-09-16T10:32:37.457184+0000","mapping_epoch":34,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5693440,"num_objects":1390,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15285,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19248","reported_seq":19505,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:37.496627+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:37.496627+0000","last_peered":"2024-09-16T10:32:37.496627+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:37.496627+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5795864,"num_objects":1416,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1416,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15305,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2887}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19145","reported_seq":19208,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:37.483781+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:37.483781+0000","last_peered":"2024-09-16T10:32:37.483781+0000","last_clean":"2024-09-16T10:32:37.483781+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:37.483781+0000","last_undegraded":"2024-09-16T10:32:37.483781+0000","last_fullsized":"2024-09-16T10:32:37.483781+0000","mapping_epoch":34,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5394432,"num_objects":1317,"num_object_clones":0,"num_object_copies":2634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19145,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19003","reported_seq":19166,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:33.571892+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:33.571892+0000","last_peered":"2024-09-16T10:32:33.571892+0000","last_clean":"2024-09-16T10:32:33.571892+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:33.571892+0000","last_undegraded":"2024-09-16T10:32:33.571892+0000","last_fullsized":"2024-09-16T10:32:33.571892+0000","mapping_epoch":33,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6152192,"num_objects":1502,"num_object_clones":0,"num_object_copies":3004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15645,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18821","reported_seq":19157,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.613221+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:33.613221+0000","last_peered":"2024-09-16T10:32:33.613221+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:33.613221+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6348800,"num_objects":1550,"num_object_clones":0,"num_object_copies":3100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15446,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2584}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19454","reported_seq":19866,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.643045+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:33.643045+0000","last_peered":"2024-09-16T10:32:33.643045+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:33.643045+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6004736,"num_objects":1466,"num_object_clones":0,"num_object_copies":2932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15696,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2821}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'18953","reported_seq":19134,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:33.591823+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:33.591823+0000","last_peered":"2024-09-16T10:32:33.591823+0000","last_clean":"2024-09-16T10:32:33.591823+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:33.591823+0000","last_undegraded":"2024-09-16T10:32:33.591823+0000","last_fullsized":"2024-09-16T10:32:33.591823+0000","mapping_epoch":33,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5885952,"num_objects":1437,"num_object_clones":0,"num_object_copies":2874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14756,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19091","reported_seq":27601,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:37.719547+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:37.719547+0000","last_peered":"2024-09-16T10:32:37.719547+0000","last_clean":"2024-09-16T10:32:37.719547+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:37.719547+0000","last_undegraded":"2024-09-16T10:32:37.719547+0000","last_fullsized":"2024-09-16T10:32:37.719547+0000","mapping_epoch":43,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5612486,"num_objects":1372,"num_object_clones":0,"num_object_copies":2744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19089,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19504","reported_seq":19783,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.604921+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:33.604921+0000","last_peered":"2024-09-16T10:32:33.604921+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:33.604921+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6292422,"num_objects":1538,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19502,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2799}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19032","reported_seq":19232,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:37.398773+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:37.398773+0000","last_peered":"2024-09-16T10:32:37.398773+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:37.398773+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5682118,"num_objects":1389,"num_object_clones":0,"num_object_copies":2778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19030,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2844}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19132","reported_seq":27620,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:37.300615+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:37.300615+0000","last_peered":"2024-09-16T10:32:37.300615+0000","last_clean":"2024-09-16T10:32:37.300615+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:37.300615+0000","last_undegraded":"2024-09-16T10:32:37.300615+0000","last_fullsized":"2024-09-16T10:32:37.300615+0000","mapping_epoch":47,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5915590,"num_objects":1446,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19130,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18568","reported_seq":18822,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.610825+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:33.610825+0000","last_peered":"2024-09-16T10:32:33.610825+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:33.610825+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8500","ondisk_log_start":"44'8500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6005219,"num_objects":1467,"num_object_clones":0,"num_object_copies":2934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18567,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19360","reported_seq":19640,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.686377+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:33.686377+0000","last_peered":"2024-09-16T10:32:33.686377+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:33.686377+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6366633,"num_objects":1557,"num_object_clones":0,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19357,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2710}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18660","reported_seq":21133,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.717397+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:33.717397+0000","last_peered":"2024-09-16T10:32:33.717397+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:33.717397+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8600","ondisk_log_start":"44'8600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6063046,"num_objects":1482,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18658,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1195,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1494}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19355","reported_seq":19600,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:33.637284+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:33.637284+0000","last_peered":"2024-09-16T10:32:33.637284+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:33.637284+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6313868,"num_objects":1545,"num_object_clones":0,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19352,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2777}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":95449614,"num_objects":23320,"num_object_clones":0,"num_object_copies":46640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36950,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23320,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":282985,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14396,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":397078528,"data_stored":397001737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160720,"ondisk_log_size":160720,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019002,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498552,"kb_used_data":182252,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93873288,"statfs":{"total":96636764160,"available":96126246912,"internally_reserved":0,"allocated":186626048,"data_stored":186347402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":323874480},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":2.2719999999999998},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":6.6920000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.8290000000000002},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.2389999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691946,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496056,"kb_used_data":67372,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93875784,"statfs":{"total":96636764160,"available":96128802816,"internally_reserved":0,"allocated":68988928,"data_stored":68652474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":438955690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":2.2109999999999999},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":2.1829999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":14.089},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":1.1799999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215228,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565000,"kb_used_data":140480,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93806840,"statfs":{"total":96636764160,"available":96058204160,"internally_reserved":0,"allocated":143851520,"data_stored":143580560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":434695848},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,3,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":9.2699999999999996},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":9.2910000000000004}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":1.5900000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.073}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":142905344,"data_stored":142901731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68321280,"data_stored":68255787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":185851904,"data_stored":185844219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:42.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:42.930 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:42.930 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:42.957 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-16T10:32:42.745689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":86598177,"num_objects":21161,"num_object_clones":0,"num_object_copies":42322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285148,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14483,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1596600,"kb_used_data":378712,"kb_used_omap":12,"kb_used_meta":1217843,"kb_avail":281518920,"statfs":{"total":289910292480,"available":288275374080,"internally_reserved":0,"allocated":387801088,"data_stored":386915028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1247071234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,6,12,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22016000,"num_objects":-5375,"num_object_clones":0,"num_object_copies":-10750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5375,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003755"},"pg_stats":[{"pgid":"2.f","version":"48'19151","reported_seq":27619,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:38.692004+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:38.692004+0000","last_peered":"2024-09-16T10:32:38.692004+0000","last_clean":"2024-09-16T10:32:38.692004+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:38.692004+0000","last_undegraded":"2024-09-16T10:32:38.692004+0000","last_fullsized":"2024-09-16T10:32:38.692004+0000","mapping_epoch":45,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5394432,"num_objects":1317,"num_object_clones":0,"num_object_copies":2634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19151,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19211","reported_seq":19374,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.472042+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:42.472042+0000","last_peered":"2024-09-16T10:32:42.472042+0000","last_clean":"2024-09-16T10:32:42.472042+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:42.472042+0000","last_undegraded":"2024-09-16T10:32:42.472042+0000","last_fullsized":"2024-09-16T10:32:42.472042+0000","mapping_epoch":34,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5124096,"num_objects":1251,"num_object_clones":0,"num_object_copies":2502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19380","reported_seq":19637,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:42.489007+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:42.489007+0000","last_peered":"2024-09-16T10:32:42.489007+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:42.489007+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5255192,"num_objects":1284,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15437,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19292","reported_seq":19355,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.444032+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:42.444032+0000","last_peered":"2024-09-16T10:32:42.444032+0000","last_clean":"2024-09-16T10:32:42.444032+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:42.444032+0000","last_undegraded":"2024-09-16T10:32:42.444032+0000","last_fullsized":"2024-09-16T10:32:42.444032+0000","mapping_epoch":34,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4792320,"num_objects":1170,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19292,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19147","reported_seq":19310,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:38.717504+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:38.717504+0000","last_peered":"2024-09-16T10:32:38.717504+0000","last_clean":"2024-09-16T10:32:38.717504+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:38.717504+0000","last_undegraded":"2024-09-16T10:32:38.717504+0000","last_fullsized":"2024-09-16T10:32:38.717504+0000","mapping_epoch":33,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5562368,"num_objects":1358,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15789,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18990","reported_seq":19326,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.671542+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:38.671542+0000","last_peered":"2024-09-16T10:32:38.671542+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:38.671542+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5656576,"num_objects":1381,"num_object_clones":0,"num_object_copies":2762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15615,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2753}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19611","reported_seq":20023,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.718281+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:38.718281+0000","last_peered":"2024-09-16T10:32:38.718281+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:38.718281+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5361664,"num_objects":1309,"num_object_clones":0,"num_object_copies":2618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15853,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2978}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19076","reported_seq":19258,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:38.709364+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:38.709364+0000","last_peered":"2024-09-16T10:32:38.709364+0000","last_clean":"2024-09-16T10:32:38.709364+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:38.709364+0000","last_undegraded":"2024-09-16T10:32:38.709364+0000","last_fullsized":"2024-09-16T10:32:38.709364+0000","mapping_epoch":33,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5382144,"num_objects":1314,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14879,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19091","reported_seq":27601,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:37.719547+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:37.719547+0000","last_peered":"2024-09-16T10:32:37.719547+0000","last_clean":"2024-09-16T10:32:37.719547+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:37.719547+0000","last_undegraded":"2024-09-16T10:32:37.719547+0000","last_fullsized":"2024-09-16T10:32:37.719547+0000","mapping_epoch":43,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5612486,"num_objects":1372,"num_object_clones":0,"num_object_copies":2744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19089,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19644","reported_seq":19923,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.609002+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:38.609002+0000","last_peered":"2024-09-16T10:32:38.609002+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:38.609002+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5718982,"num_objects":1398,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19642,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2939}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19171","reported_seq":19371,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:42.374944+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:42.374944+0000","last_peered":"2024-09-16T10:32:42.374944+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:42.374944+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5112774,"num_objects":1250,"num_object_clones":0,"num_object_copies":2500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19169,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2983}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19276","reported_seq":27762,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.445085+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:42.445085+0000","last_peered":"2024-09-16T10:32:42.445085+0000","last_clean":"2024-09-16T10:32:42.445085+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:42.445085+0000","last_undegraded":"2024-09-16T10:32:42.445085+0000","last_fullsized":"2024-09-16T10:32:42.445085+0000","mapping_epoch":47,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5325766,"num_objects":1302,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19274,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18722","reported_seq":18976,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.694113+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:38.694113+0000","last_peered":"2024-09-16T10:32:38.694113+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:38.694113+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5374435,"num_objects":1313,"num_object_clones":0,"num_object_copies":2626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18721,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19504","reported_seq":19784,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.672767+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:38.672767+0000","last_peered":"2024-09-16T10:32:38.672767+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:38.672767+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5776809,"num_objects":1413,"num_object_clones":0,"num_object_copies":2826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19501,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2854}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18810","reported_seq":21455,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.638013+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:38.638013+0000","last_peered":"2024-09-16T10:32:38.638013+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:38.638013+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5448646,"num_objects":1332,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18808,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19505","reported_seq":19750,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.745807+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:38.745807+0000","last_peered":"2024-09-16T10:32:38.745807+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:38.745807+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5699468,"num_objects":1395,"num_object_clones":0,"num_object_copies":2790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19502,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2927}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":86598158,"num_objects":21159,"num_object_clones":0,"num_object_copies":42318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285146,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14482,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385355776,"data_stored":385278985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019003,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516160,"kb_used_data":177720,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93855680,"statfs":{"total":96636764160,"available":96108216320,"internally_reserved":0,"allocated":181985280,"data_stored":181706634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":2.4990000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":19.518999999999998},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.4630000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691946,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496056,"kb_used_data":67372,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93875784,"statfs":{"total":96636764160,"available":96128802816,"internally_reserved":0,"allocated":68988928,"data_stored":68652474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":438955690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":2.2109999999999999},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":2.1829999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":14.089},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":1.1799999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215229,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584384,"kb_used_data":133620,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93787456,"statfs":{"total":96636764160,"available":96038354944,"internally_reserved":0,"allocated":136826880,"data_stored":136555920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":461565608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":1.4179999999999999},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.373}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":1.363},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.3819999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135819264,"data_stored":135815651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68321280,"data_stored":68255787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":181215232,"data_stored":181207547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:42.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:43.527 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:43.527 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:43.552 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-16T10:32:42.745689+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":86598177,"num_objects":21161,"num_object_clones":0,"num_object_copies":42322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21161,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285148,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14483,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1596600,"kb_used_data":378712,"kb_used_omap":12,"kb_used_meta":1217843,"kb_avail":281518920,"statfs":{"total":289910292480,"available":288275374080,"internally_reserved":0,"allocated":387801088,"data_stored":386915028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1247071234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,6,12,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22016000,"num_objects":-5375,"num_object_clones":0,"num_object_copies":-10750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5375,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003755"},"pg_stats":[{"pgid":"2.f","version":"48'19151","reported_seq":27619,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:38.692004+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:38.692004+0000","last_peered":"2024-09-16T10:32:38.692004+0000","last_clean":"2024-09-16T10:32:38.692004+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:38.692004+0000","last_undegraded":"2024-09-16T10:32:38.692004+0000","last_fullsized":"2024-09-16T10:32:38.692004+0000","mapping_epoch":45,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5394432,"num_objects":1317,"num_object_clones":0,"num_object_copies":2634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19151,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19211","reported_seq":19374,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.472042+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:42.472042+0000","last_peered":"2024-09-16T10:32:42.472042+0000","last_clean":"2024-09-16T10:32:42.472042+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:42.472042+0000","last_undegraded":"2024-09-16T10:32:42.472042+0000","last_fullsized":"2024-09-16T10:32:42.472042+0000","mapping_epoch":34,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5124096,"num_objects":1251,"num_object_clones":0,"num_object_copies":2502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19380","reported_seq":19637,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:42.489007+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:42.489007+0000","last_peered":"2024-09-16T10:32:42.489007+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:42.489007+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5255192,"num_objects":1284,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15437,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19292","reported_seq":19355,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.444032+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:42.444032+0000","last_peered":"2024-09-16T10:32:42.444032+0000","last_clean":"2024-09-16T10:32:42.444032+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:42.444032+0000","last_undegraded":"2024-09-16T10:32:42.444032+0000","last_fullsized":"2024-09-16T10:32:42.444032+0000","mapping_epoch":34,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4792320,"num_objects":1170,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19292,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19147","reported_seq":19310,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:38.717504+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:38.717504+0000","last_peered":"2024-09-16T10:32:38.717504+0000","last_clean":"2024-09-16T10:32:38.717504+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:38.717504+0000","last_undegraded":"2024-09-16T10:32:38.717504+0000","last_fullsized":"2024-09-16T10:32:38.717504+0000","mapping_epoch":33,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5562368,"num_objects":1358,"num_object_clones":0,"num_object_copies":2716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15789,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'18990","reported_seq":19326,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.671542+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:38.671542+0000","last_peered":"2024-09-16T10:32:38.671542+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:38.671542+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5656576,"num_objects":1381,"num_object_clones":0,"num_object_copies":2762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15615,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2753}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19611","reported_seq":20023,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.718281+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:38.718281+0000","last_peered":"2024-09-16T10:32:38.718281+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:38.718281+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5361664,"num_objects":1309,"num_object_clones":0,"num_object_copies":2618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15853,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2978}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19076","reported_seq":19258,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:38.709364+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:38.709364+0000","last_peered":"2024-09-16T10:32:38.709364+0000","last_clean":"2024-09-16T10:32:38.709364+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:38.709364+0000","last_undegraded":"2024-09-16T10:32:38.709364+0000","last_fullsized":"2024-09-16T10:32:38.709364+0000","mapping_epoch":33,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5382144,"num_objects":1314,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14879,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19091","reported_seq":27601,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:37.719547+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:37.719547+0000","last_peered":"2024-09-16T10:32:37.719547+0000","last_clean":"2024-09-16T10:32:37.719547+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:37.719547+0000","last_undegraded":"2024-09-16T10:32:37.719547+0000","last_fullsized":"2024-09-16T10:32:37.719547+0000","mapping_epoch":43,"log_start":"44'9000","ondisk_log_start":"44'9000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5612486,"num_objects":1372,"num_object_clones":0,"num_object_copies":2744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19089,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19644","reported_seq":19923,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.609002+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:38.609002+0000","last_peered":"2024-09-16T10:32:38.609002+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:38.609002+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5718982,"num_objects":1398,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19642,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2939}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19171","reported_seq":19371,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:42.374944+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:42.374944+0000","last_peered":"2024-09-16T10:32:42.374944+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:42.374944+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5112774,"num_objects":1250,"num_object_clones":0,"num_object_copies":2500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19169,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2983}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19276","reported_seq":27762,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.445085+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:42.445085+0000","last_peered":"2024-09-16T10:32:42.445085+0000","last_clean":"2024-09-16T10:32:42.445085+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:42.445085+0000","last_undegraded":"2024-09-16T10:32:42.445085+0000","last_fullsized":"2024-09-16T10:32:42.445085+0000","mapping_epoch":47,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5325766,"num_objects":1302,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19274,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18722","reported_seq":18976,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.694113+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:38.694113+0000","last_peered":"2024-09-16T10:32:38.694113+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:38.694113+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8700","ondisk_log_start":"44'8700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5374435,"num_objects":1313,"num_object_clones":0,"num_object_copies":2626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18721,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2762}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19504","reported_seq":19784,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.672767+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:38.672767+0000","last_peered":"2024-09-16T10:32:38.672767+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:38.672767+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5776809,"num_objects":1413,"num_object_clones":0,"num_object_copies":2826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19501,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2854}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18810","reported_seq":21455,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.638013+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:38.638013+0000","last_peered":"2024-09-16T10:32:38.638013+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:38.638013+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5448646,"num_objects":1332,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18808,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19505","reported_seq":19750,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:38.745807+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:38.745807+0000","last_peered":"2024-09-16T10:32:38.745807+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:38.745807+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5699468,"num_objects":1395,"num_object_clones":0,"num_object_copies":2790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19502,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2927}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":86598158,"num_objects":21159,"num_object_clones":0,"num_object_copies":42318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285146,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14482,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385355776,"data_stored":385278985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160781,"ondisk_log_size":160781,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019003,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516160,"kb_used_data":177720,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93855680,"statfs":{"total":96636764160,"available":96108216320,"internally_reserved":0,"allocated":181985280,"data_stored":181706634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":2.4990000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":19.518999999999998},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.4630000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691946,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496056,"kb_used_data":67372,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93875784,"statfs":{"total":96636764160,"available":96128802816,"internally_reserved":0,"allocated":68988928,"data_stored":68652474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":438955690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":2.2109999999999999},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":2.1829999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":14.089},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":1.1799999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215229,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584384,"kb_used_data":133620,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93787456,"statfs":{"total":96636764160,"available":96038354944,"internally_reserved":0,"allocated":136826880,"data_stored":136555920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":461565608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,7,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":1.4179999999999999},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.373}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":1.363},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.3819999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135819264,"data_stored":135815651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":68321280,"data_stored":68255787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":181215232,"data_stored":181207547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:43.554 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:32:43.554 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'19380", 'reported_seq': 19637, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:42.489007+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:42.489007+0000', 'last_peered': '2024-09-16T10:32:42.489007+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:42.489007+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'9300", 'ondisk_log_start': "44'9300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5255192, 'num_objects': 1284, 'num_object_clones': 0, 'num_object_copies': 2568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1284, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15437, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3019}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:43.554 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:32:43.554 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'18990", 'reported_seq': 19326, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:38.671542+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:32:38.671542+0000', 'last_peered': '2024-09-16T10:32:38.671542+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:32:38.671542+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'8900", 'ondisk_log_start': "44'8900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5656576, 'num_objects': 1381, 'num_object_clones': 0, 'num_object_copies': 2762, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15615, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2753}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:43.554 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:32:43.554 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'19611", 'reported_seq': 20023, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:38.718281+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:32:38.718281+0000', 'last_peered': '2024-09-16T10:32:38.718281+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:32:38.718281+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'9600", 'ondisk_log_start': "44'9600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5361664, 'num_objects': 1309, 'num_object_clones': 0, 'num_object_copies': 2618, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15853, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2978}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:43.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:32:43.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'19644", 'reported_seq': 19923, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:38.609002+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:32:38.609002+0000', 'last_peered': '2024-09-16T10:32:38.609002+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:32:38.609002+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'9600", 'ondisk_log_start': "44'9600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5718982, 'num_objects': 1398, 'num_object_clones': 0, 'num_object_copies': 2796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19642, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2939}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:43.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:32:43.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'19171", 'reported_seq': 19371, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:42.374944+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:42.374944+0000', 'last_peered': '2024-09-16T10:32:42.374944+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:42.374944+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'9100", 'ondisk_log_start': "44'9100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5112774, 'num_objects': 1250, 'num_object_clones': 0, 'num_object_copies': 2500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19169, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2983}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:43.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:32:43.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'18722", 'reported_seq': 18976, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:38.694113+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:32:38.694113+0000', 'last_peered': '2024-09-16T10:32:38.694113+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:32:38.694113+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'8700", 'ondisk_log_start': "44'8700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5374435, 'num_objects': 1313, 'num_object_clones': 0, 'num_object_copies': 2626, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18721, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2762}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:43.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:32:43.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'19504", 'reported_seq': 19784, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:38.672767+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:32:38.672767+0000', 'last_peered': '2024-09-16T10:32:38.672767+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:32:38.672767+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'9500", 'ondisk_log_start': "44'9500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5776809, 'num_objects': 1413, 'num_object_clones': 0, 'num_object_copies': 2826, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19501, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2854}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:43.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:32:43.555 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'18810", 'reported_seq': 21455, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:38.638013+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:32:38.638013+0000', 'last_peered': '2024-09-16T10:32:38.638013+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:32:38.638013+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'8800", 'ondisk_log_start': "44'8800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5448646, 'num_objects': 1332, 'num_object_clones': 0, 'num_object_copies': 2664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2890, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18808, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1558}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:43.556 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:32:43.556 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'19505", 'reported_seq': 19750, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:38.745807+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:32:38.745807+0000', 'last_peered': '2024-09-16T10:32:38.745807+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:32:38.745807+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'9500", 'ondisk_log_start': "44'9500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5699468, 'num_objects': 1395, 'num_object_clones': 0, 'num_object_copies': 2790, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19502, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2927}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:43.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:32:44.278 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:44.279 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36864/42322 objects degraded (87.104%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":653,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":21161,"data_bytes":86598177,"bytes_used":1634918400,"bytes_avail":288275374080,"bytes_total":289910292480,"degraded_objects":36864,"degraded_total":42322,"degraded_ratio":0.87103634043759759,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":169,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":447},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:44.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:44.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:44.904 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:44.907 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:44.933 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-09-16T10:32:44.746148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80024097,"num_objects":19556,"num_object_clones":0,"num_object_copies":39112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19556,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":286753,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14567,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160889,"ondisk_log_size":160889,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1590872,"kb_used_data":364784,"kb_used_omap":12,"kb_used_meta":1226035,"kb_avail":281524648,"statfs":{"total":289910292480,"available":288281239552,"internally_reserved":0,"allocated":373538816,"data_stored":372652756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1255459842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,7,7,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25550848,"num_objects":-6238,"num_object_clones":0,"num_object_copies":-12476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6238,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003773"},"pg_stats":[{"pgid":"2.f","version":"48'19293","reported_seq":27761,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.768816+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:43.768816+0000","last_peered":"2024-09-16T10:32:43.768816+0000","last_clean":"2024-09-16T10:32:43.768816+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:43.768816+0000","last_undegraded":"2024-09-16T10:32:43.768816+0000","last_fullsized":"2024-09-16T10:32:43.768816+0000","mapping_epoch":45,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4812800,"num_objects":1175,"num_object_clones":0,"num_object_copies":2350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19293,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19211","reported_seq":19374,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.472042+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:42.472042+0000","last_peered":"2024-09-16T10:32:42.472042+0000","last_clean":"2024-09-16T10:32:42.472042+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:42.472042+0000","last_undegraded":"2024-09-16T10:32:42.472042+0000","last_fullsized":"2024-09-16T10:32:42.472042+0000","mapping_epoch":34,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5124096,"num_objects":1251,"num_object_clones":0,"num_object_copies":2502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19380","reported_seq":19637,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:42.489007+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:42.489007+0000","last_peered":"2024-09-16T10:32:42.489007+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:42.489007+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5255192,"num_objects":1284,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15437,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19292","reported_seq":19355,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.444032+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:42.444032+0000","last_peered":"2024-09-16T10:32:42.444032+0000","last_clean":"2024-09-16T10:32:42.444032+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:42.444032+0000","last_undegraded":"2024-09-16T10:32:42.444032+0000","last_fullsized":"2024-09-16T10:32:42.444032+0000","mapping_epoch":34,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4792320,"num_objects":1170,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19292,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19300","reported_seq":19463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.787347+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:43.787347+0000","last_peered":"2024-09-16T10:32:43.787347+0000","last_clean":"2024-09-16T10:32:43.787347+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:43.787347+0000","last_undegraded":"2024-09-16T10:32:43.787347+0000","last_fullsized":"2024-09-16T10:32:43.787347+0000","mapping_epoch":33,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4935680,"num_objects":1205,"num_object_clones":0,"num_object_copies":2410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15942,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19133","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.628579+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:43.628579+0000","last_peered":"2024-09-16T10:32:43.628579+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:43.628579+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5070848,"num_objects":1238,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15758,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19754","reported_seq":20166,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.703166+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:43.703166+0000","last_peered":"2024-09-16T10:32:43.703166+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:43.703166+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4775936,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15996,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19210","reported_seq":19392,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.675051+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:43.675051+0000","last_peered":"2024-09-16T10:32:43.675051+0000","last_clean":"2024-09-16T10:32:43.675051+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:43.675051+0000","last_undegraded":"2024-09-16T10:32:43.675051+0000","last_fullsized":"2024-09-16T10:32:43.675051+0000","mapping_epoch":33,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4833280,"num_objects":1180,"num_object_clones":0,"num_object_copies":2360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15013,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19227","reported_seq":27737,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.882144+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:42.882144+0000","last_peered":"2024-09-16T10:32:42.882144+0000","last_clean":"2024-09-16T10:32:42.882144+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:42.882144+0000","last_undegraded":"2024-09-16T10:32:42.882144+0000","last_fullsized":"2024-09-16T10:32:42.882144+0000","mapping_epoch":43,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5055430,"num_objects":1236,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19225,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19802","reported_seq":20081,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.705058+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:43.705058+0000","last_peered":"2024-09-16T10:32:43.705058+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:43.705058+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5071814,"num_objects":1240,"num_object_clones":0,"num_object_copies":2480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19800,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3097}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19171","reported_seq":19371,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:42.374944+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:42.374944+0000","last_peered":"2024-09-16T10:32:42.374944+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:42.374944+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5112774,"num_objects":1250,"num_object_clones":0,"num_object_copies":2500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19169,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2983}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19276","reported_seq":27762,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.445085+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:42.445085+0000","last_peered":"2024-09-16T10:32:42.445085+0000","last_clean":"2024-09-16T10:32:42.445085+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:42.445085+0000","last_undegraded":"2024-09-16T10:32:42.445085+0000","last_fullsized":"2024-09-16T10:32:42.445085+0000","mapping_epoch":47,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5325766,"num_objects":1302,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19274,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18863","reported_seq":19117,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.769958+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:43.769958+0000","last_peered":"2024-09-16T10:32:43.769958+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:43.769958+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4796899,"num_objects":1172,"num_object_clones":0,"num_object_copies":2344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18862,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19644","reported_seq":19924,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.796043+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:43.796043+0000","last_peered":"2024-09-16T10:32:43.796043+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:43.796043+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5203369,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19641,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2994}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18964","reported_seq":21776,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.824357+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:43.824357+0000","last_peered":"2024-09-16T10:32:43.824357+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:43.824357+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4817862,"num_objects":1178,"num_object_clones":0,"num_object_copies":2356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18962,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19666","reported_seq":19911,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.768732+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:43.768732+0000","last_peered":"2024-09-16T10:32:43.768732+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:43.768732+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5040012,"num_objects":1234,"num_object_clones":0,"num_object_copies":2468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19663,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":80024078,"num_objects":19554,"num_object_clones":0,"num_object_copies":39108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":286751,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14566,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":370835456,"data_stored":370758665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019003,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516160,"kb_used_data":177720,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93855680,"statfs":{"total":96636764160,"available":96108216320,"internally_reserved":0,"allocated":181985280,"data_stored":181706634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":2.4990000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":19.518999999999998},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.4630000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691947,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493132,"kb_used_data":60352,"kb_used_omap":4,"kb_used_meta":432763,"kb_avail":93878708,"statfs":{"total":96636764160,"available":96131796992,"internally_reserved":0,"allocated":61800448,"data_stored":61463994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":443149994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":7.5469999999999997},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":7.5060000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.624},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.66100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215230,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581580,"kb_used_data":126712,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93790260,"statfs":{"total":96636764160,"available":96041226240,"internally_reserved":0,"allocated":129753088,"data_stored":129482128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.34799999999999998},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128532480,"data_stored":128528867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61087744,"data_stored":61022251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":181215232,"data_stored":181207547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:47.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:48.539 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:48.542 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:48.569 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-16T10:32:46.746615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80024097,"num_objects":19556,"num_object_clones":0,"num_object_copies":39112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19556,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":286753,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14567,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160889,"ondisk_log_size":160889,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1590872,"kb_used_data":364784,"kb_used_omap":12,"kb_used_meta":1226035,"kb_avail":281524648,"statfs":{"total":289910292480,"available":288281239552,"internally_reserved":0,"allocated":373538816,"data_stored":372652756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1255459842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,7,7,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19095552,"num_objects":-4662,"num_object_clones":0,"num_object_copies":-9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4662,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002818"},"pg_stats":[{"pgid":"2.f","version":"48'19293","reported_seq":27761,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.768816+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:43.768816+0000","last_peered":"2024-09-16T10:32:43.768816+0000","last_clean":"2024-09-16T10:32:43.768816+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:43.768816+0000","last_undegraded":"2024-09-16T10:32:43.768816+0000","last_fullsized":"2024-09-16T10:32:43.768816+0000","mapping_epoch":45,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4812800,"num_objects":1175,"num_object_clones":0,"num_object_copies":2350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19293,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19211","reported_seq":19374,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.472042+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:42.472042+0000","last_peered":"2024-09-16T10:32:42.472042+0000","last_clean":"2024-09-16T10:32:42.472042+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:42.472042+0000","last_undegraded":"2024-09-16T10:32:42.472042+0000","last_fullsized":"2024-09-16T10:32:42.472042+0000","mapping_epoch":34,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5124096,"num_objects":1251,"num_object_clones":0,"num_object_copies":2502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15424,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19380","reported_seq":19637,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:42.489007+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:42.489007+0000","last_peered":"2024-09-16T10:32:42.489007+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:42.489007+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5255192,"num_objects":1284,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15437,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3019}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19292","reported_seq":19355,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.444032+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:42.444032+0000","last_peered":"2024-09-16T10:32:42.444032+0000","last_clean":"2024-09-16T10:32:42.444032+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:42.444032+0000","last_undegraded":"2024-09-16T10:32:42.444032+0000","last_fullsized":"2024-09-16T10:32:42.444032+0000","mapping_epoch":34,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4792320,"num_objects":1170,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19292,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19300","reported_seq":19463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.787347+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:43.787347+0000","last_peered":"2024-09-16T10:32:43.787347+0000","last_clean":"2024-09-16T10:32:43.787347+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:43.787347+0000","last_undegraded":"2024-09-16T10:32:43.787347+0000","last_fullsized":"2024-09-16T10:32:43.787347+0000","mapping_epoch":33,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4935680,"num_objects":1205,"num_object_clones":0,"num_object_copies":2410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15942,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19133","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.628579+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:43.628579+0000","last_peered":"2024-09-16T10:32:43.628579+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:43.628579+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5070848,"num_objects":1238,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15758,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19754","reported_seq":20166,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.703166+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:43.703166+0000","last_peered":"2024-09-16T10:32:43.703166+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:43.703166+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4775936,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15996,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19210","reported_seq":19392,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.675051+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:43.675051+0000","last_peered":"2024-09-16T10:32:43.675051+0000","last_clean":"2024-09-16T10:32:43.675051+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:43.675051+0000","last_undegraded":"2024-09-16T10:32:43.675051+0000","last_fullsized":"2024-09-16T10:32:43.675051+0000","mapping_epoch":33,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4833280,"num_objects":1180,"num_object_clones":0,"num_object_copies":2360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15013,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19227","reported_seq":27737,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.882144+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:42.882144+0000","last_peered":"2024-09-16T10:32:42.882144+0000","last_clean":"2024-09-16T10:32:42.882144+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:42.882144+0000","last_undegraded":"2024-09-16T10:32:42.882144+0000","last_fullsized":"2024-09-16T10:32:42.882144+0000","mapping_epoch":43,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5055430,"num_objects":1236,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19225,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19802","reported_seq":20081,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.705058+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:43.705058+0000","last_peered":"2024-09-16T10:32:43.705058+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:43.705058+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5071814,"num_objects":1240,"num_object_clones":0,"num_object_copies":2480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19800,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3097}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19171","reported_seq":19371,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:42.374944+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:42.374944+0000","last_peered":"2024-09-16T10:32:42.374944+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:42.374944+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5112774,"num_objects":1250,"num_object_clones":0,"num_object_copies":2500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19169,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2983}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19276","reported_seq":27762,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:42.445085+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:42.445085+0000","last_peered":"2024-09-16T10:32:42.445085+0000","last_clean":"2024-09-16T10:32:42.445085+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:42.445085+0000","last_undegraded":"2024-09-16T10:32:42.445085+0000","last_fullsized":"2024-09-16T10:32:42.445085+0000","mapping_epoch":47,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5325766,"num_objects":1302,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19274,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18863","reported_seq":19117,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.769958+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:43.769958+0000","last_peered":"2024-09-16T10:32:43.769958+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:43.769958+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4796899,"num_objects":1172,"num_object_clones":0,"num_object_copies":2344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18862,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19644","reported_seq":19924,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.796043+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:43.796043+0000","last_peered":"2024-09-16T10:32:43.796043+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:43.796043+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5203369,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19641,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2994}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18964","reported_seq":21776,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.824357+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:43.824357+0000","last_peered":"2024-09-16T10:32:43.824357+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:43.824357+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4817862,"num_objects":1178,"num_object_clones":0,"num_object_copies":2356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18962,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19666","reported_seq":19911,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.768732+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:43.768732+0000","last_peered":"2024-09-16T10:32:43.768732+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:43.768732+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5040012,"num_objects":1234,"num_object_clones":0,"num_object_copies":2468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19663,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":80024078,"num_objects":19554,"num_object_clones":0,"num_object_copies":39108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":286751,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14566,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":370835456,"data_stored":370758665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160886,"ondisk_log_size":160886,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019003,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":516160,"kb_used_data":177720,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93855680,"statfs":{"total":96636764160,"available":96108216320,"internally_reserved":0,"allocated":181985280,"data_stored":181706634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":2.4990000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":19.518999999999998},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.4630000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691947,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":493132,"kb_used_data":60352,"kb_used_omap":4,"kb_used_meta":432763,"kb_avail":93878708,"statfs":{"total":96636764160,"available":96131796992,"internally_reserved":0,"allocated":61800448,"data_stored":61463994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":443149994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":7.5469999999999997},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":7.5060000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.624},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.66100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215230,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581580,"kb_used_data":126712,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93790260,"statfs":{"total":96636764160,"available":96041226240,"internally_reserved":0,"allocated":129753088,"data_stored":129482128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.34799999999999998},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128532480,"data_stored":128528867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61087744,"data_stored":61022251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":181215232,"data_stored":181207547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:48.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:49.158 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:49.158 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:49.186 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-16T10:32:48.747039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76501537,"num_objects":18696,"num_object_clones":0,"num_object_copies":37392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18696,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287613,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14567,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1575184,"kb_used_data":353192,"kb_used_omap":12,"kb_used_meta":1221939,"kb_avail":281540336,"statfs":{"total":289910292480,"available":288297304064,"internally_reserved":0,"allocated":361668608,"data_stored":360782548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1251265538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,10,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22618112,"num_objects":-5522,"num_object_clones":0,"num_object_copies":-11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5522,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002717"},"pg_stats":[{"pgid":"2.f","version":"48'19293","reported_seq":27761,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.768816+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:43.768816+0000","last_peered":"2024-09-16T10:32:43.768816+0000","last_clean":"2024-09-16T10:32:43.768816+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:43.768816+0000","last_undegraded":"2024-09-16T10:32:43.768816+0000","last_fullsized":"2024-09-16T10:32:43.768816+0000","mapping_epoch":45,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4812800,"num_objects":1175,"num_object_clones":0,"num_object_copies":2350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19293,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19372","reported_seq":19536,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:47.507790+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:47.507790+0000","last_peered":"2024-09-16T10:32:47.507790+0000","last_clean":"2024-09-16T10:32:47.507790+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:47.507790+0000","last_undegraded":"2024-09-16T10:32:47.507790+0000","last_fullsized":"2024-09-16T10:32:47.507790+0000","mapping_epoch":34,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4464640,"num_objects":1090,"num_object_clones":0,"num_object_copies":2180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15585,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19513","reported_seq":19770,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:47.435474+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:47.435474+0000","last_peered":"2024-09-16T10:32:47.435474+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:47.435474+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4710424,"num_objects":1151,"num_object_clones":0,"num_object_copies":2302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15570,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3152}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19413","reported_seq":19476,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:47.426336+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:47.426336+0000","last_peered":"2024-09-16T10:32:47.426336+0000","last_clean":"2024-09-16T10:32:47.426336+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:47.426336+0000","last_undegraded":"2024-09-16T10:32:47.426336+0000","last_fullsized":"2024-09-16T10:32:47.426336+0000","mapping_epoch":34,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4296704,"num_objects":1049,"num_object_clones":0,"num_object_copies":2098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19413,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19300","reported_seq":19463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.787347+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:43.787347+0000","last_peered":"2024-09-16T10:32:43.787347+0000","last_clean":"2024-09-16T10:32:43.787347+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:43.787347+0000","last_undegraded":"2024-09-16T10:32:43.787347+0000","last_fullsized":"2024-09-16T10:32:43.787347+0000","mapping_epoch":33,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4935680,"num_objects":1205,"num_object_clones":0,"num_object_copies":2410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15942,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19133","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.628579+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:43.628579+0000","last_peered":"2024-09-16T10:32:43.628579+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:43.628579+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5070848,"num_objects":1238,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15758,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19754","reported_seq":20166,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.703166+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:43.703166+0000","last_peered":"2024-09-16T10:32:43.703166+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:43.703166+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4775936,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15996,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19210","reported_seq":19392,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.675051+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:43.675051+0000","last_peered":"2024-09-16T10:32:43.675051+0000","last_clean":"2024-09-16T10:32:43.675051+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:43.675051+0000","last_undegraded":"2024-09-16T10:32:43.675051+0000","last_fullsized":"2024-09-16T10:32:43.675051+0000","mapping_epoch":33,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4833280,"num_objects":1180,"num_object_clones":0,"num_object_copies":2360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15013,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19383","reported_seq":27893,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:47.883732+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:47.883732+0000","last_peered":"2024-09-16T10:32:47.883732+0000","last_clean":"2024-09-16T10:32:47.883732+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:47.883732+0000","last_undegraded":"2024-09-16T10:32:47.883732+0000","last_fullsized":"2024-09-16T10:32:47.883732+0000","mapping_epoch":43,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4416454,"num_objects":1080,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19381,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19802","reported_seq":20081,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.705058+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:43.705058+0000","last_peered":"2024-09-16T10:32:43.705058+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:43.705058+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5071814,"num_objects":1240,"num_object_clones":0,"num_object_copies":2480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19800,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3097}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19320","reported_seq":19520,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:47.420984+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:47.420984+0000","last_peered":"2024-09-16T10:32:47.420984+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:47.420984+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4502470,"num_objects":1101,"num_object_clones":0,"num_object_copies":2202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19318,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3132}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19416","reported_seq":27903,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:47.468852+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:47.468852+0000","last_peered":"2024-09-16T10:32:47.468852+0000","last_clean":"2024-09-16T10:32:47.468852+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:47.468852+0000","last_undegraded":"2024-09-16T10:32:47.468852+0000","last_fullsized":"2024-09-16T10:32:47.468852+0000","mapping_epoch":47,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4752326,"num_objects":1162,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19414,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18863","reported_seq":19117,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.769958+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:43.769958+0000","last_peered":"2024-09-16T10:32:43.769958+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:43.769958+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4796899,"num_objects":1172,"num_object_clones":0,"num_object_copies":2344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18862,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19644","reported_seq":19924,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.796043+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:43.796043+0000","last_peered":"2024-09-16T10:32:43.796043+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:43.796043+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5203369,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19641,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2994}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18964","reported_seq":21776,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.824357+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:43.824357+0000","last_peered":"2024-09-16T10:32:43.824357+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:43.824357+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4817862,"num_objects":1178,"num_object_clones":0,"num_object_copies":2356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18962,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19666","reported_seq":19911,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.768732+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:43.768732+0000","last_peered":"2024-09-16T10:32:43.768732+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:43.768732+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5040012,"num_objects":1234,"num_object_clones":0,"num_object_copies":2468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19663,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":76501518,"num_objects":18694,"num_object_clones":0,"num_object_copies":37388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287611,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14566,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358858752,"data_stored":358781961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160746,"ondisk_log_size":160746,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515692,"kb_used_data":173156,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93856148,"statfs":{"total":96636764160,"available":96108695552,"internally_reserved":0,"allocated":177311744,"data_stored":177033098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":2.3460000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":6.9660000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.9960000000000004},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":4.141}]}]},{"osd":1,"up_from":40,"seq":171798691948,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477912,"kb_used_data":53324,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93893928,"statfs":{"total":96636764160,"available":96147382272,"internally_reserved":0,"allocated":54603776,"data_stored":54267322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":434761386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":1.24},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":2.1190000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":3.2599999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215230,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581580,"kb_used_data":126712,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93790260,"statfs":{"total":96636764160,"available":96041226240,"internally_reserved":0,"allocated":129753088,"data_stored":129482128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.34799999999999998},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128532480,"data_stored":128528867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53796864,"data_stored":53731371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176529408,"data_stored":176521723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:49.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:32:49.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'19513", 'reported_seq': 19770, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:47.435474+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:47.435474+0000', 'last_peered': '2024-09-16T10:32:47.435474+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:47.435474+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'9500", 'ondisk_log_start': "44'9500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4710424, 'num_objects': 1151, 'num_object_clones': 0, 'num_object_copies': 2302, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1151, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15570, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3152}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:49.187 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'19133", 'reported_seq': 19469, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:43.628579+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:32:43.628579+0000', 'last_peered': '2024-09-16T10:32:43.628579+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:32:43.628579+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'9100", 'ondisk_log_start': "44'9100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5070848, 'num_objects': 1238, 'num_object_clones': 0, 'num_object_copies': 2476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1238, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15758, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2896}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'19754", 'reported_seq': 20166, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:43.703166+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:32:43.703166+0000', 'last_peered': '2024-09-16T10:32:43.703166+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:32:43.703166+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'9700", 'ondisk_log_start': "44'9700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4775936, 'num_objects': 1166, 'num_object_clones': 0, 'num_object_copies': 2332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15996, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'19802", 'reported_seq': 20081, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:43.705058+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:32:43.705058+0000', 'last_peered': '2024-09-16T10:32:43.705058+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:32:43.705058+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'9800", 'ondisk_log_start': "44'9800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5071814, 'num_objects': 1240, 'num_object_clones': 0, 'num_object_copies': 2480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19800, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3097}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'19320", 'reported_seq': 19520, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:47.420984+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:47.420984+0000', 'last_peered': '2024-09-16T10:32:47.420984+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:47.420984+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'9300", 'ondisk_log_start': "44'9300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4502470, 'num_objects': 1101, 'num_object_clones': 0, 'num_object_copies': 2202, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19318, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3132}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'18863", 'reported_seq': 19117, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:43.769958+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:32:43.769958+0000', 'last_peered': '2024-09-16T10:32:43.769958+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:32:43.769958+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'8800", 'ondisk_log_start': "44'8800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4796899, 'num_objects': 1172, 'num_object_clones': 0, 'num_object_copies': 2344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18862, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2903}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:32:49.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'19644", 'reported_seq': 19924, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:43.796043+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:32:43.796043+0000', 'last_peered': '2024-09-16T10:32:43.796043+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:32:43.796043+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'9600", 'ondisk_log_start': "44'9600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5203369, 'num_objects': 1273, 'num_object_clones': 0, 'num_object_copies': 2546, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19641, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2994}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:49.189 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:32:49.189 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'18964", 'reported_seq': 21776, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:43.824357+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:32:43.824357+0000', 'last_peered': '2024-09-16T10:32:43.824357+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:32:43.824357+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'8900", 'ondisk_log_start': "44'8900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4817862, 'num_objects': 1178, 'num_object_clones': 0, 'num_object_copies': 2356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2806, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18962, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1365, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1628}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:49.189 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:32:49.189 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'19666", 'reported_seq': 19911, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:43.768732+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:32:43.768732+0000', 'last_peered': '2024-09-16T10:32:43.768732+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:32:43.768732+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'9600", 'ondisk_log_start': "44'9600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5040012, 'num_objects': 1234, 'num_object_clones': 0, 'num_object_copies': 2468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19663, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3088}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:49.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:32:49.868 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:49.869 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36780/37392 objects degraded (98.363%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":658,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":18696,"data_bytes":76501537,"bytes_used":1612988416,"bytes_avail":288297304064,"bytes_total":289910292480,"degraded_objects":36780,"degraded_total":37392,"degraded_ratio":0.98363286264441596,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":460},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:49.894 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:49.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:50.466 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:50.468 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:50.488 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-16T10:32:48.747039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":76501537,"num_objects":18696,"num_object_clones":0,"num_object_copies":37392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18696,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287613,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14567,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160749,"ondisk_log_size":160749,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1575184,"kb_used_data":353192,"kb_used_omap":12,"kb_used_meta":1221939,"kb_avail":281540336,"statfs":{"total":289910292480,"available":288297304064,"internally_reserved":0,"allocated":361668608,"data_stored":360782548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1251265538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,10,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22618112,"num_objects":-5522,"num_object_clones":0,"num_object_copies":-11044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5522,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002717"},"pg_stats":[{"pgid":"2.f","version":"48'19293","reported_seq":27761,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.768816+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:43.768816+0000","last_peered":"2024-09-16T10:32:43.768816+0000","last_clean":"2024-09-16T10:32:43.768816+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:43.768816+0000","last_undegraded":"2024-09-16T10:32:43.768816+0000","last_fullsized":"2024-09-16T10:32:43.768816+0000","mapping_epoch":45,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4812800,"num_objects":1175,"num_object_clones":0,"num_object_copies":2350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19293,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19372","reported_seq":19536,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:47.507790+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:47.507790+0000","last_peered":"2024-09-16T10:32:47.507790+0000","last_clean":"2024-09-16T10:32:47.507790+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:47.507790+0000","last_undegraded":"2024-09-16T10:32:47.507790+0000","last_fullsized":"2024-09-16T10:32:47.507790+0000","mapping_epoch":34,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4464640,"num_objects":1090,"num_object_clones":0,"num_object_copies":2180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15585,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19513","reported_seq":19770,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:47.435474+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:47.435474+0000","last_peered":"2024-09-16T10:32:47.435474+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:47.435474+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4710424,"num_objects":1151,"num_object_clones":0,"num_object_copies":2302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1151,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15570,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3152}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19413","reported_seq":19476,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:47.426336+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:47.426336+0000","last_peered":"2024-09-16T10:32:47.426336+0000","last_clean":"2024-09-16T10:32:47.426336+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:47.426336+0000","last_undegraded":"2024-09-16T10:32:47.426336+0000","last_fullsized":"2024-09-16T10:32:47.426336+0000","mapping_epoch":34,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4296704,"num_objects":1049,"num_object_clones":0,"num_object_copies":2098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19413,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19300","reported_seq":19463,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.787347+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:43.787347+0000","last_peered":"2024-09-16T10:32:43.787347+0000","last_clean":"2024-09-16T10:32:43.787347+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:43.787347+0000","last_undegraded":"2024-09-16T10:32:43.787347+0000","last_fullsized":"2024-09-16T10:32:43.787347+0000","mapping_epoch":33,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4935680,"num_objects":1205,"num_object_clones":0,"num_object_copies":2410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15942,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19133","reported_seq":19469,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.628579+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:43.628579+0000","last_peered":"2024-09-16T10:32:43.628579+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:43.628579+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5070848,"num_objects":1238,"num_object_clones":0,"num_object_copies":2476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15758,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2896}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19754","reported_seq":20166,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.703166+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:43.703166+0000","last_peered":"2024-09-16T10:32:43.703166+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:43.703166+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4775936,"num_objects":1166,"num_object_clones":0,"num_object_copies":2332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15996,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3121}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19210","reported_seq":19392,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:43.675051+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:43.675051+0000","last_peered":"2024-09-16T10:32:43.675051+0000","last_clean":"2024-09-16T10:32:43.675051+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:43.675051+0000","last_undegraded":"2024-09-16T10:32:43.675051+0000","last_fullsized":"2024-09-16T10:32:43.675051+0000","mapping_epoch":33,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4833280,"num_objects":1180,"num_object_clones":0,"num_object_copies":2360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15013,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19383","reported_seq":27893,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:47.883732+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:47.883732+0000","last_peered":"2024-09-16T10:32:47.883732+0000","last_clean":"2024-09-16T10:32:47.883732+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:47.883732+0000","last_undegraded":"2024-09-16T10:32:47.883732+0000","last_fullsized":"2024-09-16T10:32:47.883732+0000","mapping_epoch":43,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4416454,"num_objects":1080,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19381,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19802","reported_seq":20081,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.705058+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:43.705058+0000","last_peered":"2024-09-16T10:32:43.705058+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:43.705058+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5071814,"num_objects":1240,"num_object_clones":0,"num_object_copies":2480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19800,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3097}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19320","reported_seq":19520,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:47.420984+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:47.420984+0000","last_peered":"2024-09-16T10:32:47.420984+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:47.420984+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4502470,"num_objects":1101,"num_object_clones":0,"num_object_copies":2202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19318,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3132}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19416","reported_seq":27903,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:47.468852+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:47.468852+0000","last_peered":"2024-09-16T10:32:47.468852+0000","last_clean":"2024-09-16T10:32:47.468852+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:47.468852+0000","last_undegraded":"2024-09-16T10:32:47.468852+0000","last_fullsized":"2024-09-16T10:32:47.468852+0000","mapping_epoch":47,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4752326,"num_objects":1162,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19414,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18863","reported_seq":19117,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.769958+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:43.769958+0000","last_peered":"2024-09-16T10:32:43.769958+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:43.769958+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8800","ondisk_log_start":"44'8800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4796899,"num_objects":1172,"num_object_clones":0,"num_object_copies":2344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18862,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19644","reported_seq":19924,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.796043+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:43.796043+0000","last_peered":"2024-09-16T10:32:43.796043+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:43.796043+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5203369,"num_objects":1273,"num_object_clones":0,"num_object_copies":2546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19641,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2994}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'18964","reported_seq":21776,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.824357+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:43.824357+0000","last_peered":"2024-09-16T10:32:43.824357+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:43.824357+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4817862,"num_objects":1178,"num_object_clones":0,"num_object_copies":2356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18962,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1365,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19666","reported_seq":19911,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:43.768732+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:43.768732+0000","last_peered":"2024-09-16T10:32:43.768732+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:43.768732+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5040012,"num_objects":1234,"num_object_clones":0,"num_object_copies":2468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19663,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3088}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":76501518,"num_objects":18694,"num_object_clones":0,"num_object_copies":37388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36780,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18694,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287611,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14566,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":358858752,"data_stored":358781961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160746,"ondisk_log_size":160746,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515692,"kb_used_data":173156,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93856148,"statfs":{"total":96636764160,"available":96108695552,"internally_reserved":0,"allocated":177311744,"data_stored":177033098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":2.3460000000000001},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":6.9660000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":6.9960000000000004},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":4.141}]}]},{"osd":1,"up_from":40,"seq":171798691948,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477912,"kb_used_data":53324,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93893928,"statfs":{"total":96636764160,"available":96147382272,"internally_reserved":0,"allocated":54603776,"data_stored":54267322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":434761386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":1.24},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":2.1190000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":3.2599999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215230,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581580,"kb_used_data":126712,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93790260,"statfs":{"total":96636764160,"available":96041226240,"internally_reserved":0,"allocated":129753088,"data_stored":129482128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.69899999999999995}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.34799999999999998},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.379}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128532480,"data_stored":128528867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53796864,"data_stored":53731371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":176529408,"data_stored":176521723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:53.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:54.037 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:54.040 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:54.064 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-16T10:32:52.748009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67445281,"num_objects":16485,"num_object_clones":0,"num_object_copies":32970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289824,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14658,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160860,"ondisk_log_size":160860,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1555392,"kb_used_data":341588,"kb_used_omap":12,"kb_used_meta":1213747,"kb_avail":281560128,"statfs":{"total":289910292480,"available":288317571072,"internally_reserved":0,"allocated":349786112,"data_stored":348900052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1242876930},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,3,5,3,5,13,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22024192,"num_objects":-5377,"num_object_clones":0,"num_object_copies":-10754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5377,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002792"},"pg_stats":[{"pgid":"2.f","version":"48'19449","reported_seq":27916,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:48.767381+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:48.767381+0000","last_peered":"2024-09-16T10:32:48.767381+0000","last_clean":"2024-09-16T10:32:48.767381+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:48.767381+0000","last_undegraded":"2024-09-16T10:32:48.767381+0000","last_fullsized":"2024-09-16T10:32:48.767381+0000","mapping_epoch":45,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4173824,"num_objects":1019,"num_object_clones":0,"num_object_copies":2038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19449,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19526","reported_seq":19690,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:52.449732+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:52.449732+0000","last_peered":"2024-09-16T10:32:52.449732+0000","last_clean":"2024-09-16T10:32:52.449732+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:52.449732+0000","last_undegraded":"2024-09-16T10:32:52.449732+0000","last_fullsized":"2024-09-16T10:32:52.449732+0000","mapping_epoch":34,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3833856,"num_objects":936,"num_object_clones":0,"num_object_copies":1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15739,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19654","reported_seq":19911,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:52.441456+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:52.441456+0000","last_peered":"2024-09-16T10:32:52.441456+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:52.441456+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4132888,"num_objects":1010,"num_object_clones":0,"num_object_copies":2020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15711,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19564","reported_seq":19627,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:52.476050+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:52.476050+0000","last_peered":"2024-09-16T10:32:52.476050+0000","last_clean":"2024-09-16T10:32:52.476050+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:52.476050+0000","last_undegraded":"2024-09-16T10:32:52.476050+0000","last_fullsized":"2024-09-16T10:32:52.476050+0000","mapping_epoch":34,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3678208,"num_objects":898,"num_object_clones":0,"num_object_copies":1796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19564,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19445","reported_seq":19608,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:48.819926+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:48.819926+0000","last_peered":"2024-09-16T10:32:48.819926+0000","last_clean":"2024-09-16T10:32:48.819926+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:48.819926+0000","last_undegraded":"2024-09-16T10:32:48.819926+0000","last_fullsized":"2024-09-16T10:32:48.819926+0000","mapping_epoch":33,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4341760,"num_objects":1060,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16087,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19292","reported_seq":19628,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.809058+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:48.809058+0000","last_peered":"2024-09-16T10:32:48.809058+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:48.809058+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4419584,"num_objects":1079,"num_object_clones":0,"num_object_copies":2158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15917,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19882","reported_seq":20294,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.804307+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:48.804307+0000","last_peered":"2024-09-16T10:32:48.804307+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:48.804307+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4251648,"num_objects":1038,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16124,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3249}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19358","reported_seq":19540,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:48.823170+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:48.823170+0000","last_peered":"2024-09-16T10:32:48.823170+0000","last_clean":"2024-09-16T10:32:48.823170+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:48.823170+0000","last_undegraded":"2024-09-16T10:32:48.823170+0000","last_fullsized":"2024-09-16T10:32:48.823170+0000","mapping_epoch":33,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4227072,"num_objects":1032,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15161,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19383","reported_seq":27893,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:47.883732+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:47.883732+0000","last_peered":"2024-09-16T10:32:47.883732+0000","last_clean":"2024-09-16T10:32:47.883732+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:47.883732+0000","last_undegraded":"2024-09-16T10:32:47.883732+0000","last_fullsized":"2024-09-16T10:32:47.883732+0000","mapping_epoch":43,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4416454,"num_objects":1080,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19381,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19929","reported_seq":20207,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.833001+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:48.833001+0000","last_peered":"2024-09-16T10:32:48.833001+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:48.833001+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4551622,"num_objects":1113,"num_object_clones":0,"num_object_copies":2226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19927,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3224}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19457","reported_seq":19657,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:52.441432+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:52.441432+0000","last_peered":"2024-09-16T10:32:52.441432+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:52.441432+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3941318,"num_objects":964,"num_object_clones":0,"num_object_copies":1928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19455,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3269}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19575","reported_seq":28063,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:52.408106+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:52.408106+0000","last_peered":"2024-09-16T10:32:52.408106+0000","last_clean":"2024-09-16T10:32:52.408106+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:52.408106+0000","last_undegraded":"2024-09-16T10:32:52.408106+0000","last_fullsized":"2024-09-16T10:32:52.408106+0000","mapping_epoch":47,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4101062,"num_objects":1003,"num_object_clones":0,"num_object_copies":2006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19573,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18980","reported_seq":19230,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.851033+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:48.851033+0000","last_peered":"2024-09-16T10:32:48.851033+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:48.851033+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4317667,"num_objects":1055,"num_object_clones":0,"num_object_copies":2110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18979,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3020}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19807","reported_seq":20087,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.833447+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:48.833447+0000","last_peered":"2024-09-16T10:32:48.833447+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:48.833447+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4535721,"num_objects":1110,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19804,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19121","reported_seq":22117,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.883510+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:48.883510+0000","last_peered":"2024-09-16T10:32:48.883510+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:48.883510+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4174790,"num_objects":1021,"num_object_clones":0,"num_object_copies":2042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19119,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19835","reported_seq":20080,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.845120+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:48.845120+0000","last_peered":"2024-09-16T10:32:48.845120+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:48.845120+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4347788,"num_objects":1065,"num_object_clones":0,"num_object_copies":2130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19832,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":67445262,"num_objects":16483,"num_object_clones":0,"num_object_copies":32966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289822,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14657,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":347070464,"data_stored":346993673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160857,"ondisk_log_size":160857,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515208,"kb_used_data":168568,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93856632,"statfs":{"total":96636764160,"available":96109191168,"internally_reserved":0,"allocated":172613632,"data_stored":172334986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":1.28}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.3260000000000001},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":1.3879999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691948,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477912,"kb_used_data":53324,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93893928,"statfs":{"total":96636764160,"available":96147382272,"internally_reserved":0,"allocated":54603776,"data_stored":54267322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":434761386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":1.24},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":2.1190000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":3.2599999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215231,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562272,"kb_used_data":119696,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93809568,"statfs":{"total":96636764160,"available":96060997632,"internally_reserved":0,"allocated":122568704,"data_stored":122297744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":453177000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,3,5,1,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":1.3939999999999999},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":1.353}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.31900000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.4650000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121368576,"data_stored":121364963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53796864,"data_stored":53731371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171905024,"data_stored":171897339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:54.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:54.603 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:54.603 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:54.625 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2024-09-16T10:32:52.748009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67445281,"num_objects":16485,"num_object_clones":0,"num_object_copies":32970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289824,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14658,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160860,"ondisk_log_size":160860,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1555392,"kb_used_data":341588,"kb_used_omap":12,"kb_used_meta":1213747,"kb_avail":281560128,"statfs":{"total":289910292480,"available":288317571072,"internally_reserved":0,"allocated":349786112,"data_stored":348900052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1242876930},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,3,5,3,5,13,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22024192,"num_objects":-5377,"num_object_clones":0,"num_object_copies":-10754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5377,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002792"},"pg_stats":[{"pgid":"2.f","version":"48'19449","reported_seq":27916,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:48.767381+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:48.767381+0000","last_peered":"2024-09-16T10:32:48.767381+0000","last_clean":"2024-09-16T10:32:48.767381+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:48.767381+0000","last_undegraded":"2024-09-16T10:32:48.767381+0000","last_fullsized":"2024-09-16T10:32:48.767381+0000","mapping_epoch":45,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4173824,"num_objects":1019,"num_object_clones":0,"num_object_copies":2038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19449,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19526","reported_seq":19690,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:52.449732+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:52.449732+0000","last_peered":"2024-09-16T10:32:52.449732+0000","last_clean":"2024-09-16T10:32:52.449732+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:52.449732+0000","last_undegraded":"2024-09-16T10:32:52.449732+0000","last_fullsized":"2024-09-16T10:32:52.449732+0000","mapping_epoch":34,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3833856,"num_objects":936,"num_object_clones":0,"num_object_copies":1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15739,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19654","reported_seq":19911,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:52.441456+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:52.441456+0000","last_peered":"2024-09-16T10:32:52.441456+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:52.441456+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4132888,"num_objects":1010,"num_object_clones":0,"num_object_copies":2020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15711,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19564","reported_seq":19627,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:52.476050+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:52.476050+0000","last_peered":"2024-09-16T10:32:52.476050+0000","last_clean":"2024-09-16T10:32:52.476050+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:52.476050+0000","last_undegraded":"2024-09-16T10:32:52.476050+0000","last_fullsized":"2024-09-16T10:32:52.476050+0000","mapping_epoch":34,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3678208,"num_objects":898,"num_object_clones":0,"num_object_copies":1796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19564,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19445","reported_seq":19608,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:48.819926+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:48.819926+0000","last_peered":"2024-09-16T10:32:48.819926+0000","last_clean":"2024-09-16T10:32:48.819926+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:48.819926+0000","last_undegraded":"2024-09-16T10:32:48.819926+0000","last_fullsized":"2024-09-16T10:32:48.819926+0000","mapping_epoch":33,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4341760,"num_objects":1060,"num_object_clones":0,"num_object_copies":2120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16087,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19292","reported_seq":19628,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.809058+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:48.809058+0000","last_peered":"2024-09-16T10:32:48.809058+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:48.809058+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4419584,"num_objects":1079,"num_object_clones":0,"num_object_copies":2158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15917,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'19882","reported_seq":20294,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.804307+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:48.804307+0000","last_peered":"2024-09-16T10:32:48.804307+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:48.804307+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4251648,"num_objects":1038,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16124,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3249}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19358","reported_seq":19540,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:48.823170+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:48.823170+0000","last_peered":"2024-09-16T10:32:48.823170+0000","last_clean":"2024-09-16T10:32:48.823170+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:48.823170+0000","last_undegraded":"2024-09-16T10:32:48.823170+0000","last_fullsized":"2024-09-16T10:32:48.823170+0000","mapping_epoch":33,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4227072,"num_objects":1032,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15161,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19383","reported_seq":27893,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:47.883732+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:47.883732+0000","last_peered":"2024-09-16T10:32:47.883732+0000","last_clean":"2024-09-16T10:32:47.883732+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:47.883732+0000","last_undegraded":"2024-09-16T10:32:47.883732+0000","last_fullsized":"2024-09-16T10:32:47.883732+0000","mapping_epoch":43,"log_start":"44'9300","ondisk_log_start":"44'9300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4416454,"num_objects":1080,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19381,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'19929","reported_seq":20207,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.833001+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:48.833001+0000","last_peered":"2024-09-16T10:32:48.833001+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:48.833001+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4551622,"num_objects":1113,"num_object_clones":0,"num_object_copies":2226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19927,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3224}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19457","reported_seq":19657,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:52.441432+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:52.441432+0000","last_peered":"2024-09-16T10:32:52.441432+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:52.441432+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3941318,"num_objects":964,"num_object_clones":0,"num_object_copies":1928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19455,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3269}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19575","reported_seq":28063,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:52.408106+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:52.408106+0000","last_peered":"2024-09-16T10:32:52.408106+0000","last_clean":"2024-09-16T10:32:52.408106+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:52.408106+0000","last_undegraded":"2024-09-16T10:32:52.408106+0000","last_fullsized":"2024-09-16T10:32:52.408106+0000","mapping_epoch":47,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4101062,"num_objects":1003,"num_object_clones":0,"num_object_copies":2006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19573,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'18980","reported_seq":19230,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.851033+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:48.851033+0000","last_peered":"2024-09-16T10:32:48.851033+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:48.851033+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'8900","ondisk_log_start":"44'8900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4317667,"num_objects":1055,"num_object_clones":0,"num_object_copies":2110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18979,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3020}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19807","reported_seq":20087,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.833447+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:48.833447+0000","last_peered":"2024-09-16T10:32:48.833447+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:48.833447+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4535721,"num_objects":1110,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19804,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3157}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19121","reported_seq":22117,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.883510+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:48.883510+0000","last_peered":"2024-09-16T10:32:48.883510+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:48.883510+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4174790,"num_objects":1021,"num_object_clones":0,"num_object_copies":2042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19119,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19835","reported_seq":20080,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:48.845120+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:48.845120+0000","last_peered":"2024-09-16T10:32:48.845120+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:48.845120+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4347788,"num_objects":1065,"num_object_clones":0,"num_object_copies":2130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19832,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":67445262,"num_objects":16483,"num_object_clones":0,"num_object_copies":32966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16483,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":289822,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14657,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":347070464,"data_stored":346993673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160857,"ondisk_log_size":160857,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515208,"kb_used_data":168568,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93856632,"statfs":{"total":96636764160,"available":96109191168,"internally_reserved":0,"allocated":172613632,"data_stored":172334986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":1.28}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.3260000000000001},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":1.3879999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691948,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477912,"kb_used_data":53324,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93893928,"statfs":{"total":96636764160,"available":96147382272,"internally_reserved":0,"allocated":54603776,"data_stored":54267322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":434761386},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":1.24},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":2.1190000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":3.2599999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215231,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562272,"kb_used_data":119696,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93809568,"statfs":{"total":96636764160,"available":96060997632,"internally_reserved":0,"allocated":122568704,"data_stored":122297744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":453177000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,3,5,1,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":1.3939999999999999},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":1.353}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.31900000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.4650000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":121368576,"data_stored":121364963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":53796864,"data_stored":53731371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171905024,"data_stored":171897339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:54.626 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:32:54.626 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'19654", 'reported_seq': 19911, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:52.441456+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:52.441456+0000', 'last_peered': '2024-09-16T10:32:52.441456+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:52.441456+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'9600", 'ondisk_log_start': "44'9600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4132888, 'num_objects': 1010, 'num_object_clones': 0, 'num_object_copies': 2020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1010, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15711, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3293}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:54.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:32:54.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'19292", 'reported_seq': 19628, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:48.809058+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:32:48.809058+0000', 'last_peered': '2024-09-16T10:32:48.809058+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:32:48.809058+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'9200", 'ondisk_log_start': "44'9200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4419584, 'num_objects': 1079, 'num_object_clones': 0, 'num_object_copies': 2158, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1079, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 15917, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3055}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:54.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:32:54.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'19882", 'reported_seq': 20294, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:48.804307+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:32:48.804307+0000', 'last_peered': '2024-09-16T10:32:48.804307+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:32:48.804307+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'9800", 'ondisk_log_start': "44'9800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4251648, 'num_objects': 1038, 'num_object_clones': 0, 'num_object_copies': 2076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16124, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3249}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:54.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:32:54.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'19929", 'reported_seq': 20207, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:48.833001+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:32:48.833001+0000', 'last_peered': '2024-09-16T10:32:48.833001+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:32:48.833001+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'9900", 'ondisk_log_start': "44'9900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4551622, 'num_objects': 1113, 'num_object_clones': 0, 'num_object_copies': 2226, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1113, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19927, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3224}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:54.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:32:54.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'19457", 'reported_seq': 19657, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:52.441432+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:52.441432+0000', 'last_peered': '2024-09-16T10:32:52.441432+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:52.441432+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'9400", 'ondisk_log_start': "44'9400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3941318, 'num_objects': 964, 'num_object_clones': 0, 'num_object_copies': 1928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19455, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3269}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:32:54.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:32:54.627 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'18980", 'reported_seq': 19230, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:48.851033+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:32:48.851033+0000', 'last_peered': '2024-09-16T10:32:48.851033+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:32:48.851033+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'8900", 'ondisk_log_start': "44'8900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4317667, 'num_objects': 1055, 'num_object_clones': 0, 'num_object_copies': 2110, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 18979, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3020}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:54.628 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:32:54.628 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'19807", 'reported_seq': 20087, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:48.833447+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:32:48.833447+0000', 'last_peered': '2024-09-16T10:32:48.833447+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:32:48.833447+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'9800", 'ondisk_log_start': "44'9800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4535721, 'num_objects': 1110, 'num_object_clones': 0, 'num_object_copies': 2220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19804, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:54.628 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:32:54.628 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'19121", 'reported_seq': 22117, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:48.883510+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:32:48.883510+0000', 'last_peered': '2024-09-16T10:32:48.883510+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:32:48.883510+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'9100", 'ondisk_log_start': "44'9100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4174790, 'num_objects': 1021, 'num_object_clones': 0, 'num_object_copies': 2042, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2715, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19119, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1456, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1694}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:54.628 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:32:54.628 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'19835", 'reported_seq': 20080, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:48.845120+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:32:48.845120+0000', 'last_peered': '2024-09-16T10:32:48.845120+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:32:48.845120+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'9800", 'ondisk_log_start': "44'9800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4347788, 'num_objects': 1065, 'num_object_clones': 0, 'num_object_copies': 2130, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19832, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3257}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:32:54.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:32:55.316 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:55.317 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36604/29750 objects degraded (123.039%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":664,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":14875,"data_bytes":60850721,"bytes_used":1590845440,"bytes_avail":288319447040,"bytes_total":289910292480,"degraded_objects":36604,"degraded_total":29750,"degraded_ratio":1.2303865546218486,"recovering_objects_per_sec":21,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":523},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:32:55.343 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:32:55.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:55.921 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:55.924 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:55.949 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-16T10:32:54.748480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":60850721,"num_objects":14875,"num_object_clones":0,"num_object_copies":29750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":291434,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14743,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160970,"ondisk_log_size":160970,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1553560,"kb_used_data":327480,"kb_used_omap":12,"kb_used_meta":1226035,"kb_avail":281561960,"statfs":{"total":289910292480,"available":288319447040,"internally_reserved":0,"allocated":335339520,"data_stored":334453460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1255459842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,6,11,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25747456,"num_objects":-6286,"num_object_clones":0,"num_object_copies":-12572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-260,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6286,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002791"},"pg_stats":[{"pgid":"2.f","version":"48'19590","reported_seq":28058,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:53.943756+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:53.943756+0000","last_peered":"2024-09-16T10:32:53.943756+0000","last_clean":"2024-09-16T10:32:53.943756+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:53.943756+0000","last_undegraded":"2024-09-16T10:32:53.943756+0000","last_fullsized":"2024-09-16T10:32:53.943756+0000","mapping_epoch":45,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596288,"num_objects":878,"num_object_clones":0,"num_object_copies":1756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19590,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19526","reported_seq":19690,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:52.449732+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:52.449732+0000","last_peered":"2024-09-16T10:32:52.449732+0000","last_clean":"2024-09-16T10:32:52.449732+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:52.449732+0000","last_undegraded":"2024-09-16T10:32:52.449732+0000","last_fullsized":"2024-09-16T10:32:52.449732+0000","mapping_epoch":34,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3833856,"num_objects":936,"num_object_clones":0,"num_object_copies":1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15739,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19654","reported_seq":19911,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:52.441456+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:52.441456+0000","last_peered":"2024-09-16T10:32:52.441456+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:52.441456+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4132888,"num_objects":1010,"num_object_clones":0,"num_object_copies":2020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15711,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3293}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19564","reported_seq":19627,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:52.476050+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:52.476050+0000","last_peered":"2024-09-16T10:32:52.476050+0000","last_clean":"2024-09-16T10:32:52.476050+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:52.476050+0000","last_undegraded":"2024-09-16T10:32:52.476050+0000","last_fullsized":"2024-09-16T10:32:52.476050+0000","mapping_epoch":34,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3678208,"num_objects":898,"num_object_clones":0,"num_object_copies":1796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19564,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19582","reported_seq":19745,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:53.866015+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:53.866015+0000","last_peered":"2024-09-16T10:32:53.866015+0000","last_clean":"2024-09-16T10:32:53.866015+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:53.866015+0000","last_undegraded":"2024-09-16T10:32:53.866015+0000","last_fullsized":"2024-09-16T10:32:53.866015+0000","mapping_epoch":33,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3780608,"num_objects":923,"num_object_clones":0,"num_object_copies":1846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16224,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19448","reported_seq":19783,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.833476+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:53.833476+0000","last_peered":"2024-09-16T10:32:53.833476+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:53.833476+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3780608,"num_objects":923,"num_object_clones":0,"num_object_copies":1846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16073,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20029","reported_seq":20441,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.903891+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:53.903891+0000","last_peered":"2024-09-16T10:32:53.903891+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:53.903891+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3649536,"num_objects":891,"num_object_clones":0,"num_object_copies":1782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16271,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19487","reported_seq":19669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:53.696924+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:53.696924+0000","last_peered":"2024-09-16T10:32:53.696924+0000","last_clean":"2024-09-16T10:32:53.696924+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:53.696924+0000","last_undegraded":"2024-09-16T10:32:53.696924+0000","last_fullsized":"2024-09-16T10:32:53.696924+0000","mapping_epoch":33,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3698688,"num_objects":903,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15290,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19523","reported_seq":28033,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:53.074868+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:53.074868+0000","last_peered":"2024-09-16T10:32:53.074868+0000","last_clean":"2024-09-16T10:32:53.074868+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:53.074868+0000","last_undegraded":"2024-09-16T10:32:53.074868+0000","last_fullsized":"2024-09-16T10:32:53.074868+0000","mapping_epoch":43,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3843014,"num_objects":940,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19521,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20066","reported_seq":20345,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.811385+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:53.811385+0000","last_peered":"2024-09-16T10:32:53.811385+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:53.811385+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3990470,"num_objects":976,"num_object_clones":0,"num_object_copies":1952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20064,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19457","reported_seq":19657,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:52.441432+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:52.441432+0000","last_peered":"2024-09-16T10:32:52.441432+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:52.441432+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3941318,"num_objects":964,"num_object_clones":0,"num_object_copies":1928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19455,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3269}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19575","reported_seq":28063,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:52.408106+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:52.408106+0000","last_peered":"2024-09-16T10:32:52.408106+0000","last_clean":"2024-09-16T10:32:52.408106+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:52.408106+0000","last_undegraded":"2024-09-16T10:32:52.408106+0000","last_fullsized":"2024-09-16T10:32:52.408106+0000","mapping_epoch":47,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4101062,"num_objects":1003,"num_object_clones":0,"num_object_copies":2006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19573,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19125","reported_seq":19379,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.940702+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:53.940702+0000","last_peered":"2024-09-16T10:32:53.940702+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:53.940702+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3723747,"num_objects":910,"num_object_clones":0,"num_object_copies":1820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19124,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19976","reported_seq":20256,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.879704+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:53.879704+0000","last_peered":"2024-09-16T10:32:53.879704+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:53.879704+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3843497,"num_objects":941,"num_object_clones":0,"num_object_copies":1882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19973,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19268","reported_seq":22432,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.921690+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:53.921690+0000","last_peered":"2024-09-16T10:32:53.921690+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:53.921690+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3572678,"num_objects":874,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19266,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1541,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1756}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19997","reported_seq":20242,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.910999+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:53.910999+0000","last_peered":"2024-09-16T10:32:53.910999+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:53.910999+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3684236,"num_objects":903,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19994,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":60850702,"num_objects":14873,"num_object_clones":0,"num_object_copies":29746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":291432,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14742,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":332591104,"data_stored":332514313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160967,"ondisk_log_size":160967,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":515208,"kb_used_data":168568,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93856632,"statfs":{"total":96636764160,"available":96109191168,"internally_reserved":0,"allocated":172613632,"data_stored":172334986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":1.28}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":1.3260000000000001},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":1.3879999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691949,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474904,"kb_used_data":46224,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93896936,"statfs":{"total":96636764160,"available":96150462464,"internally_reserved":0,"allocated":47333376,"data_stored":46996922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":438955690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":2.395},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":7.1180000000000003}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":7.1459999999999999},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":7.2539999999999996}]}]},{"osd":0,"up_from":24,"seq":103079215232,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563448,"kb_used_data":112688,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93808392,"statfs":{"total":96636764160,"available":96059793408,"internally_reserved":0,"allocated":115392512,"data_stored":115121552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":461565608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.23699999999999999},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":4.9740000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":9.1880000000000006},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":5.0529999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":114106368,"data_stored":114102755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":46579712,"data_stored":46514219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":171905024,"data_stored":171897339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:58.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:32:59.525 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:32:59.527 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:32:59.556 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-16T10:32:58.749393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":57426465,"num_objects":14039,"num_object_clones":0,"num_object_copies":28078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14039,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292270,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14743,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161006,"ondisk_log_size":161006,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1555900,"kb_used_data":315872,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281559620,"statfs":{"total":289910292480,"available":288317050880,"internally_reserved":0,"allocated":323452928,"data_stored":322566868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,7,10,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22597632,"num_objects":-5517,"num_object_clones":0,"num_object_copies":-11034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002778"},"pg_stats":[{"pgid":"2.f","version":"48'19590","reported_seq":28058,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:53.943756+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:53.943756+0000","last_peered":"2024-09-16T10:32:53.943756+0000","last_clean":"2024-09-16T10:32:53.943756+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:53.943756+0000","last_undegraded":"2024-09-16T10:32:53.943756+0000","last_fullsized":"2024-09-16T10:32:53.943756+0000","mapping_epoch":45,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596288,"num_objects":878,"num_object_clones":0,"num_object_copies":1756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19590,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19674","reported_seq":19838,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:57.458610+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:57.458610+0000","last_peered":"2024-09-16T10:32:57.458610+0000","last_clean":"2024-09-16T10:32:57.458610+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:57.458610+0000","last_undegraded":"2024-09-16T10:32:57.458610+0000","last_fullsized":"2024-09-16T10:32:57.458610+0000","mapping_epoch":34,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227648,"num_objects":788,"num_object_clones":0,"num_object_copies":1576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15887,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19798","reported_seq":20054,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:57.550091+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:57.550091+0000","last_peered":"2024-09-16T10:32:57.550091+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:57.550091+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3543064,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15855,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3437}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19690","reported_seq":19753,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:57.531974+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:57.531974+0000","last_peered":"2024-09-16T10:32:57.531974+0000","last_clean":"2024-09-16T10:32:57.531974+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:57.531974+0000","last_undegraded":"2024-09-16T10:32:57.531974+0000","last_fullsized":"2024-09-16T10:32:57.531974+0000","mapping_epoch":34,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3162112,"num_objects":772,"num_object_clones":0,"num_object_copies":1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19690,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19582","reported_seq":19745,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:53.866015+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:53.866015+0000","last_peered":"2024-09-16T10:32:53.866015+0000","last_clean":"2024-09-16T10:32:53.866015+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:53.866015+0000","last_undegraded":"2024-09-16T10:32:53.866015+0000","last_fullsized":"2024-09-16T10:32:53.866015+0000","mapping_epoch":33,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3780608,"num_objects":923,"num_object_clones":0,"num_object_copies":1846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16224,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19448","reported_seq":19783,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.833476+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:53.833476+0000","last_peered":"2024-09-16T10:32:53.833476+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:53.833476+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3780608,"num_objects":923,"num_object_clones":0,"num_object_copies":1846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16073,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20029","reported_seq":20441,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.903891+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:53.903891+0000","last_peered":"2024-09-16T10:32:53.903891+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:53.903891+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3649536,"num_objects":891,"num_object_clones":0,"num_object_copies":1782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16271,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19487","reported_seq":19669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:53.696924+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:53.696924+0000","last_peered":"2024-09-16T10:32:53.696924+0000","last_clean":"2024-09-16T10:32:53.696924+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:53.696924+0000","last_undegraded":"2024-09-16T10:32:53.696924+0000","last_fullsized":"2024-09-16T10:32:53.696924+0000","mapping_epoch":33,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3698688,"num_objects":903,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15290,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19661","reported_seq":28167,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:58.048552+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:58.048552+0000","last_peered":"2024-09-16T10:32:58.048552+0000","last_clean":"2024-09-16T10:32:58.048552+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:58.048552+0000","last_undegraded":"2024-09-16T10:32:58.048552+0000","last_fullsized":"2024-09-16T10:32:58.048552+0000","mapping_epoch":43,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3277766,"num_objects":802,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19659,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20066","reported_seq":20345,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.811385+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:53.811385+0000","last_peered":"2024-09-16T10:32:53.811385+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:53.811385+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3990470,"num_objects":976,"num_object_clones":0,"num_object_copies":1952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20064,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19601","reported_seq":19797,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:57.445492+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:57.445492+0000","last_peered":"2024-09-16T10:32:57.445492+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:57.445492+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3351494,"num_objects":820,"num_object_clones":0,"num_object_copies":1640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19599,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19711","reported_seq":28200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:57.540987+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:57.540987+0000","last_peered":"2024-09-16T10:32:57.540987+0000","last_clean":"2024-09-16T10:32:57.540987+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:57.540987+0000","last_undegraded":"2024-09-16T10:32:57.540987+0000","last_fullsized":"2024-09-16T10:32:57.540987+0000","mapping_epoch":47,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3544006,"num_objects":867,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19709,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19125","reported_seq":19379,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.940702+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:53.940702+0000","last_peered":"2024-09-16T10:32:53.940702+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:53.940702+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3723747,"num_objects":910,"num_object_clones":0,"num_object_copies":1820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19124,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19976","reported_seq":20256,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.879704+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:53.879704+0000","last_peered":"2024-09-16T10:32:53.879704+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:53.879704+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3843497,"num_objects":941,"num_object_clones":0,"num_object_copies":1882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19973,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19268","reported_seq":22432,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.921690+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:53.921690+0000","last_peered":"2024-09-16T10:32:53.921690+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:53.921690+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3572678,"num_objects":874,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19266,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1541,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1756}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19997","reported_seq":20242,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.910999+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:53.910999+0000","last_peered":"2024-09-16T10:32:53.910999+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:53.910999+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3684236,"num_objects":903,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19994,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":57426446,"num_objects":14037,"num_object_clones":0,"num_object_copies":28074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292268,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14742,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":320778240,"data_stored":320701449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161003,"ondisk_log_size":161003,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019006,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498328,"kb_used_data":163984,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93873512,"statfs":{"total":96636764160,"available":96126476288,"internally_reserved":0,"allocated":167919616,"data_stored":167640970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":342355632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":3.4260000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.73},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.27600000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494124,"kb_used_data":39200,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93877716,"statfs":{"total":96636764160,"available":96130781184,"internally_reserved":0,"allocated":40140800,"data_stored":39804346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.7549999999999999},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":3.7959999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215232,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563448,"kb_used_data":112688,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93808392,"statfs":{"total":96636764160,"available":96059793408,"internally_reserved":0,"allocated":115392512,"data_stored":115121552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":461565608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.23699999999999999},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":4.9740000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":9.1880000000000006},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":5.0529999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":114106368,"data_stored":114102755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39419904,"data_stored":39354411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167251968,"data_stored":167244283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:32:59.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:00.045 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-16T10:32:58.749393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":57426465,"num_objects":14039,"num_object_clones":0,"num_object_copies":28078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14039,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292270,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14743,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161006,"ondisk_log_size":161006,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1555900,"kb_used_data":315872,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281559620,"statfs":{"total":289910292480,"available":288317050880,"internally_reserved":0,"allocated":323452928,"data_stored":322566868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,6,7,10,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22597632,"num_objects":-5517,"num_object_clones":0,"num_object_copies":-11034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002778"},"pg_stats":[{"pgid":"2.f","version":"48'19590","reported_seq":28058,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:53.943756+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:53.943756+0000","last_peered":"2024-09-16T10:32:53.943756+0000","last_clean":"2024-09-16T10:32:53.943756+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:53.943756+0000","last_undegraded":"2024-09-16T10:32:53.943756+0000","last_fullsized":"2024-09-16T10:32:53.943756+0000","mapping_epoch":45,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3596288,"num_objects":878,"num_object_clones":0,"num_object_copies":1756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19590,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19674","reported_seq":19838,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:57.458610+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:57.458610+0000","last_peered":"2024-09-16T10:32:57.458610+0000","last_clean":"2024-09-16T10:32:57.458610+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:57.458610+0000","last_undegraded":"2024-09-16T10:32:57.458610+0000","last_fullsized":"2024-09-16T10:32:57.458610+0000","mapping_epoch":34,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227648,"num_objects":788,"num_object_clones":0,"num_object_copies":1576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15887,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19798","reported_seq":20054,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:57.550091+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:57.550091+0000","last_peered":"2024-09-16T10:32:57.550091+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:57.550091+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3543064,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15855,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3437}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19690","reported_seq":19753,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:57.531974+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:57.531974+0000","last_peered":"2024-09-16T10:32:57.531974+0000","last_clean":"2024-09-16T10:32:57.531974+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:57.531974+0000","last_undegraded":"2024-09-16T10:32:57.531974+0000","last_fullsized":"2024-09-16T10:32:57.531974+0000","mapping_epoch":34,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3162112,"num_objects":772,"num_object_clones":0,"num_object_copies":1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19690,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19582","reported_seq":19745,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:53.866015+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:53.866015+0000","last_peered":"2024-09-16T10:32:53.866015+0000","last_clean":"2024-09-16T10:32:53.866015+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:53.866015+0000","last_undegraded":"2024-09-16T10:32:53.866015+0000","last_fullsized":"2024-09-16T10:32:53.866015+0000","mapping_epoch":33,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3780608,"num_objects":923,"num_object_clones":0,"num_object_copies":1846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16224,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19448","reported_seq":19783,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.833476+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:53.833476+0000","last_peered":"2024-09-16T10:32:53.833476+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:53.833476+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3780608,"num_objects":923,"num_object_clones":0,"num_object_copies":1846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16073,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3211}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20029","reported_seq":20441,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.903891+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:53.903891+0000","last_peered":"2024-09-16T10:32:53.903891+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:53.903891+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3649536,"num_objects":891,"num_object_clones":0,"num_object_copies":1782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16271,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3396}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19487","reported_seq":19669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:53.696924+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:53.696924+0000","last_peered":"2024-09-16T10:32:53.696924+0000","last_clean":"2024-09-16T10:32:53.696924+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:53.696924+0000","last_undegraded":"2024-09-16T10:32:53.696924+0000","last_fullsized":"2024-09-16T10:32:53.696924+0000","mapping_epoch":33,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3698688,"num_objects":903,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15290,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19661","reported_seq":28167,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:58.048552+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:58.048552+0000","last_peered":"2024-09-16T10:32:58.048552+0000","last_clean":"2024-09-16T10:32:58.048552+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:58.048552+0000","last_undegraded":"2024-09-16T10:32:58.048552+0000","last_fullsized":"2024-09-16T10:32:58.048552+0000","mapping_epoch":43,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3277766,"num_objects":802,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19659,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20066","reported_seq":20345,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.811385+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:53.811385+0000","last_peered":"2024-09-16T10:32:53.811385+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:53.811385+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3990470,"num_objects":976,"num_object_clones":0,"num_object_copies":1952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20064,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19601","reported_seq":19797,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:57.445492+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:57.445492+0000","last_peered":"2024-09-16T10:32:57.445492+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:57.445492+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3351494,"num_objects":820,"num_object_clones":0,"num_object_copies":1640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19599,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19711","reported_seq":28200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:57.540987+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:57.540987+0000","last_peered":"2024-09-16T10:32:57.540987+0000","last_clean":"2024-09-16T10:32:57.540987+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:57.540987+0000","last_undegraded":"2024-09-16T10:32:57.540987+0000","last_fullsized":"2024-09-16T10:32:57.540987+0000","mapping_epoch":47,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3544006,"num_objects":867,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19709,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19125","reported_seq":19379,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.940702+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:53.940702+0000","last_peered":"2024-09-16T10:32:53.940702+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:53.940702+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9100","ondisk_log_start":"44'9100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3723747,"num_objects":910,"num_object_clones":0,"num_object_copies":1820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19124,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3165}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'19976","reported_seq":20256,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.879704+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:53.879704+0000","last_peered":"2024-09-16T10:32:53.879704+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:53.879704+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3843497,"num_objects":941,"num_object_clones":0,"num_object_copies":1882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19973,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3326}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19268","reported_seq":22432,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.921690+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:53.921690+0000","last_peered":"2024-09-16T10:32:53.921690+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:53.921690+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3572678,"num_objects":874,"num_object_clones":0,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19266,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1541,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1756}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'19997","reported_seq":20242,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:53.910999+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:53.910999+0000","last_peered":"2024-09-16T10:32:53.910999+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:53.910999+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3684236,"num_objects":903,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19994,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3419}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":57426446,"num_objects":14037,"num_object_clones":0,"num_object_copies":28074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14037,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292268,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14742,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":320778240,"data_stored":320701449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161003,"ondisk_log_size":161003,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019006,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498328,"kb_used_data":163984,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93873512,"statfs":{"total":96636764160,"available":96126476288,"internally_reserved":0,"allocated":167919616,"data_stored":167640970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":342355632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":3.4260000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.73},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.27600000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494124,"kb_used_data":39200,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93877716,"statfs":{"total":96636764160,"available":96130781184,"internally_reserved":0,"allocated":40140800,"data_stored":39804346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.7549999999999999},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":3.7959999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215232,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563448,"kb_used_data":112688,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93808392,"statfs":{"total":96636764160,"available":96059793408,"internally_reserved":0,"allocated":115392512,"data_stored":115121552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":461565608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,4,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.23699999999999999},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":4.9740000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":9.1880000000000006},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":5.0529999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":114106368,"data_stored":114102755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39419904,"data_stored":39354411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167251968,"data_stored":167244283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:00.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:00.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'19798", 'reported_seq': 20054, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:57.550091+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:32:57.550091+0000', 'last_peered': '2024-09-16T10:32:57.550091+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:32:57.550091+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'9700", 'ondisk_log_start': "44'9700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3543064, 'num_objects': 866, 'num_object_clones': 0, 'num_object_copies': 1732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 866, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 15855, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3437}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:00.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:00.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'19448", 'reported_seq': 19783, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:53.833476+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:32:53.833476+0000', 'last_peered': '2024-09-16T10:32:53.833476+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:32:53.833476+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'9400", 'ondisk_log_start': "44'9400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3780608, 'num_objects': 923, 'num_object_clones': 0, 'num_object_copies': 1846, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16073, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3211}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:00.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:00.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20029", 'reported_seq': 20441, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:53.903891+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:32:53.903891+0000', 'last_peered': '2024-09-16T10:32:53.903891+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:32:53.903891+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3649536, 'num_objects': 891, 'num_object_clones': 0, 'num_object_copies': 1782, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16271, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:00.047 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:00.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'20066", 'reported_seq': 20345, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:53.811385+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:32:53.811385+0000', 'last_peered': '2024-09-16T10:32:53.811385+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:32:53.811385+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3990470, 'num_objects': 976, 'num_object_clones': 0, 'num_object_copies': 1952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20064, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3361}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:00.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:00.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'19601", 'reported_seq': 19797, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:57.445492+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:32:57.445492+0000', 'last_peered': '2024-09-16T10:32:57.445492+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:32:57.445492+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'9600", 'ondisk_log_start': "44'9600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3351494, 'num_objects': 820, 'num_object_clones': 0, 'num_object_copies': 1640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19599, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3413}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:00.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:00.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'19125", 'reported_seq': 19379, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:53.940702+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:32:53.940702+0000', 'last_peered': '2024-09-16T10:32:53.940702+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:32:53.940702+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'9100", 'ondisk_log_start': "44'9100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3723747, 'num_objects': 910, 'num_object_clones': 0, 'num_object_copies': 1820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 910, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19124, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3165}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:00.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:00.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'19976", 'reported_seq': 20256, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:53.879704+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:32:53.879704+0000', 'last_peered': '2024-09-16T10:32:53.879704+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:32:53.879704+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'9900", 'ondisk_log_start': "44'9900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3843497, 'num_objects': 941, 'num_object_clones': 0, 'num_object_copies': 1882, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19973, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3326}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:00.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:33:00.048 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'19268", 'reported_seq': 22432, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:53.921690+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:32:53.921690+0000', 'last_peered': '2024-09-16T10:32:53.921690+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:32:53.921690+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'9200", 'ondisk_log_start': "44'9200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3572678, 'num_objects': 874, 'num_object_clones': 0, 'num_object_copies': 1748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2630, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19266, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1541, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1756}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:00.049 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:00.049 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'19997", 'reported_seq': 20242, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:32:53.910999+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:32:53.910999+0000', 'last_peered': '2024-09-16T10:32:53.910999+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:32:53.910999+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'9900", 'ondisk_log_start': "44'9900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3684236, 'num_objects': 903, 'num_object_clones': 0, 'num_object_copies': 1806, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19994, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3419}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:00.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:00.709 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:00.709 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36604/28078 objects degraded (130.365%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":669,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":14039,"data_bytes":57426465,"bytes_used":1593241600,"bytes_avail":288317050880,"bytes_total":289910292480,"degraded_objects":36604,"degraded_total":28078,"degraded_ratio":1.303654106417836,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":459},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:00.729 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:00.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:01.273 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:01.276 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:01.299 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-16T10:33:00.749896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51450401,"num_objects":12580,"num_object_clones":0,"num_object_copies":25160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293729,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14835,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160765,"ondisk_log_size":160765,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1553180,"kb_used_data":309044,"kb_used_omap":12,"kb_used_meta":1244083,"kb_avail":281562340,"statfs":{"total":289910292480,"available":288319836160,"internally_reserved":0,"allocated":316461056,"data_stored":315574996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1273940994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,7,8,13],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25051136,"num_objects":-6116,"num_object_clones":0,"num_object_copies":-12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6116,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"2.f","version":"48'19744","reported_seq":28212,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:58.948533+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:32:58.948533+0000","last_peered":"2024-09-16T10:32:58.948533+0000","last_clean":"2024-09-16T10:32:58.948533+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:32:58.948533+0000","last_undegraded":"2024-09-16T10:32:58.948533+0000","last_fullsized":"2024-09-16T10:32:58.948533+0000","mapping_epoch":45,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2965504,"num_objects":724,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19744,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19674","reported_seq":19838,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:57.458610+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:32:57.458610+0000","last_peered":"2024-09-16T10:32:57.458610+0000","last_clean":"2024-09-16T10:32:57.458610+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:32:57.458610+0000","last_undegraded":"2024-09-16T10:32:57.458610+0000","last_fullsized":"2024-09-16T10:32:57.458610+0000","mapping_epoch":34,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3227648,"num_objects":788,"num_object_clones":0,"num_object_copies":1576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15887,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19798","reported_seq":20054,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:57.550091+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:32:57.550091+0000","last_peered":"2024-09-16T10:32:57.550091+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:32:57.550091+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3543064,"num_objects":866,"num_object_clones":0,"num_object_copies":1732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15855,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3437}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19690","reported_seq":19753,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:57.531974+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:32:57.531974+0000","last_peered":"2024-09-16T10:32:57.531974+0000","last_clean":"2024-09-16T10:32:57.531974+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:32:57.531974+0000","last_undegraded":"2024-09-16T10:32:57.531974+0000","last_fullsized":"2024-09-16T10:32:57.531974+0000","mapping_epoch":34,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3162112,"num_objects":772,"num_object_clones":0,"num_object_copies":1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19690,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19726","reported_seq":19889,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:58.907689+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:32:58.907689+0000","last_peered":"2024-09-16T10:32:58.907689+0000","last_clean":"2024-09-16T10:32:58.907689+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:32:58.907689+0000","last_undegraded":"2024-09-16T10:32:58.907689+0000","last_fullsized":"2024-09-16T10:32:58.907689+0000","mapping_epoch":33,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3190784,"num_objects":779,"num_object_clones":0,"num_object_copies":1558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16368,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19588","reported_seq":19924,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:58.952892+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:32:58.952892+0000","last_peered":"2024-09-16T10:32:58.952892+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:32:58.952892+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3207168,"num_objects":783,"num_object_clones":0,"num_object_copies":1566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16213,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3351}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20180","reported_seq":20590,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:58.928644+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:32:58.928644+0000","last_peered":"2024-09-16T10:32:58.928644+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:32:58.928644+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3031040,"num_objects":740,"num_object_clones":0,"num_object_copies":1480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16422,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3547}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19633","reported_seq":19815,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:58.939237+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:32:58.939237+0000","last_peered":"2024-09-16T10:32:58.939237+0000","last_clean":"2024-09-16T10:32:58.939237+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:32:58.939237+0000","last_undegraded":"2024-09-16T10:32:58.939237+0000","last_fullsized":"2024-09-16T10:32:58.939237+0000","mapping_epoch":33,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3100672,"num_objects":757,"num_object_clones":0,"num_object_copies":1514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15436,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19661","reported_seq":28167,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:58.048552+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:32:58.048552+0000","last_peered":"2024-09-16T10:32:58.048552+0000","last_clean":"2024-09-16T10:32:58.048552+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:32:58.048552+0000","last_undegraded":"2024-09-16T10:32:58.048552+0000","last_fullsized":"2024-09-16T10:32:58.048552+0000","mapping_epoch":43,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3277766,"num_objects":802,"num_object_clones":0,"num_object_copies":1604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19659,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20230","reported_seq":20508,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:58.955323+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:32:58.955323+0000","last_peered":"2024-09-16T10:32:58.955323+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:32:58.955323+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3318726,"num_objects":812,"num_object_clones":0,"num_object_copies":1624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20228,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3525}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.310622+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:28.310622+0000","last_peered":"2024-09-16T10:31:28.310622+0000","last_clean":"2024-09-16T10:31:28.310622+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:28.310622+0000","last_undegraded":"2024-09-16T10:31:28.310622+0000","last_fullsized":"2024-09-16T10:31:28.310622+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19601","reported_seq":19797,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:57.445492+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:32:57.445492+0000","last_peered":"2024-09-16T10:32:57.445492+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:32:57.445492+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3351494,"num_objects":820,"num_object_clones":0,"num_object_copies":1640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19599,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3413}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19711","reported_seq":28200,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:32:57.540987+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:32:57.540987+0000","last_peered":"2024-09-16T10:32:57.540987+0000","last_clean":"2024-09-16T10:32:57.540987+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:32:57.540987+0000","last_undegraded":"2024-09-16T10:32:57.540987+0000","last_fullsized":"2024-09-16T10:32:57.540987+0000","mapping_epoch":47,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3544006,"num_objects":867,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19709,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19260","reported_seq":19514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:58.990932+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:32:58.990932+0000","last_peered":"2024-09-16T10:32:58.990932+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:32:58.990932+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9200","ondisk_log_start":"44'9200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3170787,"num_objects":775,"num_object_clones":0,"num_object_copies":1550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19259,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3300}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20127","reported_seq":20404,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:59.003771+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:32:59.003771+0000","last_peered":"2024-09-16T10:32:59.003771+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:32:59.003771+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3225001,"num_objects":790,"num_object_clones":0,"num_object_copies":1580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20124,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19401","reported_seq":22751,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:59.004642+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:32:59.004642+0000","last_peered":"2024-09-16T10:32:59.004642+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:32:59.004642+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3027910,"num_objects":741,"num_object_clones":0,"num_object_copies":1482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19399,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1633,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1797}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20138","reported_seq":20383,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:32:58.847182+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:32:58.847182+0000","last_peered":"2024-09-16T10:32:58.847182+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:32:58.847182+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3106700,"num_objects":762,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20135,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":51450382,"num_objects":12578,"num_object_clones":0,"num_object_copies":25156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12578,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":293727,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14834,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":313528320,"data_stored":313451529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160762,"ondisk_log_size":160762,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019006,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":498328,"kb_used_data":163984,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93873512,"statfs":{"total":96636764160,"available":96126476288,"internally_reserved":0,"allocated":167919616,"data_stored":167640970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":342355632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.35799999999999998},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":3.4260000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":3.73},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.27600000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":494124,"kb_used_data":39200,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93877716,"statfs":{"total":96636764160,"available":96130781184,"internally_reserved":0,"allocated":40140800,"data_stored":39804346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":0.26500000000000001},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.29299999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.7549999999999999},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":3.7959999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215233,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560728,"kb_used_data":105860,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93811112,"statfs":{"total":96636764160,"available":96062578688,"internally_reserved":0,"allocated":108400640,"data_stored":108129680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.20000000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":1.087}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.54400000000000004},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":106856448,"data_stored":106852835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39419904,"data_stored":39354411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":167251968,"data_stored":167244283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:04.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:04.862 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:04.864 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:04.890 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-16T10:33:04.753934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":42111521,"num_objects":10300,"num_object_clones":0,"num_object_copies":20600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10300,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296009,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14935,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1514764,"kb_used_data":291108,"kb_used_omap":12,"kb_used_meta":1223603,"kb_avail":281600756,"statfs":{"total":289910292480,"available":288359174144,"internally_reserved":0,"allocated":298094592,"data_stored":297208532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1252969474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,8,2,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25333760,"num_objects":-6185,"num_object_clones":0,"num_object_copies":-12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005924"},"pg_stats":[{"pgid":"2.f","version":"48'19862","reported_seq":28330,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.969597+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:03.969597+0000","last_peered":"2024-09-16T10:33:03.969597+0000","last_clean":"2024-09-16T10:33:03.969597+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:03.969597+0000","last_undegraded":"2024-09-16T10:33:03.969597+0000","last_fullsized":"2024-09-16T10:33:03.969597+0000","mapping_epoch":45,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2482176,"num_objects":606,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19862,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19814","reported_seq":19978,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:02.527591+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:02.527591+0000","last_peered":"2024-09-16T10:33:02.527591+0000","last_clean":"2024-09-16T10:33:02.527591+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:02.527591+0000","last_undegraded":"2024-09-16T10:33:02.527591+0000","last_fullsized":"2024-09-16T10:33:02.527591+0000","mapping_epoch":34,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2654208,"num_objects":648,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16027,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19964","reported_seq":20221,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:02.533956+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:02.533956+0000","last_peered":"2024-09-16T10:33:02.533956+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:02.533956+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2863128,"num_objects":700,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16021,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3603}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19838","reported_seq":19901,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:02.553324+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:02.553324+0000","last_peered":"2024-09-16T10:33:02.553324+0000","last_clean":"2024-09-16T10:33:02.553324+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:02.553324+0000","last_undegraded":"2024-09-16T10:33:02.553324+0000","last_fullsized":"2024-09-16T10:33:02.553324+0000","mapping_epoch":34,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":624,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19838,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19866","reported_seq":20029,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:04.027475+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:04.027475+0000","last_peered":"2024-09-16T10:33:04.027475+0000","last_clean":"2024-09-16T10:33:04.027475+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:04.027475+0000","last_undegraded":"2024-09-16T10:33:04.027475+0000","last_fullsized":"2024-09-16T10:33:04.027475+0000","mapping_epoch":33,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2617344,"num_objects":639,"num_object_clones":0,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16508,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19725","reported_seq":20061,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.015129+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:04.015129+0000","last_peered":"2024-09-16T10:33:04.015129+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:04.015129+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2646016,"num_objects":646,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16350,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20308","reported_seq":20720,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.059865+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:04.059865+0000","last_peered":"2024-09-16T10:33:04.059865+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:04.059865+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2506752,"num_objects":612,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16550,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19780","reported_seq":19962,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.998552+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:03.998552+0000","last_peered":"2024-09-16T10:33:03.998552+0000","last_clean":"2024-09-16T10:33:03.998552+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:03.998552+0000","last_undegraded":"2024-09-16T10:33:03.998552+0000","last_fullsized":"2024-09-16T10:33:03.998552+0000","mapping_epoch":33,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2498560,"num_objects":610,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15583,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19799","reported_seq":28309,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.059971+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:03.059971+0000","last_peered":"2024-09-16T10:33:03.059971+0000","last_clean":"2024-09-16T10:33:03.059971+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:03.059971+0000","last_undegraded":"2024-09-16T10:33:03.059971+0000","last_fullsized":"2024-09-16T10:33:03.059971+0000","mapping_epoch":43,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2712518,"num_objects":664,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19797,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20370","reported_seq":20649,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.111494+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:04.111494+0000","last_peered":"2024-09-16T10:33:04.111494+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:04.111494+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2745286,"num_objects":672,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20368,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19755","reported_seq":19954,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:02.489286+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:02.489286+0000","last_peered":"2024-09-16T10:33:02.489286+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:02.489286+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2720710,"num_objects":666,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19753,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3567}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19868","reported_seq":28357,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:02.510039+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:02.510039+0000","last_peered":"2024-09-16T10:33:02.510039+0000","last_clean":"2024-09-16T10:33:02.510039+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:02.510039+0000","last_undegraded":"2024-09-16T10:33:02.510039+0000","last_fullsized":"2024-09-16T10:33:02.510039+0000","mapping_epoch":47,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2900934,"num_objects":710,"num_object_clones":0,"num_object_copies":1420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19866,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19413","reported_seq":19667,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.040972+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:04.040972+0000","last_peered":"2024-09-16T10:33:04.040972+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:04.040972+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2544099,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19412,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20280","reported_seq":20560,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.046464+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:04.046464+0000","last_peered":"2024-09-16T10:33:04.046464+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:04.046464+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2598313,"num_objects":637,"num_object_clones":0,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20277,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19533","reported_seq":23082,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.085263+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:04.085263+0000","last_peered":"2024-09-16T10:33:04.085263+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:04.085263+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487238,"num_objects":609,"num_object_clones":0,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19531,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1733,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20267","reported_seq":20512,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.046076+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:04.046076+0000","last_peered":"2024-09-16T10:33:04.046076+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:04.046076+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2578316,"num_objects":633,"num_object_clones":0,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20264,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":42111502,"num_objects":10298,"num_object_clones":0,"num_object_copies":20596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296007,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14934,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295006208,"data_stored":294929417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019007,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497800,"kb_used_data":159360,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93874040,"statfs":{"total":96636764160,"available":96127016960,"internally_reserved":0,"allocated":163184640,"data_stored":162905994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.24299999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.238},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.9160000000000004}]}]},{"osd":1,"up_from":40,"seq":171798691951,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475192,"kb_used_data":32544,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93896648,"statfs":{"total":96636764160,"available":96150167552,"internally_reserved":0,"allocated":33325056,"data_stored":32988602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":12.798},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.7530000000000001},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":1.0580000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215234,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541772,"kb_used_data":99204,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93830068,"statfs":{"total":96636764160,"available":96081989632,"internally_reserved":0,"allocated":101584896,"data_stored":101313936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":453177000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100151296,"data_stored":100147683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32395264,"data_stored":32329771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162459648,"data_stored":162451963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:04.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:05.463 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:05.466 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:05.483 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-09-16T10:33:04.753934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":42111521,"num_objects":10300,"num_object_clones":0,"num_object_copies":20600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10300,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296009,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14935,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1514764,"kb_used_data":291108,"kb_used_omap":12,"kb_used_meta":1223603,"kb_avail":281600756,"statfs":{"total":289910292480,"available":288359174144,"internally_reserved":0,"allocated":298094592,"data_stored":297208532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1252969474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,8,2,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25333760,"num_objects":-6185,"num_object_clones":0,"num_object_copies":-12370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005924"},"pg_stats":[{"pgid":"2.f","version":"48'19862","reported_seq":28330,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.969597+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:03.969597+0000","last_peered":"2024-09-16T10:33:03.969597+0000","last_clean":"2024-09-16T10:33:03.969597+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:03.969597+0000","last_undegraded":"2024-09-16T10:33:03.969597+0000","last_fullsized":"2024-09-16T10:33:03.969597+0000","mapping_epoch":45,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2482176,"num_objects":606,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19862,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19814","reported_seq":19978,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:02.527591+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:02.527591+0000","last_peered":"2024-09-16T10:33:02.527591+0000","last_clean":"2024-09-16T10:33:02.527591+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:02.527591+0000","last_undegraded":"2024-09-16T10:33:02.527591+0000","last_fullsized":"2024-09-16T10:33:02.527591+0000","mapping_epoch":34,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2654208,"num_objects":648,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16027,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19964","reported_seq":20221,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:02.533956+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:02.533956+0000","last_peered":"2024-09-16T10:33:02.533956+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:02.533956+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2863128,"num_objects":700,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16021,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3603}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19838","reported_seq":19901,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:02.553324+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:02.553324+0000","last_peered":"2024-09-16T10:33:02.553324+0000","last_clean":"2024-09-16T10:33:02.553324+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:02.553324+0000","last_undegraded":"2024-09-16T10:33:02.553324+0000","last_fullsized":"2024-09-16T10:33:02.553324+0000","mapping_epoch":34,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":624,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19838,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19866","reported_seq":20029,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:04.027475+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:04.027475+0000","last_peered":"2024-09-16T10:33:04.027475+0000","last_clean":"2024-09-16T10:33:04.027475+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:04.027475+0000","last_undegraded":"2024-09-16T10:33:04.027475+0000","last_fullsized":"2024-09-16T10:33:04.027475+0000","mapping_epoch":33,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2617344,"num_objects":639,"num_object_clones":0,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16508,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19725","reported_seq":20061,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.015129+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:04.015129+0000","last_peered":"2024-09-16T10:33:04.015129+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:04.015129+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2646016,"num_objects":646,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16350,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20308","reported_seq":20720,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.059865+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:04.059865+0000","last_peered":"2024-09-16T10:33:04.059865+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:04.059865+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2506752,"num_objects":612,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16550,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19780","reported_seq":19962,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.998552+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:03.998552+0000","last_peered":"2024-09-16T10:33:03.998552+0000","last_clean":"2024-09-16T10:33:03.998552+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:03.998552+0000","last_undegraded":"2024-09-16T10:33:03.998552+0000","last_fullsized":"2024-09-16T10:33:03.998552+0000","mapping_epoch":33,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2498560,"num_objects":610,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15583,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19799","reported_seq":28309,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.059971+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:03.059971+0000","last_peered":"2024-09-16T10:33:03.059971+0000","last_clean":"2024-09-16T10:33:03.059971+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:03.059971+0000","last_undegraded":"2024-09-16T10:33:03.059971+0000","last_fullsized":"2024-09-16T10:33:03.059971+0000","mapping_epoch":43,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2712518,"num_objects":664,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19797,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20370","reported_seq":20649,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.111494+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:04.111494+0000","last_peered":"2024-09-16T10:33:04.111494+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:04.111494+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2745286,"num_objects":672,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20368,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19755","reported_seq":19954,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:02.489286+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:02.489286+0000","last_peered":"2024-09-16T10:33:02.489286+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:02.489286+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2720710,"num_objects":666,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19753,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3567}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19868","reported_seq":28357,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:02.510039+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:02.510039+0000","last_peered":"2024-09-16T10:33:02.510039+0000","last_clean":"2024-09-16T10:33:02.510039+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:02.510039+0000","last_undegraded":"2024-09-16T10:33:02.510039+0000","last_fullsized":"2024-09-16T10:33:02.510039+0000","mapping_epoch":47,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2900934,"num_objects":710,"num_object_clones":0,"num_object_copies":1420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19866,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19413","reported_seq":19667,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.040972+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:04.040972+0000","last_peered":"2024-09-16T10:33:04.040972+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:04.040972+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2544099,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19412,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20280","reported_seq":20560,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.046464+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:04.046464+0000","last_peered":"2024-09-16T10:33:04.046464+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:04.046464+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2598313,"num_objects":637,"num_object_clones":0,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20277,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19533","reported_seq":23082,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.085263+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:04.085263+0000","last_peered":"2024-09-16T10:33:04.085263+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:04.085263+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487238,"num_objects":609,"num_object_clones":0,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19531,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1733,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20267","reported_seq":20512,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.046076+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:04.046076+0000","last_peered":"2024-09-16T10:33:04.046076+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:04.046076+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2578316,"num_objects":633,"num_object_clones":0,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20264,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":42111502,"num_objects":10298,"num_object_clones":0,"num_object_copies":20596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296007,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14934,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295006208,"data_stored":294929417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019007,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497800,"kb_used_data":159360,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93874040,"statfs":{"total":96636764160,"available":96127016960,"internally_reserved":0,"allocated":163184640,"data_stored":162905994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.24299999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.238},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.9160000000000004}]}]},{"osd":1,"up_from":40,"seq":171798691951,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475192,"kb_used_data":32544,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93896648,"statfs":{"total":96636764160,"available":96150167552,"internally_reserved":0,"allocated":33325056,"data_stored":32988602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":12.798},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.7530000000000001},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":1.0580000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215234,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541772,"kb_used_data":99204,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93830068,"statfs":{"total":96636764160,"available":96081989632,"internally_reserved":0,"allocated":101584896,"data_stored":101313936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":453177000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100151296,"data_stored":100147683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32395264,"data_stored":32329771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162459648,"data_stored":162451963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:05.485 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:05.485 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'19964", 'reported_seq': 20221, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:02.533956+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:33:02.533956+0000', 'last_peered': '2024-09-16T10:33:02.533956+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:02.533956+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'9900", 'ondisk_log_start': "44'9900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2863128, 'num_objects': 700, 'num_object_clones': 0, 'num_object_copies': 1400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 700, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16021, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3603}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:05.485 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:05.485 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'19725", 'reported_seq': 20061, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:04.015129+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:04.015129+0000', 'last_peered': '2024-09-16T10:33:04.015129+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:04.015129+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'9700", 'ondisk_log_start': "44'9700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2646016, 'num_objects': 646, 'num_object_clones': 0, 'num_object_copies': 1292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16350, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3488}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:05.485 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:05.485 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20308", 'reported_seq': 20720, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:04.059865+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:04.059865+0000', 'last_peered': '2024-09-16T10:33:04.059865+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:04.059865+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2506752, 'num_objects': 612, 'num_object_clones': 0, 'num_object_copies': 1224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 612, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16550, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3675}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:05.485 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:05.485 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'20370", 'reported_seq': 20649, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:04.111494+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:04.111494+0000', 'last_peered': '2024-09-16T10:33:04.111494+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:04.111494+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2745286, 'num_objects': 672, 'num_object_clones': 0, 'num_object_copies': 1344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20368, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3665}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:05.486 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:05.486 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'19755", 'reported_seq': 19954, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:02.489286+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:02.489286+0000', 'last_peered': '2024-09-16T10:33:02.489286+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:02.489286+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'9700", 'ondisk_log_start': "44'9700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2720710, 'num_objects': 666, 'num_object_clones': 0, 'num_object_copies': 1332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 666, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19753, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3567}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:05.486 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:05.486 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'19413", 'reported_seq': 19667, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:04.040972+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:04.040972+0000', 'last_peered': '2024-09-16T10:33:04.040972+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:04.040972+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'9400", 'ondisk_log_start': "44'9400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2544099, 'num_objects': 622, 'num_object_clones': 0, 'num_object_copies': 1244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19412, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3453}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:05.486 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:05.486 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20280", 'reported_seq': 20560, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:04.046464+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:04.046464+0000', 'last_peered': '2024-09-16T10:33:04.046464+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:04.046464+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10200", 'ondisk_log_start': "44'10200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2598313, 'num_objects': 637, 'num_object_clones': 0, 'num_object_copies': 1274, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20277, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3630}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:05.486 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:33:05.486 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'19533", 'reported_seq': 23082, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:04.085263+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:33:04.085263+0000', 'last_peered': '2024-09-16T10:33:04.085263+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:33:04.085263+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'9500", 'ondisk_log_start': "44'9500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2487238, 'num_objects': 609, 'num_object_clones': 0, 'num_object_copies': 1218, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2438, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19531, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1733, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1829}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:05.486 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:05.486 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20267", 'reported_seq': 20512, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:04.046076+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:04.046076+0000', 'last_peered': '2024-09-16T10:33:04.046076+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:04.046076+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10200", 'ondisk_log_start': "44'10200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2578316, 'num_objects': 633, 'num_object_clones': 0, 'num_object_copies': 1266, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20264, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3689}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:05.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:06.168 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:06.168 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36412/20600 objects degraded (176.757%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":674,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":10300,"data_bytes":42111521,"bytes_used":1551118336,"bytes_avail":288359174144,"bytes_total":289910292480,"degraded_objects":36412,"degraded_total":20600,"degraded_ratio":1.7675728155339805,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":277,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":515},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:06.196 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:06.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:06.764 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:06.764 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:06.791 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-16T10:33:06.754448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":42111521,"num_objects":10300,"num_object_clones":0,"num_object_copies":20600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10300,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296009,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14935,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1514764,"kb_used_data":291108,"kb_used_omap":12,"kb_used_meta":1223603,"kb_avail":281600756,"statfs":{"total":289910292480,"available":288359174144,"internally_reserved":0,"allocated":298094592,"data_stored":297208532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1252969474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,3,8,2,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18739200,"num_objects":-4575,"num_object_clones":0,"num_object_copies":-9150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4575,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005967"},"pg_stats":[{"pgid":"2.f","version":"48'19862","reported_seq":28330,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.969597+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:03.969597+0000","last_peered":"2024-09-16T10:33:03.969597+0000","last_clean":"2024-09-16T10:33:03.969597+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:03.969597+0000","last_undegraded":"2024-09-16T10:33:03.969597+0000","last_fullsized":"2024-09-16T10:33:03.969597+0000","mapping_epoch":45,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2482176,"num_objects":606,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19862,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19814","reported_seq":19978,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:02.527591+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:02.527591+0000","last_peered":"2024-09-16T10:33:02.527591+0000","last_clean":"2024-09-16T10:33:02.527591+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:02.527591+0000","last_undegraded":"2024-09-16T10:33:02.527591+0000","last_fullsized":"2024-09-16T10:33:02.527591+0000","mapping_epoch":34,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2654208,"num_objects":648,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16027,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'19964","reported_seq":20221,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:02.533956+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:02.533956+0000","last_peered":"2024-09-16T10:33:02.533956+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:02.533956+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2863128,"num_objects":700,"num_object_clones":0,"num_object_copies":1400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16021,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3603}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19838","reported_seq":19901,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:02.553324+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:02.553324+0000","last_peered":"2024-09-16T10:33:02.553324+0000","last_clean":"2024-09-16T10:33:02.553324+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:02.553324+0000","last_undegraded":"2024-09-16T10:33:02.553324+0000","last_fullsized":"2024-09-16T10:33:02.553324+0000","mapping_epoch":34,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":624,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19838,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19866","reported_seq":20029,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:04.027475+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:04.027475+0000","last_peered":"2024-09-16T10:33:04.027475+0000","last_clean":"2024-09-16T10:33:04.027475+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:04.027475+0000","last_undegraded":"2024-09-16T10:33:04.027475+0000","last_fullsized":"2024-09-16T10:33:04.027475+0000","mapping_epoch":33,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2617344,"num_objects":639,"num_object_clones":0,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16508,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19725","reported_seq":20061,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.015129+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:04.015129+0000","last_peered":"2024-09-16T10:33:04.015129+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:04.015129+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2646016,"num_objects":646,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16350,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20308","reported_seq":20720,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.059865+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:04.059865+0000","last_peered":"2024-09-16T10:33:04.059865+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:04.059865+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2506752,"num_objects":612,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16550,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19780","reported_seq":19962,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.998552+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:03.998552+0000","last_peered":"2024-09-16T10:33:03.998552+0000","last_clean":"2024-09-16T10:33:03.998552+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:03.998552+0000","last_undegraded":"2024-09-16T10:33:03.998552+0000","last_fullsized":"2024-09-16T10:33:03.998552+0000","mapping_epoch":33,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2498560,"num_objects":610,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15583,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19799","reported_seq":28309,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.059971+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:03.059971+0000","last_peered":"2024-09-16T10:33:03.059971+0000","last_clean":"2024-09-16T10:33:03.059971+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:03.059971+0000","last_undegraded":"2024-09-16T10:33:03.059971+0000","last_fullsized":"2024-09-16T10:33:03.059971+0000","mapping_epoch":43,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2712518,"num_objects":664,"num_object_clones":0,"num_object_copies":1328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19797,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20370","reported_seq":20649,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.111494+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:04.111494+0000","last_peered":"2024-09-16T10:33:04.111494+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:04.111494+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2745286,"num_objects":672,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20368,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19755","reported_seq":19954,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:02.489286+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:02.489286+0000","last_peered":"2024-09-16T10:33:02.489286+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:02.489286+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2720710,"num_objects":666,"num_object_clones":0,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19753,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3567}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'19868","reported_seq":28357,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:02.510039+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:02.510039+0000","last_peered":"2024-09-16T10:33:02.510039+0000","last_clean":"2024-09-16T10:33:02.510039+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:02.510039+0000","last_undegraded":"2024-09-16T10:33:02.510039+0000","last_fullsized":"2024-09-16T10:33:02.510039+0000","mapping_epoch":47,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2900934,"num_objects":710,"num_object_clones":0,"num_object_copies":1420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19866,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19413","reported_seq":19667,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.040972+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:04.040972+0000","last_peered":"2024-09-16T10:33:04.040972+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:04.040972+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2544099,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19412,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20280","reported_seq":20560,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.046464+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:04.046464+0000","last_peered":"2024-09-16T10:33:04.046464+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:04.046464+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2598313,"num_objects":637,"num_object_clones":0,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20277,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19533","reported_seq":23082,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.085263+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:04.085263+0000","last_peered":"2024-09-16T10:33:04.085263+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:04.085263+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487238,"num_objects":609,"num_object_clones":0,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19531,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1733,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20267","reported_seq":20512,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.046076+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:04.046076+0000","last_peered":"2024-09-16T10:33:04.046076+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:04.046076+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2578316,"num_objects":633,"num_object_clones":0,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20264,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":42111502,"num_objects":10298,"num_object_clones":0,"num_object_copies":20596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10298,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296007,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14934,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295006208,"data_stored":294929417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160842,"ondisk_log_size":160842,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019007,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497800,"kb_used_data":159360,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93874040,"statfs":{"total":96636764160,"available":96127016960,"internally_reserved":0,"allocated":163184640,"data_stored":162905994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":0.24299999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.238},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":7.9160000000000004}]}]},{"osd":1,"up_from":40,"seq":171798691951,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475192,"kb_used_data":32544,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93896648,"statfs":{"total":96636764160,"available":96150167552,"internally_reserved":0,"allocated":33325056,"data_stored":32988602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":12.798},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.70999999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":4.7530000000000001},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":1.0580000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215234,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541772,"kb_used_data":99204,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93830068,"statfs":{"total":96636764160,"available":96081989632,"internally_reserved":0,"allocated":101584896,"data_stored":101313936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":453177000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100151296,"data_stored":100147683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32395264,"data_stored":32329771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162459648,"data_stored":162451963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:09.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:10.353 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:10.356 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:10.378 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-16T10:33:08.754836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":38519329,"num_objects":9423,"num_object_clones":0,"num_object_copies":18846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9423,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296886,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14935,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1515484,"kb_used_data":279552,"kb_used_omap":12,"kb_used_meta":1235891,"kb_avail":281600036,"statfs":{"total":289910292480,"available":288358436864,"internally_reserved":0,"allocated":286261248,"data_stored":285375188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1265552386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,6,6,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22331392,"num_objects":-5452,"num_object_clones":0,"num_object_copies":-10904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5452,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":192,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005872"},"pg_stats":[{"pgid":"2.f","version":"48'19862","reported_seq":28330,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.969597+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:03.969597+0000","last_peered":"2024-09-16T10:33:03.969597+0000","last_clean":"2024-09-16T10:33:03.969597+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:03.969597+0000","last_undegraded":"2024-09-16T10:33:03.969597+0000","last_fullsized":"2024-09-16T10:33:03.969597+0000","mapping_epoch":45,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2482176,"num_objects":606,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19862,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19940","reported_seq":20104,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:07.586262+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:07.586262+0000","last_peered":"2024-09-16T10:33:07.586262+0000","last_clean":"2024-09-16T10:33:07.586262+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:07.586262+0000","last_undegraded":"2024-09-16T10:33:07.586262+0000","last_fullsized":"2024-09-16T10:33:07.586262+0000","mapping_epoch":34,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2138112,"num_objects":522,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16153,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20121","reported_seq":20378,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:07.526697+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:07.526697+0000","last_peered":"2024-09-16T10:33:07.526697+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:07.526697+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2220056,"num_objects":543,"num_object_clones":0,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16178,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19976","reported_seq":20039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:07.487313+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:07.487313+0000","last_peered":"2024-09-16T10:33:07.487313+0000","last_clean":"2024-09-16T10:33:07.487313+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:07.487313+0000","last_undegraded":"2024-09-16T10:33:07.487313+0000","last_fullsized":"2024-09-16T10:33:07.487313+0000","mapping_epoch":34,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1990656,"num_objects":486,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19976,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'19866","reported_seq":20029,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:04.027475+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:04.027475+0000","last_peered":"2024-09-16T10:33:04.027475+0000","last_clean":"2024-09-16T10:33:04.027475+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:04.027475+0000","last_undegraded":"2024-09-16T10:33:04.027475+0000","last_fullsized":"2024-09-16T10:33:04.027475+0000","mapping_epoch":33,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2617344,"num_objects":639,"num_object_clones":0,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16508,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19725","reported_seq":20061,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.015129+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:04.015129+0000","last_peered":"2024-09-16T10:33:04.015129+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:04.015129+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2646016,"num_objects":646,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16350,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3488}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20308","reported_seq":20720,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.059865+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:04.059865+0000","last_peered":"2024-09-16T10:33:04.059865+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:04.059865+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2506752,"num_objects":612,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16550,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19780","reported_seq":19962,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:03.998552+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:03.998552+0000","last_peered":"2024-09-16T10:33:03.998552+0000","last_clean":"2024-09-16T10:33:03.998552+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:03.998552+0000","last_undegraded":"2024-09-16T10:33:03.998552+0000","last_fullsized":"2024-09-16T10:33:03.998552+0000","mapping_epoch":33,"log_start":"44'9700","ondisk_log_start":"44'9700","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2498560,"num_objects":610,"num_object_clones":0,"num_object_copies":1220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15583,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19956","reported_seq":28466,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:08.236292+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:08.236292+0000","last_peered":"2024-09-16T10:33:08.236292+0000","last_clean":"2024-09-16T10:33:08.236292+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:08.236292+0000","last_undegraded":"2024-09-16T10:33:08.236292+0000","last_fullsized":"2024-09-16T10:33:08.236292+0000","mapping_epoch":43,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2069446,"num_objects":507,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19954,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20370","reported_seq":20649,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.111494+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:04.111494+0000","last_peered":"2024-09-16T10:33:04.111494+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:04.111494+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2745286,"num_objects":672,"num_object_clones":0,"num_object_copies":1344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20368,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3665}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19905","reported_seq":20104,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:07.520428+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:07.520428+0000","last_peered":"2024-09-16T10:33:07.520428+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:07.520428+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2106310,"num_objects":516,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19903,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3717}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20017","reported_seq":28506,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:07.480892+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:07.480892+0000","last_peered":"2024-09-16T10:33:07.480892+0000","last_clean":"2024-09-16T10:33:07.480892+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:07.480892+0000","last_undegraded":"2024-09-16T10:33:07.480892+0000","last_fullsized":"2024-09-16T10:33:07.480892+0000","mapping_epoch":47,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2290630,"num_objects":561,"num_object_clones":0,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20015,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19413","reported_seq":19667,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.040972+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:04.040972+0000","last_peered":"2024-09-16T10:33:04.040972+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:04.040972+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9400","ondisk_log_start":"44'9400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2544099,"num_objects":622,"num_object_clones":0,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19412,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3453}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20280","reported_seq":20560,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.046464+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:04.046464+0000","last_peered":"2024-09-16T10:33:04.046464+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:04.046464+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2598313,"num_objects":637,"num_object_clones":0,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20277,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3630}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19533","reported_seq":23082,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.085263+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:04.085263+0000","last_peered":"2024-09-16T10:33:04.085263+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:04.085263+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487238,"num_objects":609,"num_object_clones":0,"num_object_copies":1218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19531,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1733,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1829}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20267","reported_seq":20512,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:04.046076+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:04.046076+0000","last_peered":"2024-09-16T10:33:04.046076+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:04.046076+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2578316,"num_objects":633,"num_object_clones":0,"num_object_copies":1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20264,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3689}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":38519310,"num_objects":9421,"num_object_clones":0,"num_object_copies":18842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9421,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":296884,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14934,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282984448,"data_stored":282907657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160719,"ondisk_log_size":160719,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019008,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497212,"kb_used_data":154676,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93874628,"statfs":{"total":96636764160,"available":96127619072,"internally_reserved":0,"allocated":158388224,"data_stored":158109578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":8.2799999999999994},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":1.173}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.8069999999999999},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":10.273}]}]},{"osd":1,"up_from":40,"seq":171798691952,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476500,"kb_used_data":25672,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93895340,"statfs":{"total":96636764160,"available":96148828160,"internally_reserved":0,"allocated":26288128,"data_stored":25951674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":1.9139999999999999},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":2.0649999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":11.253},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":10.656000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215234,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541772,"kb_used_data":99204,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93830068,"statfs":{"total":96636764160,"available":96081989632,"internally_reserved":0,"allocated":101584896,"data_stored":101313936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":453177000},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":0.61899999999999999},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100151296,"data_stored":100147683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25174016,"data_stored":25108523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157659136,"data_stored":157651451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:10.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:10.929 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:10.931 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:10.956 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-16T10:33:10.755326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":32563745,"num_objects":7969,"num_object_clones":0,"num_object_copies":15938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7969,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":298340,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15024,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1516904,"kb_used_data":272772,"kb_used_omap":12,"kb_used_meta":1244083,"kb_avail":281598616,"statfs":{"total":289910292480,"available":288356982784,"internally_reserved":0,"allocated":279318528,"data_stored":278432468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1273940994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,1,4,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24862720,"num_objects":-6070,"num_object_clones":0,"num_object_copies":-12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6070,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005933"},"pg_stats":[{"pgid":"2.f","version":"48'19999","reported_seq":28467,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:09.079901+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:09.079901+0000","last_peered":"2024-09-16T10:33:09.079901+0000","last_clean":"2024-09-16T10:33:09.079901+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:09.079901+0000","last_undegraded":"2024-09-16T10:33:09.079901+0000","last_fullsized":"2024-09-16T10:33:09.079901+0000","mapping_epoch":45,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1921024,"num_objects":469,"num_object_clones":0,"num_object_copies":938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19940","reported_seq":20104,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:07.586262+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:07.586262+0000","last_peered":"2024-09-16T10:33:07.586262+0000","last_clean":"2024-09-16T10:33:07.586262+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:07.586262+0000","last_undegraded":"2024-09-16T10:33:07.586262+0000","last_fullsized":"2024-09-16T10:33:07.586262+0000","mapping_epoch":34,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2138112,"num_objects":522,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16153,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20121","reported_seq":20378,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:07.526697+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:07.526697+0000","last_peered":"2024-09-16T10:33:07.526697+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:07.526697+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2220056,"num_objects":543,"num_object_clones":0,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16178,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19976","reported_seq":20039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:07.487313+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:07.487313+0000","last_peered":"2024-09-16T10:33:07.487313+0000","last_clean":"2024-09-16T10:33:07.487313+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:07.487313+0000","last_undegraded":"2024-09-16T10:33:07.487313+0000","last_fullsized":"2024-09-16T10:33:07.487313+0000","mapping_epoch":34,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1990656,"num_objects":486,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19976,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20020","reported_seq":20183,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:09.082732+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:09.082732+0000","last_peered":"2024-09-16T10:33:09.082732+0000","last_clean":"2024-09-16T10:33:09.082732+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:09.082732+0000","last_undegraded":"2024-09-16T10:33:09.082732+0000","last_fullsized":"2024-09-16T10:33:09.082732+0000","mapping_epoch":33,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1986560,"num_objects":485,"num_object_clones":0,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16662,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19862","reported_seq":20198,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.038187+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:09.038187+0000","last_peered":"2024-09-16T10:33:09.038187+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:09.038187+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2084864,"num_objects":509,"num_object_clones":0,"num_object_copies":1018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16487,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3625}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20448","reported_seq":20860,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.091599+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:09.091599+0000","last_peered":"2024-09-16T10:33:09.091599+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:09.091599+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933312,"num_objects":472,"num_object_clones":0,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16690,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19924","reported_seq":20106,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:09.114901+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:09.114901+0000","last_peered":"2024-09-16T10:33:09.114901+0000","last_clean":"2024-09-16T10:33:09.114901+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:09.114901+0000","last_undegraded":"2024-09-16T10:33:09.114901+0000","last_fullsized":"2024-09-16T10:33:09.114901+0000","mapping_epoch":33,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1908736,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15727,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19956","reported_seq":28466,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:08.236292+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:08.236292+0000","last_peered":"2024-09-16T10:33:08.236292+0000","last_clean":"2024-09-16T10:33:08.236292+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:08.236292+0000","last_undegraded":"2024-09-16T10:33:08.236292+0000","last_fullsized":"2024-09-16T10:33:08.236292+0000","mapping_epoch":43,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2069446,"num_objects":507,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19954,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20530","reported_seq":20808,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.102004+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:09.102004+0000","last_peered":"2024-09-16T10:33:09.102004+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:09.102004+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2089926,"num_objects":512,"num_object_clones":0,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20528,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3825}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19905","reported_seq":20104,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:07.520428+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:07.520428+0000","last_peered":"2024-09-16T10:33:07.520428+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:07.520428+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2106310,"num_objects":516,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19903,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3717}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20017","reported_seq":28506,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:07.480892+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:07.480892+0000","last_peered":"2024-09-16T10:33:07.480892+0000","last_clean":"2024-09-16T10:33:07.480892+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:07.480892+0000","last_undegraded":"2024-09-16T10:33:07.480892+0000","last_fullsized":"2024-09-16T10:33:07.480892+0000","mapping_epoch":47,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2290630,"num_objects":561,"num_object_clones":0,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20015,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19553","reported_seq":19807,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.028434+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:09.028434+0000","last_peered":"2024-09-16T10:33:09.028434+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:09.028434+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1970659,"num_objects":482,"num_object_clones":0,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19552,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20424","reported_seq":20704,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.063346+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:09.063346+0000","last_peered":"2024-09-16T10:33:09.063346+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:09.063346+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2008489,"num_objects":493,"num_object_clones":0,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20421,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19672","reported_seq":23398,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.118274+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:09.118274+0000","last_peered":"2024-09-16T10:33:09.118274+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:09.118274+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1917894,"num_objects":470,"num_object_clones":0,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19670,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1822,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20426","reported_seq":20671,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.099655+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:09.099655+0000","last_peered":"2024-09-16T10:33:09.099655+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:09.099655+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1927052,"num_objects":474,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20423,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":32563726,"num_objects":7967,"num_object_clones":0,"num_object_copies":15934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7967,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":298338,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15023,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275955712,"data_stored":275878921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160673,"ondisk_log_size":160673,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019008,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497212,"kb_used_data":154676,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93874628,"statfs":{"total":96636764160,"available":96127619072,"internally_reserved":0,"allocated":158388224,"data_stored":158109578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":8.2799999999999994},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":1.173}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.8069999999999999},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":10.273}]}]},{"osd":1,"up_from":40,"seq":171798691952,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476500,"kb_used_data":25672,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93895340,"statfs":{"total":96636764160,"available":96148828160,"internally_reserved":0,"allocated":26288128,"data_stored":25951674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":1.9139999999999999},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":2.0649999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":11.253},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":10.656000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215235,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543192,"kb_used_data":92424,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93828648,"statfs":{"total":96636764160,"available":96080535552,"internally_reserved":0,"allocated":94642176,"data_stored":94371216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":461565608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":2.0739999999999998},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":5.7210000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":6.7939999999999996},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.823}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93122560,"data_stored":93118947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25174016,"data_stored":25108523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157659136,"data_stored":157651451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:10.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:10.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20121", 'reported_seq': 20378, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:07.526697+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:33:07.526697+0000', 'last_peered': '2024-09-16T10:33:07.526697+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:07.526697+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10100", 'ondisk_log_start': "44'10100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2220056, 'num_objects': 543, 'num_object_clones': 0, 'num_object_copies': 1086, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 543, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16178, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3760}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:10.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:10.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'19862", 'reported_seq': 20198, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:09.038187+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:09.038187+0000', 'last_peered': '2024-09-16T10:33:09.038187+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:09.038187+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'9800", 'ondisk_log_start': "44'9800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2084864, 'num_objects': 509, 'num_object_clones': 0, 'num_object_copies': 1018, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16487, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3625}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:10.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:10.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20448", 'reported_seq': 20860, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:09.091599+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:09.091599+0000', 'last_peered': '2024-09-16T10:33:09.091599+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:09.091599+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1933312, 'num_objects': 472, 'num_object_clones': 0, 'num_object_copies': 944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16690, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:10.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:10.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'20530", 'reported_seq': 20808, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:09.102004+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:09.102004+0000', 'last_peered': '2024-09-16T10:33:09.102004+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:09.102004+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2089926, 'num_objects': 512, 'num_object_clones': 0, 'num_object_copies': 1024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20528, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3825}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:10.958 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:10.959 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'19905", 'reported_seq': 20104, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:07.520428+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:07.520428+0000', 'last_peered': '2024-09-16T10:33:07.520428+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:07.520428+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'9900", 'ondisk_log_start': "44'9900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2106310, 'num_objects': 516, 'num_object_clones': 0, 'num_object_copies': 1032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19903, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3717}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:10.959 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:10.959 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'19553", 'reported_seq': 19807, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:09.028434+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:09.028434+0000', 'last_peered': '2024-09-16T10:33:09.028434+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:09.028434+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'9500", 'ondisk_log_start': "44'9500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1970659, 'num_objects': 482, 'num_object_clones': 0, 'num_object_copies': 964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19552, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:10.959 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:10.959 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20424", 'reported_seq': 20704, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:09.063346+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:09.063346+0000', 'last_peered': '2024-09-16T10:33:09.063346+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:09.063346+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2008489, 'num_objects': 493, 'num_object_clones': 0, 'num_object_copies': 986, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 493, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20421, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3774}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:10.959 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:33:10.959 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'19672", 'reported_seq': 23398, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:09.118274+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:33:09.118274+0000', 'last_peered': '2024-09-16T10:33:09.118274+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:33:09.118274+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'9600", 'ondisk_log_start': "44'9600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1917894, 'num_objects': 470, 'num_object_clones': 0, 'num_object_copies': 940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2349, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19670, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1822, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1879}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:10.959 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:10.959 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20426", 'reported_seq': 20671, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:09.099655+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:09.099655+0000', 'last_peered': '2024-09-16T10:33:09.099655+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:09.099655+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1927052, 'num_objects': 474, 'num_object_clones': 0, 'num_object_copies': 948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20423, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3848}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:10.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:11.632 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:11.632 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36323/15938 objects degraded (227.902%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":680,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":7969,"data_bytes":32563745,"bytes_used":1553309696,"bytes_avail":288356982784,"bytes_total":289910292480,"degraded_objects":36323,"degraded_total":15938,"degraded_ratio":2.2790186974526287,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":281,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":505},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:11.657 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:11.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:12.217 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:12.218 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:12.246 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-16T10:33:10.755326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":32563745,"num_objects":7969,"num_object_clones":0,"num_object_copies":15938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7969,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":298340,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15024,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160676,"ondisk_log_size":160676,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1516904,"kb_used_data":272772,"kb_used_omap":12,"kb_used_meta":1244083,"kb_avail":281598616,"statfs":{"total":289910292480,"available":288356982784,"internally_reserved":0,"allocated":279318528,"data_stored":278432468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1273940994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,7,1,4,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24862720,"num_objects":-6070,"num_object_clones":0,"num_object_copies":-12140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6070,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005933"},"pg_stats":[{"pgid":"2.f","version":"48'19999","reported_seq":28467,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:09.079901+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:09.079901+0000","last_peered":"2024-09-16T10:33:09.079901+0000","last_clean":"2024-09-16T10:33:09.079901+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:09.079901+0000","last_undegraded":"2024-09-16T10:33:09.079901+0000","last_fullsized":"2024-09-16T10:33:09.079901+0000","mapping_epoch":45,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1921024,"num_objects":469,"num_object_clones":0,"num_object_copies":938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19999,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'19940","reported_seq":20104,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:07.586262+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:07.586262+0000","last_peered":"2024-09-16T10:33:07.586262+0000","last_clean":"2024-09-16T10:33:07.586262+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:07.586262+0000","last_undegraded":"2024-09-16T10:33:07.586262+0000","last_fullsized":"2024-09-16T10:33:07.586262+0000","mapping_epoch":34,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2138112,"num_objects":522,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16153,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20121","reported_seq":20378,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:07.526697+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:07.526697+0000","last_peered":"2024-09-16T10:33:07.526697+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:07.526697+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2220056,"num_objects":543,"num_object_clones":0,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":543,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16178,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'19976","reported_seq":20039,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:07.487313+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:07.487313+0000","last_peered":"2024-09-16T10:33:07.487313+0000","last_clean":"2024-09-16T10:33:07.487313+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:07.487313+0000","last_undegraded":"2024-09-16T10:33:07.487313+0000","last_fullsized":"2024-09-16T10:33:07.487313+0000","mapping_epoch":34,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1990656,"num_objects":486,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19976,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20020","reported_seq":20183,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:09.082732+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:09.082732+0000","last_peered":"2024-09-16T10:33:09.082732+0000","last_clean":"2024-09-16T10:33:09.082732+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:09.082732+0000","last_undegraded":"2024-09-16T10:33:09.082732+0000","last_fullsized":"2024-09-16T10:33:09.082732+0000","mapping_epoch":33,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1986560,"num_objects":485,"num_object_clones":0,"num_object_copies":970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16662,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'19862","reported_seq":20198,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.038187+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:09.038187+0000","last_peered":"2024-09-16T10:33:09.038187+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:09.038187+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2084864,"num_objects":509,"num_object_clones":0,"num_object_copies":1018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16487,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3625}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20448","reported_seq":20860,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.091599+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:09.091599+0000","last_peered":"2024-09-16T10:33:09.091599+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:09.091599+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1933312,"num_objects":472,"num_object_clones":0,"num_object_copies":944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16690,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3815}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'19924","reported_seq":20106,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:09.114901+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:09.114901+0000","last_peered":"2024-09-16T10:33:09.114901+0000","last_clean":"2024-09-16T10:33:09.114901+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:09.114901+0000","last_undegraded":"2024-09-16T10:33:09.114901+0000","last_fullsized":"2024-09-16T10:33:09.114901+0000","mapping_epoch":33,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1908736,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15727,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'19956","reported_seq":28466,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:08.236292+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:08.236292+0000","last_peered":"2024-09-16T10:33:08.236292+0000","last_clean":"2024-09-16T10:33:08.236292+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:08.236292+0000","last_undegraded":"2024-09-16T10:33:08.236292+0000","last_fullsized":"2024-09-16T10:33:08.236292+0000","mapping_epoch":43,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2069446,"num_objects":507,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19954,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20530","reported_seq":20808,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.102004+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:09.102004+0000","last_peered":"2024-09-16T10:33:09.102004+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:09.102004+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2089926,"num_objects":512,"num_object_clones":0,"num_object_copies":1024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20528,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3825}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'19905","reported_seq":20104,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:07.520428+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:07.520428+0000","last_peered":"2024-09-16T10:33:07.520428+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:07.520428+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2106310,"num_objects":516,"num_object_clones":0,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19903,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3717}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20017","reported_seq":28506,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:07.480892+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:07.480892+0000","last_peered":"2024-09-16T10:33:07.480892+0000","last_clean":"2024-09-16T10:33:07.480892+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:07.480892+0000","last_undegraded":"2024-09-16T10:33:07.480892+0000","last_fullsized":"2024-09-16T10:33:07.480892+0000","mapping_epoch":47,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2290630,"num_objects":561,"num_object_clones":0,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20015,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19553","reported_seq":19807,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.028434+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:09.028434+0000","last_peered":"2024-09-16T10:33:09.028434+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:09.028434+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9500","ondisk_log_start":"44'9500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1970659,"num_objects":482,"num_object_clones":0,"num_object_copies":964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19552,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3593}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20424","reported_seq":20704,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.063346+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:09.063346+0000","last_peered":"2024-09-16T10:33:09.063346+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:09.063346+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2008489,"num_objects":493,"num_object_clones":0,"num_object_copies":986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20421,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3774}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19672","reported_seq":23398,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.118274+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:09.118274+0000","last_peered":"2024-09-16T10:33:09.118274+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:09.118274+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1917894,"num_objects":470,"num_object_clones":0,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19670,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1822,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1879}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20426","reported_seq":20671,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:09.099655+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:09.099655+0000","last_peered":"2024-09-16T10:33:09.099655+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:09.099655+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1927052,"num_objects":474,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20423,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3848}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":32563726,"num_objects":7967,"num_object_clones":0,"num_object_copies":15934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36323,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7967,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":298338,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15023,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":275955712,"data_stored":275878921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160673,"ondisk_log_size":160673,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019008,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497212,"kb_used_data":154676,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93874628,"statfs":{"total":96636764160,"available":96127619072,"internally_reserved":0,"allocated":158388224,"data_stored":158109578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.286,"15min":1.4930000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.184},"max":{"1min":13.170999999999999,"5min":13.74,"15min":13.74},"last":8.2799999999999994},{"interface":"front","average":{"1min":1.873,"5min":1.958,"15min":1.629},"min":{"1min":0.24099999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":8.6859999999999999,"5min":13.917,"15min":13.917},"last":1.173}]},{"osd":1,"last update":"Mon Sep 16 10:32:07 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":3.145,"15min":2.9020000000000001},"min":{"1min":0.253,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.2669999999999995,"5min":15.157999999999999,"15min":15.157999999999999},"last":2.8069999999999999},{"interface":"front","average":{"1min":2.9060000000000001,"5min":3.5289999999999999,"15min":2.706},"min":{"1min":0.248,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.7400000000000002,"5min":17.914000000000001,"15min":17.914000000000001},"last":10.273}]}]},{"osd":1,"up_from":40,"seq":171798691952,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":476500,"kb_used_data":25672,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93895340,"statfs":{"total":96636764160,"available":96148828160,"internally_reserved":0,"allocated":26288128,"data_stored":25951674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":1.9139999999999999},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":2.0649999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":11.253},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":10.656000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215235,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":543192,"kb_used_data":92424,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93828648,"statfs":{"total":96636764160,"available":96080535552,"internally_reserved":0,"allocated":94642176,"data_stored":94371216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":461565608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,2,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":2.0739999999999998},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":5.7210000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":6.7939999999999996},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":2.823}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":93122560,"data_stored":93118947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":25174016,"data_stored":25108523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157659136,"data_stored":157651451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:15.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:15.789 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:15.791 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:15.813 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-16T10:33:14.756269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23208481,"num_objects":5685,"num_object_clones":0,"num_object_copies":11370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":300624,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15104,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161060,"ondisk_log_size":161060,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1510540,"kb_used_data":254108,"kb_used_omap":12,"kb_used_meta":1256371,"kb_avail":281604980,"statfs":{"total":289910292480,"available":288363499520,"internally_reserved":0,"allocated":260206592,"data_stored":259320532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1286523906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,12,5,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25108480,"num_objects":-6130,"num_object_clones":0,"num_object_copies":-12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005883"},"pg_stats":[{"pgid":"2.f","version":"48'20131","reported_seq":28599,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:14.125210+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:14.125210+0000","last_peered":"2024-09-16T10:33:14.125210+0000","last_clean":"2024-09-16T10:33:14.125210+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:14.125210+0000","last_undegraded":"2024-09-16T10:33:14.125210+0000","last_fullsized":"2024-09-16T10:33:14.125210+0000","mapping_epoch":45,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1380352,"num_objects":337,"num_object_clones":0,"num_object_copies":674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20131,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20079","reported_seq":20243,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:12.494159+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:12.494159+0000","last_peered":"2024-09-16T10:33:12.494159+0000","last_clean":"2024-09-16T10:33:12.494159+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:12.494159+0000","last_undegraded":"2024-09-16T10:33:12.494159+0000","last_fullsized":"2024-09-16T10:33:12.494159+0000","mapping_epoch":34,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1568768,"num_objects":383,"num_object_clones":0,"num_object_copies":766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16292,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20286","reported_seq":20542,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:12.558276+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:12.558276+0000","last_peered":"2024-09-16T10:33:12.558276+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:12.558276+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1544216,"num_objects":378,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16343,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3925}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20099","reported_seq":20162,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:12.554798+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:12.554798+0000","last_peered":"2024-09-16T10:33:12.554798+0000","last_clean":"2024-09-16T10:33:12.554798+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:12.554798+0000","last_undegraded":"2024-09-16T10:33:12.554798+0000","last_fullsized":"2024-09-16T10:33:12.554798+0000","mapping_epoch":34,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1486848,"num_objects":363,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20099,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20161","reported_seq":20324,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:14.081847+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:14.081847+0000","last_peered":"2024-09-16T10:33:14.081847+0000","last_clean":"2024-09-16T10:33:14.081847+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:14.081847+0000","last_undegraded":"2024-09-16T10:33:14.081847+0000","last_fullsized":"2024-09-16T10:33:14.081847+0000","mapping_epoch":33,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1409024,"num_objects":344,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16803,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20009","reported_seq":20345,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:13.970465+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:13.970465+0000","last_peered":"2024-09-16T10:33:13.970465+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:13.970465+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1482752,"num_objects":362,"num_object_clones":0,"num_object_copies":724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16634,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20599","reported_seq":21011,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.020422+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:14.020422+0000","last_peered":"2024-09-16T10:33:14.020422+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:14.020422+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1314816,"num_objects":321,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16841,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20059","reported_seq":20241,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:14.067994+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:14.067994+0000","last_peered":"2024-09-16T10:33:14.067994+0000","last_clean":"2024-09-16T10:33:14.067994+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:14.067994+0000","last_undegraded":"2024-09-16T10:33:14.067994+0000","last_fullsized":"2024-09-16T10:33:14.067994+0000","mapping_epoch":33,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1355776,"num_objects":331,"num_object_clones":0,"num_object_copies":662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15862,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20097","reported_seq":28607,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:13.316805+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:13.316805+0000","last_peered":"2024-09-16T10:33:13.316805+0000","last_clean":"2024-09-16T10:33:13.316805+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:13.316805+0000","last_undegraded":"2024-09-16T10:33:13.316805+0000","last_fullsized":"2024-09-16T10:33:13.316805+0000","mapping_epoch":43,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1491910,"num_objects":366,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20095,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20684","reported_seq":20962,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.140507+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:14.140507+0000","last_peered":"2024-09-16T10:33:14.140507+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:14.140507+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1459142,"num_objects":358,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20682,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3979}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20041","reported_seq":20241,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:12.493039+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:12.493039+0000","last_peered":"2024-09-16T10:33:12.493039+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:12.493039+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1549254,"num_objects":380,"num_object_clones":0,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20039,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3853}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20180","reported_seq":28669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:12.532100+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:12.532100+0000","last_peered":"2024-09-16T10:33:12.532100+0000","last_clean":"2024-09-16T10:33:12.532100+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:12.532100+0000","last_undegraded":"2024-09-16T10:33:12.532100+0000","last_fullsized":"2024-09-16T10:33:12.532100+0000","mapping_epoch":47,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1622982,"num_objects":398,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20178,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19684","reported_seq":19938,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.091696+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:14.091696+0000","last_peered":"2024-09-16T10:33:14.091696+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:14.091696+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1434083,"num_objects":351,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19683,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3724}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20580","reported_seq":20859,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.128592+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:14.128592+0000","last_peered":"2024-09-16T10:33:14.128592+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:14.128592+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1369513,"num_objects":337,"num_object_clones":0,"num_object_copies":674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20577,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19810","reported_seq":23698,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.090056+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:14.090056+0000","last_peered":"2024-09-16T10:33:14.090056+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:14.090056+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1352646,"num_objects":332,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19808,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20558","reported_seq":20802,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.123922+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:14.123922+0000","last_peered":"2024-09-16T10:33:14.123922+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:14.123922+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1386380,"num_objects":342,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20555,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":23208462,"num_objects":5683,"num_object_clones":0,"num_object_copies":11366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":300622,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15103,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":257118208,"data_stored":257041417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161057,"ondisk_log_size":161057,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019009,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496680,"kb_used_data":150048,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93875160,"statfs":{"total":96636764160,"available":96128163840,"internally_reserved":0,"allocated":153649152,"data_stored":153370506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":1.976},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":2.7269999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":2.798},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.7679999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691953,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473480,"kb_used_data":18536,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93898360,"statfs":{"total":96636764160,"available":96151920640,"internally_reserved":0,"allocated":18980864,"data_stored":18644410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.70599999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.51200000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215236,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540380,"kb_used_data":85524,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93831460,"statfs":{"total":96636764160,"available":96083415040,"internally_reserved":0,"allocated":87576576,"data_stored":87305616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":2.3380000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":8.2620000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":8.5020000000000007},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.3400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86163456,"data_stored":86159843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17997824,"data_stored":17932331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":152956928,"data_stored":152949243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:15.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:16.372 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:16.374 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:16.394 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-16T10:33:14.756269+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23208481,"num_objects":5685,"num_object_clones":0,"num_object_copies":11370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":300624,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15104,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161060,"ondisk_log_size":161060,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1510540,"kb_used_data":254108,"kb_used_omap":12,"kb_used_meta":1256371,"kb_avail":281604980,"statfs":{"total":289910292480,"available":288363499520,"internally_reserved":0,"allocated":260206592,"data_stored":259320532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1286523906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,12,5,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25108480,"num_objects":-6130,"num_object_clones":0,"num_object_copies":-12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005883"},"pg_stats":[{"pgid":"2.f","version":"48'20131","reported_seq":28599,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:14.125210+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:14.125210+0000","last_peered":"2024-09-16T10:33:14.125210+0000","last_clean":"2024-09-16T10:33:14.125210+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:14.125210+0000","last_undegraded":"2024-09-16T10:33:14.125210+0000","last_fullsized":"2024-09-16T10:33:14.125210+0000","mapping_epoch":45,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1380352,"num_objects":337,"num_object_clones":0,"num_object_copies":674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20131,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20079","reported_seq":20243,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:12.494159+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:12.494159+0000","last_peered":"2024-09-16T10:33:12.494159+0000","last_clean":"2024-09-16T10:33:12.494159+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:12.494159+0000","last_undegraded":"2024-09-16T10:33:12.494159+0000","last_fullsized":"2024-09-16T10:33:12.494159+0000","mapping_epoch":34,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1568768,"num_objects":383,"num_object_clones":0,"num_object_copies":766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16292,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20286","reported_seq":20542,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:12.558276+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:12.558276+0000","last_peered":"2024-09-16T10:33:12.558276+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:12.558276+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1544216,"num_objects":378,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16343,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3925}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20099","reported_seq":20162,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:12.554798+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:12.554798+0000","last_peered":"2024-09-16T10:33:12.554798+0000","last_clean":"2024-09-16T10:33:12.554798+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:12.554798+0000","last_undegraded":"2024-09-16T10:33:12.554798+0000","last_fullsized":"2024-09-16T10:33:12.554798+0000","mapping_epoch":34,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1486848,"num_objects":363,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20099,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20161","reported_seq":20324,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:14.081847+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:14.081847+0000","last_peered":"2024-09-16T10:33:14.081847+0000","last_clean":"2024-09-16T10:33:14.081847+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:14.081847+0000","last_undegraded":"2024-09-16T10:33:14.081847+0000","last_fullsized":"2024-09-16T10:33:14.081847+0000","mapping_epoch":33,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1409024,"num_objects":344,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16803,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20009","reported_seq":20345,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:13.970465+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:13.970465+0000","last_peered":"2024-09-16T10:33:13.970465+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:13.970465+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1482752,"num_objects":362,"num_object_clones":0,"num_object_copies":724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16634,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20599","reported_seq":21011,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.020422+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:14.020422+0000","last_peered":"2024-09-16T10:33:14.020422+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:14.020422+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1314816,"num_objects":321,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16841,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20059","reported_seq":20241,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:14.067994+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:14.067994+0000","last_peered":"2024-09-16T10:33:14.067994+0000","last_clean":"2024-09-16T10:33:14.067994+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:14.067994+0000","last_undegraded":"2024-09-16T10:33:14.067994+0000","last_fullsized":"2024-09-16T10:33:14.067994+0000","mapping_epoch":33,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1355776,"num_objects":331,"num_object_clones":0,"num_object_copies":662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15862,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20097","reported_seq":28607,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:13.316805+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:13.316805+0000","last_peered":"2024-09-16T10:33:13.316805+0000","last_clean":"2024-09-16T10:33:13.316805+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:13.316805+0000","last_undegraded":"2024-09-16T10:33:13.316805+0000","last_fullsized":"2024-09-16T10:33:13.316805+0000","mapping_epoch":43,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1491910,"num_objects":366,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20095,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20684","reported_seq":20962,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.140507+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:14.140507+0000","last_peered":"2024-09-16T10:33:14.140507+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:14.140507+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1459142,"num_objects":358,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20682,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3979}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20041","reported_seq":20241,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:12.493039+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:12.493039+0000","last_peered":"2024-09-16T10:33:12.493039+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:12.493039+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1549254,"num_objects":380,"num_object_clones":0,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20039,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3853}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20180","reported_seq":28669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:12.532100+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:12.532100+0000","last_peered":"2024-09-16T10:33:12.532100+0000","last_clean":"2024-09-16T10:33:12.532100+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:12.532100+0000","last_undegraded":"2024-09-16T10:33:12.532100+0000","last_fullsized":"2024-09-16T10:33:12.532100+0000","mapping_epoch":47,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1622982,"num_objects":398,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20178,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19684","reported_seq":19938,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.091696+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:14.091696+0000","last_peered":"2024-09-16T10:33:14.091696+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:14.091696+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1434083,"num_objects":351,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19683,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3724}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20580","reported_seq":20859,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.128592+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:14.128592+0000","last_peered":"2024-09-16T10:33:14.128592+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:14.128592+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1369513,"num_objects":337,"num_object_clones":0,"num_object_copies":674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20577,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19810","reported_seq":23698,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.090056+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:14.090056+0000","last_peered":"2024-09-16T10:33:14.090056+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:14.090056+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1352646,"num_objects":332,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19808,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20558","reported_seq":20802,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.123922+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:14.123922+0000","last_peered":"2024-09-16T10:33:14.123922+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:14.123922+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1386380,"num_objects":342,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20555,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":23208462,"num_objects":5683,"num_object_clones":0,"num_object_copies":11366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":300622,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15103,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":257118208,"data_stored":257041417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161057,"ondisk_log_size":161057,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019009,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496680,"kb_used_data":150048,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93875160,"statfs":{"total":96636764160,"available":96128163840,"internally_reserved":0,"allocated":153649152,"data_stored":153370506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":1.976},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":2.7269999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":2.798},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.7679999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691953,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473480,"kb_used_data":18536,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93898360,"statfs":{"total":96636764160,"available":96151920640,"internally_reserved":0,"allocated":18980864,"data_stored":18644410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.70599999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.51200000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215236,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540380,"kb_used_data":85524,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93831460,"statfs":{"total":96636764160,"available":96083415040,"internally_reserved":0,"allocated":87576576,"data_stored":87305616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":2.3380000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":8.2620000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":8.5020000000000007},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.3400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86163456,"data_stored":86159843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17997824,"data_stored":17932331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":152956928,"data_stored":152949243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:16.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:16.395 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20286", 'reported_seq': 20542, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:12.558276+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:33:12.558276+0000', 'last_peered': '2024-09-16T10:33:12.558276+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:12.558276+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10200", 'ondisk_log_start': "44'10200", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1544216, 'num_objects': 378, 'num_object_clones': 0, 'num_object_copies': 756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 378, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16343, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3925}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:16.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:16.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20009", 'reported_seq': 20345, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:13.970465+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:13.970465+0000', 'last_peered': '2024-09-16T10:33:13.970465+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:13.970465+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1482752, 'num_objects': 362, 'num_object_clones': 0, 'num_object_copies': 724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16634, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3772}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:16.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:16.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20599", 'reported_seq': 21011, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:14.020422+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:14.020422+0000', 'last_peered': '2024-09-16T10:33:14.020422+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:14.020422+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1314816, 'num_objects': 321, 'num_object_clones': 0, 'num_object_copies': 642, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16841, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:16.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:16.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'20684", 'reported_seq': 20962, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:14.140507+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:14.140507+0000', 'last_peered': '2024-09-16T10:33:14.140507+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:14.140507+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'10600", 'ondisk_log_start': "44'10600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1459142, 'num_objects': 358, 'num_object_clones': 0, 'num_object_copies': 716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20682, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3979}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:16.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:16.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20041", 'reported_seq': 20241, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:12.493039+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:12.493039+0000', 'last_peered': '2024-09-16T10:33:12.493039+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:12.493039+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1549254, 'num_objects': 380, 'num_object_clones': 0, 'num_object_copies': 760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20039, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3853}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:16.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:16.396 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'19684", 'reported_seq': 19938, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:14.091696+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:14.091696+0000', 'last_peered': '2024-09-16T10:33:14.091696+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:14.091696+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'9600", 'ondisk_log_start': "44'9600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1434083, 'num_objects': 351, 'num_object_clones': 0, 'num_object_copies': 702, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 351, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19683, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3724}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:16.397 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:16.397 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20580", 'reported_seq': 20859, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:14.128592+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:14.128592+0000', 'last_peered': '2024-09-16T10:33:14.128592+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:14.128592+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1369513, 'num_objects': 337, 'num_object_clones': 0, 'num_object_copies': 674, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20577, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3930}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:16.397 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:33:16.397 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'19810", 'reported_seq': 23698, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:14.090056+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:33:14.090056+0000', 'last_peered': '2024-09-16T10:33:14.090056+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:33:14.090056+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'9800", 'ondisk_log_start': "44'9800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1352646, 'num_objects': 332, 'num_object_clones': 0, 'num_object_copies': 664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2269, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19808, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1902, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:16.397 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:16.397 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20558", 'reported_seq': 20802, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:14.123922+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:14.123922+0000', 'last_peered': '2024-09-16T10:33:14.123922+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:14.123922+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1386380, 'num_objects': 342, 'num_object_clones': 0, 'num_object_copies': 684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20555, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:16.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:17.079 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:17.079 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36243/11370 objects degraded (318.760%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":685,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":5685,"data_bytes":23208481,"bytes_used":1546792960,"bytes_avail":288363499520,"bytes_total":289910292480,"degraded_objects":36243,"degraded_total":11370,"degraded_ratio":3.1875989445910289,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":169,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":384},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:17.114 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:17.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:17.681 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:17.682 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:17.702 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-16T10:33:16.756747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23208481,"num_objects":5685,"num_object_clones":0,"num_object_copies":11370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":300624,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15104,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161060,"ondisk_log_size":161060,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1510540,"kb_used_data":254108,"kb_used_omap":12,"kb_used_meta":1256371,"kb_avail":281604980,"statfs":{"total":289910292480,"available":288363499520,"internally_reserved":0,"allocated":260206592,"data_stored":259320532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1286523906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,12,5,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18903040,"num_objects":-4615,"num_object_clones":0,"num_object_copies":-9230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4615,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002812"},"pg_stats":[{"pgid":"2.f","version":"48'20131","reported_seq":28599,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:14.125210+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:14.125210+0000","last_peered":"2024-09-16T10:33:14.125210+0000","last_clean":"2024-09-16T10:33:14.125210+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:14.125210+0000","last_undegraded":"2024-09-16T10:33:14.125210+0000","last_fullsized":"2024-09-16T10:33:14.125210+0000","mapping_epoch":45,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1380352,"num_objects":337,"num_object_clones":0,"num_object_copies":674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20131,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20079","reported_seq":20243,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:12.494159+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:12.494159+0000","last_peered":"2024-09-16T10:33:12.494159+0000","last_clean":"2024-09-16T10:33:12.494159+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:12.494159+0000","last_undegraded":"2024-09-16T10:33:12.494159+0000","last_fullsized":"2024-09-16T10:33:12.494159+0000","mapping_epoch":34,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1568768,"num_objects":383,"num_object_clones":0,"num_object_copies":766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16292,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20286","reported_seq":20542,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:12.558276+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:12.558276+0000","last_peered":"2024-09-16T10:33:12.558276+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:12.558276+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1544216,"num_objects":378,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":378,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16343,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3925}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20099","reported_seq":20162,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:12.554798+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:12.554798+0000","last_peered":"2024-09-16T10:33:12.554798+0000","last_clean":"2024-09-16T10:33:12.554798+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:12.554798+0000","last_undegraded":"2024-09-16T10:33:12.554798+0000","last_fullsized":"2024-09-16T10:33:12.554798+0000","mapping_epoch":34,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1486848,"num_objects":363,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20099,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20161","reported_seq":20324,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:14.081847+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:14.081847+0000","last_peered":"2024-09-16T10:33:14.081847+0000","last_clean":"2024-09-16T10:33:14.081847+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:14.081847+0000","last_undegraded":"2024-09-16T10:33:14.081847+0000","last_fullsized":"2024-09-16T10:33:14.081847+0000","mapping_epoch":33,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1409024,"num_objects":344,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16803,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20009","reported_seq":20345,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:13.970465+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:13.970465+0000","last_peered":"2024-09-16T10:33:13.970465+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:13.970465+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1482752,"num_objects":362,"num_object_clones":0,"num_object_copies":724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16634,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3772}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20599","reported_seq":21011,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.020422+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:14.020422+0000","last_peered":"2024-09-16T10:33:14.020422+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:14.020422+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1314816,"num_objects":321,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16841,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20059","reported_seq":20241,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:14.067994+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:14.067994+0000","last_peered":"2024-09-16T10:33:14.067994+0000","last_clean":"2024-09-16T10:33:14.067994+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:14.067994+0000","last_undegraded":"2024-09-16T10:33:14.067994+0000","last_fullsized":"2024-09-16T10:33:14.067994+0000","mapping_epoch":33,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1355776,"num_objects":331,"num_object_clones":0,"num_object_copies":662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15862,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20097","reported_seq":28607,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:13.316805+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:13.316805+0000","last_peered":"2024-09-16T10:33:13.316805+0000","last_clean":"2024-09-16T10:33:13.316805+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:13.316805+0000","last_undegraded":"2024-09-16T10:33:13.316805+0000","last_fullsized":"2024-09-16T10:33:13.316805+0000","mapping_epoch":43,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1491910,"num_objects":366,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20095,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296689+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:28.296689+0000","last_peered":"2024-09-16T10:31:28.296689+0000","last_clean":"2024-09-16T10:31:28.296689+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:28.296689+0000","last_undegraded":"2024-09-16T10:31:28.296689+0000","last_fullsized":"2024-09-16T10:31:28.296689+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20684","reported_seq":20962,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.140507+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:14.140507+0000","last_peered":"2024-09-16T10:33:14.140507+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:14.140507+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1459142,"num_objects":358,"num_object_clones":0,"num_object_copies":716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20682,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3979}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20041","reported_seq":20241,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:12.493039+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:12.493039+0000","last_peered":"2024-09-16T10:33:12.493039+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:12.493039+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1549254,"num_objects":380,"num_object_clones":0,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20039,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3853}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20180","reported_seq":28669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:12.532100+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:12.532100+0000","last_peered":"2024-09-16T10:33:12.532100+0000","last_clean":"2024-09-16T10:33:12.532100+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:12.532100+0000","last_undegraded":"2024-09-16T10:33:12.532100+0000","last_fullsized":"2024-09-16T10:33:12.532100+0000","mapping_epoch":47,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1622982,"num_objects":398,"num_object_clones":0,"num_object_copies":796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20178,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19684","reported_seq":19938,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.091696+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:14.091696+0000","last_peered":"2024-09-16T10:33:14.091696+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:14.091696+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9600","ondisk_log_start":"44'9600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1434083,"num_objects":351,"num_object_clones":0,"num_object_copies":702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19683,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3724}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20580","reported_seq":20859,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.128592+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:14.128592+0000","last_peered":"2024-09-16T10:33:14.128592+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:14.128592+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1369513,"num_objects":337,"num_object_clones":0,"num_object_copies":674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20577,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3930}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19810","reported_seq":23698,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.090056+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:14.090056+0000","last_peered":"2024-09-16T10:33:14.090056+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:14.090056+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1352646,"num_objects":332,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19808,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20558","reported_seq":20802,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:14.123922+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:14.123922+0000","last_peered":"2024-09-16T10:33:14.123922+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:14.123922+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1386380,"num_objects":342,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20555,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":23208462,"num_objects":5683,"num_object_clones":0,"num_object_copies":11366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5683,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":300622,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15103,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":257118208,"data_stored":257041417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161057,"ondisk_log_size":161057,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019009,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496680,"kb_used_data":150048,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93875160,"statfs":{"total":96636764160,"available":96128163840,"internally_reserved":0,"allocated":153649152,"data_stored":153370506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":1.976},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":2.7269999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":2.798},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":2.7679999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691953,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":473480,"kb_used_data":18536,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93898360,"statfs":{"total":96636764160,"available":96151920640,"internally_reserved":0,"allocated":18980864,"data_stored":18644410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,4,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":0.70599999999999996}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.51200000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215236,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":540380,"kb_used_data":85524,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93831460,"statfs":{"total":96636764160,"available":96083415040,"internally_reserved":0,"allocated":87576576,"data_stored":87305616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,0,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":3.161,"5min":2.8450000000000002,"15min":2.7789999999999999},"min":{"1min":0.185,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":16.806000000000001,"5min":16.806000000000001,"15min":16.806000000000001},"last":2.3380000000000001},{"interface":"front","average":{"1min":2.8490000000000002,"5min":2.7869999999999999,"15min":2.2749999999999999},"min":{"1min":0.22600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":16.760999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":8.2620000000000005}]},{"osd":2,"last update":"Mon Sep 16 10:32:13 2024","interfaces":[{"interface":"back","average":{"1min":2.0209999999999999,"5min":2.528,"15min":1.6899999999999999},"min":{"1min":0.24299999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":8.9830000000000005,"5min":10.382999999999999,"15min":18.093},"last":8.5020000000000007},{"interface":"front","average":{"1min":2.7799999999999998,"5min":2.3650000000000002,"15min":1.8640000000000001},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":11.863,"5min":13.752000000000001,"15min":13.752000000000001},"last":1.3400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86163456,"data_stored":86159843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":17997824,"data_stored":17932331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":152956928,"data_stored":152949243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:20.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:21.289 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:21.292 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:21.313 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-16T10:33:20.757683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13713953,"num_objects":3367,"num_object_clones":0,"num_object_copies":6734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":302942,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15207,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160778,"ondisk_log_size":160778,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1463460,"kb_used_data":235712,"kb_used_omap":12,"kb_used_meta":1227699,"kb_avail":281652060,"statfs":{"total":289910292480,"available":288411709440,"internally_reserved":0,"allocated":241369088,"data_stored":240483028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1257163778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,9,9,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24805376,"num_objects":-6056,"num_object_clones":0,"num_object_copies":-12112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6056,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002847"},"pg_stats":[{"pgid":"2.f","version":"48'20289","reported_seq":28756,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:19.165140+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:19.165140+0000","last_peered":"2024-09-16T10:33:19.165140+0000","last_clean":"2024-09-16T10:33:19.165140+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:19.165140+0000","last_undegraded":"2024-09-16T10:33:19.165140+0000","last_fullsized":"2024-09-16T10:33:19.165140+0000","mapping_epoch":45,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":733184,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20289,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20221","reported_seq":20385,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:17.573404+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:17.573404+0000","last_peered":"2024-09-16T10:33:17.573404+0000","last_clean":"2024-09-16T10:33:17.573404+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:17.573404+0000","last_undegraded":"2024-09-16T10:33:17.573404+0000","last_fullsized":"2024-09-16T10:33:17.573404+0000","mapping_epoch":34,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":987136,"num_objects":241,"num_object_clones":0,"num_object_copies":482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16434,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20434","reported_seq":20691,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:17.548144+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:17.548144+0000","last_peered":"2024-09-16T10:33:17.548144+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:17.548144+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":938008,"num_objects":230,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16491,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4073}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20240","reported_seq":20303,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:17.571902+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:17.571902+0000","last_peered":"2024-09-16T10:33:17.571902+0000","last_clean":"2024-09-16T10:33:17.571902+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:17.571902+0000","last_undegraded":"2024-09-16T10:33:17.571902+0000","last_fullsized":"2024-09-16T10:33:17.571902+0000","mapping_epoch":34,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":909312,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20240,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20312","reported_seq":20475,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:19.121691+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:19.121691+0000","last_peered":"2024-09-16T10:33:19.121691+0000","last_clean":"2024-09-16T10:33:19.121691+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:19.121691+0000","last_undegraded":"2024-09-16T10:33:19.121691+0000","last_fullsized":"2024-09-16T10:33:19.121691+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":790528,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16954,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20178","reported_seq":20514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.120589+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:19.120589+0000","last_peered":"2024-09-16T10:33:19.120589+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:19.120589+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":790528,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16803,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3941}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20726","reported_seq":21137,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.163198+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:19.163198+0000","last_peered":"2024-09-16T10:33:19.163198+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:19.163198+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10700","ondisk_log_start":"44'10700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":794624,"num_objects":194,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16968,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20196","reported_seq":20378,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:19.122256+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:19.122256+0000","last_peered":"2024-09-16T10:33:19.122256+0000","last_clean":"2024-09-16T10:33:19.122256+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:19.122256+0000","last_undegraded":"2024-09-16T10:33:19.122256+0000","last_fullsized":"2024-09-16T10:33:19.122256+0000","mapping_epoch":33,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":794624,"num_objects":194,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15999,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20229","reported_seq":28739,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:18.333857+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:18.333857+0000","last_peered":"2024-09-16T10:33:18.333857+0000","last_clean":"2024-09-16T10:33:18.333857+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:18.333857+0000","last_undegraded":"2024-09-16T10:33:18.333857+0000","last_fullsized":"2024-09-16T10:33:18.333857+0000","mapping_epoch":43,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":951238,"num_objects":234,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20227,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20836","reported_seq":21116,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.021257+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:19.021257+0000","last_peered":"2024-09-16T10:33:19.021257+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:19.021257+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":836550,"num_objects":206,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20834,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4131}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20192","reported_seq":20390,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:17.589905+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:17.589905+0000","last_peered":"2024-09-16T10:33:17.589905+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:17.589905+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930758,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20190,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4004}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20333","reported_seq":28822,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:17.595371+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:17.595371+0000","last_peered":"2024-09-16T10:33:17.595371+0000","last_clean":"2024-09-16T10:33:17.595371+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:17.595371+0000","last_undegraded":"2024-09-16T10:33:17.595371+0000","last_fullsized":"2024-09-16T10:33:17.595371+0000","mapping_epoch":47,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":996294,"num_objects":245,"num_object_clones":0,"num_object_copies":490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20331,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19819","reported_seq":20073,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.118617+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:19.118617+0000","last_peered":"2024-09-16T10:33:19.118617+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:19.118617+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":881123,"num_objects":216,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19818,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20724","reported_seq":21004,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.123334+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:19.123334+0000","last_peered":"2024-09-16T10:33:19.123334+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:19.123334+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10700","ondisk_log_start":"44'10700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":779689,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20721,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19951","reported_seq":24046,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.172441+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:19.172441+0000","last_peered":"2024-09-16T10:33:19.172441+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:19.172441+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":775110,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19949,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20695","reported_seq":20941,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.150681+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:19.150681+0000","last_peered":"2024-09-16T10:33:19.150681+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:19.150681+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":825228,"num_objects":205,"num_object_clones":0,"num_object_copies":410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20692,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4117}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":13713934,"num_objects":3365,"num_object_clones":0,"num_object_copies":6730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":302940,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15206,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":238198784,"data_stored":238121993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019010,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479836,"kb_used_data":145492,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93892004,"statfs":{"total":96636764160,"available":96145412096,"internally_reserved":0,"allocated":148983808,"data_stored":148705162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":342355632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.83699999999999997},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.246}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.89400000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691954,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458352,"kb_used_data":11624,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93913488,"statfs":{"total":96636764160,"available":96167411712,"internally_reserved":0,"allocated":11902976,"data_stored":11566522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":457436842},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,2,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":3.2360000000000002},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":3.2719999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.3239999999999998},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215237,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525272,"kb_used_data":78596,"kb_used_omap":4,"kb_used_meta":446651,"kb_avail":93846568,"statfs":{"total":96636764160,"available":96098885632,"internally_reserved":0,"allocated":80482304,"data_stored":80211344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":457371304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":8.9629999999999992},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":1.7629999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":1.7010000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":8.9149999999999991}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79151104,"data_stored":79147491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10829824,"data_stored":10764331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":148217856,"data_stored":148210171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:21.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:21.921 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:21.921 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:21.945 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-16T10:33:20.757683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13713953,"num_objects":3367,"num_object_clones":0,"num_object_copies":6734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":302942,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15207,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160778,"ondisk_log_size":160778,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1463460,"kb_used_data":235712,"kb_used_omap":12,"kb_used_meta":1227699,"kb_avail":281652060,"statfs":{"total":289910292480,"available":288411709440,"internally_reserved":0,"allocated":241369088,"data_stored":240483028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1257163778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,9,9,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24805376,"num_objects":-6056,"num_object_clones":0,"num_object_copies":-12112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6056,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002847"},"pg_stats":[{"pgid":"2.f","version":"48'20289","reported_seq":28756,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:19.165140+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:19.165140+0000","last_peered":"2024-09-16T10:33:19.165140+0000","last_clean":"2024-09-16T10:33:19.165140+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:19.165140+0000","last_undegraded":"2024-09-16T10:33:19.165140+0000","last_fullsized":"2024-09-16T10:33:19.165140+0000","mapping_epoch":45,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":733184,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20289,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20221","reported_seq":20385,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:17.573404+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:17.573404+0000","last_peered":"2024-09-16T10:33:17.573404+0000","last_clean":"2024-09-16T10:33:17.573404+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:17.573404+0000","last_undegraded":"2024-09-16T10:33:17.573404+0000","last_fullsized":"2024-09-16T10:33:17.573404+0000","mapping_epoch":34,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":987136,"num_objects":241,"num_object_clones":0,"num_object_copies":482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16434,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20434","reported_seq":20691,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:17.548144+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:17.548144+0000","last_peered":"2024-09-16T10:33:17.548144+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:17.548144+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":938008,"num_objects":230,"num_object_clones":0,"num_object_copies":460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16491,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4073}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20240","reported_seq":20303,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:17.571902+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:17.571902+0000","last_peered":"2024-09-16T10:33:17.571902+0000","last_clean":"2024-09-16T10:33:17.571902+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:17.571902+0000","last_undegraded":"2024-09-16T10:33:17.571902+0000","last_fullsized":"2024-09-16T10:33:17.571902+0000","mapping_epoch":34,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":909312,"num_objects":222,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20240,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20312","reported_seq":20475,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:19.121691+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:19.121691+0000","last_peered":"2024-09-16T10:33:19.121691+0000","last_clean":"2024-09-16T10:33:19.121691+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:19.121691+0000","last_undegraded":"2024-09-16T10:33:19.121691+0000","last_fullsized":"2024-09-16T10:33:19.121691+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":790528,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16954,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20178","reported_seq":20514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.120589+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:19.120589+0000","last_peered":"2024-09-16T10:33:19.120589+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:19.120589+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":790528,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16803,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3941}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20726","reported_seq":21137,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.163198+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:19.163198+0000","last_peered":"2024-09-16T10:33:19.163198+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:19.163198+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10700","ondisk_log_start":"44'10700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":794624,"num_objects":194,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16968,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20196","reported_seq":20378,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:19.122256+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:19.122256+0000","last_peered":"2024-09-16T10:33:19.122256+0000","last_clean":"2024-09-16T10:33:19.122256+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:19.122256+0000","last_undegraded":"2024-09-16T10:33:19.122256+0000","last_fullsized":"2024-09-16T10:33:19.122256+0000","mapping_epoch":33,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":794624,"num_objects":194,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15999,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20229","reported_seq":28739,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:18.333857+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:18.333857+0000","last_peered":"2024-09-16T10:33:18.333857+0000","last_clean":"2024-09-16T10:33:18.333857+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:18.333857+0000","last_undegraded":"2024-09-16T10:33:18.333857+0000","last_fullsized":"2024-09-16T10:33:18.333857+0000","mapping_epoch":43,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":951238,"num_objects":234,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20227,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20836","reported_seq":21116,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.021257+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:19.021257+0000","last_peered":"2024-09-16T10:33:19.021257+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:19.021257+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":836550,"num_objects":206,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20834,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4131}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20192","reported_seq":20390,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:17.589905+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:17.589905+0000","last_peered":"2024-09-16T10:33:17.589905+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:17.589905+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930758,"num_objects":229,"num_object_clones":0,"num_object_copies":458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20190,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4004}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20333","reported_seq":28822,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:17.595371+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:17.595371+0000","last_peered":"2024-09-16T10:33:17.595371+0000","last_clean":"2024-09-16T10:33:17.595371+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:17.595371+0000","last_undegraded":"2024-09-16T10:33:17.595371+0000","last_fullsized":"2024-09-16T10:33:17.595371+0000","mapping_epoch":47,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":996294,"num_objects":245,"num_object_clones":0,"num_object_copies":490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20331,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19819","reported_seq":20073,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.118617+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:19.118617+0000","last_peered":"2024-09-16T10:33:19.118617+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:19.118617+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":881123,"num_objects":216,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19818,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20724","reported_seq":21004,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.123334+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:19.123334+0000","last_peered":"2024-09-16T10:33:19.123334+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:19.123334+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10700","ondisk_log_start":"44'10700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":779689,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20721,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":63,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-16T10:31:28.296526+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:28.296526+0000","last_peered":"2024-09-16T10:31:28.296526+0000","last_clean":"2024-09-16T10:31:28.296526+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:28.296526+0000","last_undegraded":"2024-09-16T10:31:28.296526+0000","last_fullsized":"2024-09-16T10:31:28.296526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19951","reported_seq":24046,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.172441+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:19.172441+0000","last_peered":"2024-09-16T10:33:19.172441+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:19.172441+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":775110,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19949,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20695","reported_seq":20941,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.150681+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:19.150681+0000","last_peered":"2024-09-16T10:33:19.150681+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:19.150681+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":825228,"num_objects":205,"num_object_clones":0,"num_object_copies":410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20692,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4117}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":13713934,"num_objects":3365,"num_object_clones":0,"num_object_copies":6730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":302940,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15206,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":238198784,"data_stored":238121993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160775,"ondisk_log_size":160775,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019010,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479836,"kb_used_data":145492,"kb_used_omap":4,"kb_used_meta":334331,"kb_avail":93892004,"statfs":{"total":96636764160,"available":96145412096,"internally_reserved":0,"allocated":148983808,"data_stored":148705162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":342355632},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.83699999999999997},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.246}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.89400000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691954,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458352,"kb_used_data":11624,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93913488,"statfs":{"total":96636764160,"available":96167411712,"internally_reserved":0,"allocated":11902976,"data_stored":11566522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":457436842},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,2,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":3.2360000000000002},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":3.2719999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.3239999999999998},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215237,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525272,"kb_used_data":78596,"kb_used_omap":4,"kb_used_meta":446651,"kb_avail":93846568,"statfs":{"total":96636764160,"available":96098885632,"internally_reserved":0,"allocated":80482304,"data_stored":80211344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":457371304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":8.9629999999999992},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":1.7629999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":1.7010000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":8.9149999999999991}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79151104,"data_stored":79147491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10829824,"data_stored":10764331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":148217856,"data_stored":148210171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:21.946 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:21.946 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20434", 'reported_seq': 20691, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:17.548144+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:33:17.548144+0000', 'last_peered': '2024-09-16T10:33:17.548144+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:17.548144+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 938008, 'num_objects': 230, 'num_object_clones': 0, 'num_object_copies': 460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 230, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16491, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4073}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20178", 'reported_seq': 20514, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:19.120589+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:19.120589+0000', 'last_peered': '2024-09-16T10:33:19.120589+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:19.120589+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10100", 'ondisk_log_start': "44'10100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 790528, 'num_objects': 193, 'num_object_clones': 0, 'num_object_copies': 386, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16803, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3941}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20726", 'reported_seq': 21137, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:19.163198+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:19.163198+0000', 'last_peered': '2024-09-16T10:33:19.163198+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:19.163198+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10700", 'ondisk_log_start': "44'10700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 794624, 'num_objects': 194, 'num_object_clones': 0, 'num_object_copies': 388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16968, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4093}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'20836", 'reported_seq': 21116, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:19.021257+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:19.021257+0000', 'last_peered': '2024-09-16T10:33:19.021257+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:19.021257+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 836550, 'num_objects': 206, 'num_object_clones': 0, 'num_object_copies': 412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20834, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4131}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20192", 'reported_seq': 20390, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:17.589905+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:17.589905+0000', 'last_peered': '2024-09-16T10:33:17.589905+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:17.589905+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10100", 'ondisk_log_start': "44'10100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 930758, 'num_objects': 229, 'num_object_clones': 0, 'num_object_copies': 458, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20190, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4004}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'19819", 'reported_seq': 20073, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:19.118617+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:19.118617+0000', 'last_peered': '2024-09-16T10:33:19.118617+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:19.118617+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'9800", 'ondisk_log_start': "44'9800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 881123, 'num_objects': 216, 'num_object_clones': 0, 'num_object_copies': 432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19818, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3859}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:21.947 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:21.948 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20724", 'reported_seq': 21004, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:19.123334+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:19.123334+0000', 'last_peered': '2024-09-16T10:33:19.123334+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:19.123334+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10700", 'ondisk_log_start': "44'10700", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 779689, 'num_objects': 193, 'num_object_clones': 0, 'num_object_copies': 386, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20721, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:21.948 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:33:21.948 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'19951", 'reported_seq': 24046, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:19.172441+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:33:19.172441+0000', 'last_peered': '2024-09-16T10:33:19.172441+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:33:19.172441+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'9900", 'ondisk_log_start': "44'9900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 775110, 'num_objects': 191, 'num_object_clones': 0, 'num_object_copies': 382, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2166, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19949, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2005, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:21.948 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:21.948 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20695", 'reported_seq': 20941, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:19.150681+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:19.150681+0000', 'last_peered': '2024-09-16T10:33:19.150681+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:19.150681+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10600", 'ondisk_log_start': "44'10600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 825228, 'num_objects': 205, 'num_object_clones': 0, 'num_object_copies': 410, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20692, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4117}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:21.948 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:22.622 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:22.622 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36140/6734 objects degraded (536.680%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":691,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":3367,"data_bytes":13713953,"bytes_used":1498583040,"bytes_avail":288411709440,"bytes_total":289910292480,"degraded_objects":36140,"degraded_total":6734,"degraded_ratio":5.3667953667953672,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":272,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":504},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:22.654 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:22.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:23.236 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:23.237 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:23.266 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-16T10:33:22.758173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":10777121,"num_objects":2650,"num_object_clones":0,"num_object_copies":5300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":303659,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15207,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160895,"ondisk_log_size":160895,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1462904,"kb_used_data":231060,"kb_used_omap":12,"kb_used_meta":1231795,"kb_avail":281652616,"statfs":{"total":289910292480,"available":288412278784,"internally_reserved":0,"allocated":236605440,"data_stored":235719380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1261358082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,7,7,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21786624,"num_objects":-5319,"num_object_clones":0,"num_object_copies":-10638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002846"},"pg_stats":[{"pgid":"2.f","version":"48'20289","reported_seq":28756,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:19.165140+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:19.165140+0000","last_peered":"2024-09-16T10:33:19.165140+0000","last_clean":"2024-09-16T10:33:19.165140+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:19.165140+0000","last_undegraded":"2024-09-16T10:33:19.165140+0000","last_fullsized":"2024-09-16T10:33:19.165140+0000","mapping_epoch":45,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":733184,"num_objects":179,"num_object_clones":0,"num_object_copies":358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20289,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20369","reported_seq":20533,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.585914+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:22.585914+0000","last_peered":"2024-09-16T10:33:22.585914+0000","last_clean":"2024-09-16T10:33:22.585914+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:22.585914+0000","last_undegraded":"2024-09-16T10:33:22.585914+0000","last_fullsized":"2024-09-16T10:33:22.585914+0000","mapping_epoch":34,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380928,"num_objects":93,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16582,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20582","reported_seq":20839,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:22.603507+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:22.603507+0000","last_peered":"2024-09-16T10:33:22.603507+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:22.603507+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331800,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16639,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4221}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20376","reported_seq":20439,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.661496+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:22.661496+0000","last_peered":"2024-09-16T10:33:22.661496+0000","last_clean":"2024-09-16T10:33:22.661496+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:22.661496+0000","last_undegraded":"2024-09-16T10:33:22.661496+0000","last_fullsized":"2024-09-16T10:33:22.661496+0000","mapping_epoch":34,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352256,"num_objects":86,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20376,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20312","reported_seq":20475,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:19.121691+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:19.121691+0000","last_peered":"2024-09-16T10:33:19.121691+0000","last_clean":"2024-09-16T10:33:19.121691+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:19.121691+0000","last_undegraded":"2024-09-16T10:33:19.121691+0000","last_fullsized":"2024-09-16T10:33:19.121691+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":790528,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16954,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20178","reported_seq":20514,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.120589+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:19.120589+0000","last_peered":"2024-09-16T10:33:19.120589+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:19.120589+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":790528,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16803,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3941}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20726","reported_seq":21137,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.163198+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:19.163198+0000","last_peered":"2024-09-16T10:33:19.163198+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:19.163198+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10700","ondisk_log_start":"44'10700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":794624,"num_objects":194,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16968,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20196","reported_seq":20378,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:19.122256+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:19.122256+0000","last_peered":"2024-09-16T10:33:19.122256+0000","last_clean":"2024-09-16T10:33:19.122256+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:19.122256+0000","last_undegraded":"2024-09-16T10:33:19.122256+0000","last_fullsized":"2024-09-16T10:33:19.122256+0000","mapping_epoch":33,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":794624,"num_objects":194,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15999,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20229","reported_seq":28739,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:18.333857+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:18.333857+0000","last_peered":"2024-09-16T10:33:18.333857+0000","last_clean":"2024-09-16T10:33:18.333857+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:18.333857+0000","last_undegraded":"2024-09-16T10:33:18.333857+0000","last_fullsized":"2024-09-16T10:33:18.333857+0000","mapping_epoch":43,"log_start":"44'10200","ondisk_log_start":"44'10200","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":951238,"num_objects":234,"num_object_clones":0,"num_object_copies":468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20227,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20836","reported_seq":21116,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.021257+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:19.021257+0000","last_peered":"2024-09-16T10:33:19.021257+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:19.021257+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":836550,"num_objects":206,"num_object_clones":0,"num_object_copies":412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20834,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4131}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20334","reported_seq":20534,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:22.632314+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:22.632314+0000","last_peered":"2024-09-16T10:33:22.632314+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:22.632314+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349126,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20332,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20476","reported_seq":28965,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.653281+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:22.653281+0000","last_peered":"2024-09-16T10:33:22.653281+0000","last_clean":"2024-09-16T10:33:22.653281+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:22.653281+0000","last_undegraded":"2024-09-16T10:33:22.653281+0000","last_fullsized":"2024-09-16T10:33:22.653281+0000","mapping_epoch":47,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410566,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20474,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19819","reported_seq":20073,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.118617+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:19.118617+0000","last_peered":"2024-09-16T10:33:19.118617+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:19.118617+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9800","ondisk_log_start":"44'9800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":881123,"num_objects":216,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19818,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3859}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20724","reported_seq":21004,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.123334+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:19.123334+0000","last_peered":"2024-09-16T10:33:19.123334+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:19.123334+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10700","ondisk_log_start":"44'10700","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":779689,"num_objects":193,"num_object_clones":0,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20721,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302601+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:29.302601+0000","last_peered":"2024-09-16T10:31:29.302601+0000","last_clean":"2024-09-16T10:31:29.302601+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:29.302601+0000","last_undegraded":"2024-09-16T10:31:29.302601+0000","last_fullsized":"2024-09-16T10:31:29.302601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'19951","reported_seq":24046,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.172441+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:19.172441+0000","last_peered":"2024-09-16T10:33:19.172441+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:19.172441+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":775110,"num_objects":191,"num_object_clones":0,"num_object_copies":382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19949,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2005,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20695","reported_seq":20941,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:19.150681+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:19.150681+0000","last_peered":"2024-09-16T10:33:19.150681+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:19.150681+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":825228,"num_objects":205,"num_object_clones":0,"num_object_copies":410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20692,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4117}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":10777102,"num_objects":2648,"num_object_clones":0,"num_object_copies":5296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2648,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":303657,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15206,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":233451520,"data_stored":233374729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160892,"ondisk_log_size":160892,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019011,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479280,"kb_used_data":140840,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93892560,"statfs":{"total":96636764160,"available":96145981440,"internally_reserved":0,"allocated":144220160,"data_stored":143941514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":5.2229999999999999},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":4.2599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691954,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458352,"kb_used_data":11624,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93913488,"statfs":{"total":96636764160,"available":96167411712,"internally_reserved":0,"allocated":11902976,"data_stored":11566522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":457436842},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2,2,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":3.0270000000000001,"5min":2.7959999999999998,"15min":3.444},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":10.02,"5min":12.236000000000001,"15min":17.675000000000001},"last":3.2360000000000002},{"interface":"front","average":{"1min":3.3290000000000002,"5min":2.798,"15min":2.7400000000000002},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.157,"5min":12.195,"15min":16.850999999999999},"last":3.2719999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:32:19 2024","interfaces":[{"interface":"back","average":{"1min":2.7040000000000002,"5min":2.6659999999999999,"15min":2.4849999999999999},"min":{"1min":0.22800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":9.2720000000000002,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.3239999999999998},{"interface":"front","average":{"1min":2.8319999999999999,"5min":2.7890000000000001,"15min":2.851},"min":{"1min":0.25700000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":8.1489999999999991,"5min":11.074,"15min":11.412000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215237,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525272,"kb_used_data":78596,"kb_used_omap":4,"kb_used_meta":446651,"kb_avail":93846568,"statfs":{"total":96636764160,"available":96098885632,"internally_reserved":0,"allocated":80482304,"data_stored":80211344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":457371304},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":8.9629999999999992},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":1.7629999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":1.7010000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":8.9149999999999991}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79151104,"data_stored":79147491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10829824,"data_stored":10764331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143470592,"data_stored":143462907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:25.800 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Removed 164319 objects 2024-09-16T10:33:25.800 INFO:tasks.rep_lost_unfound_delete.radosbench.client.admin.smithi050.stdout:Clean up completed and total clean up time :373.222 2024-09-16T10:33:26.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:26.572 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:26.573 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:26.584 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-16T10:33:24.758648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4117025,"num_objects":1024,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":305285,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15304,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1457272,"kb_used_data":217240,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281658248,"statfs":{"total":289910292480,"available":288418045952,"internally_reserved":0,"allocated":222453760,"data_stored":221567700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,8,11,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25473024,"num_objects":-6219,"num_object_clones":0,"num_object_copies":-12438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-280,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6219,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":280,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002837"},"pg_stats":[{"pgid":"2.f","version":"48'20422","reported_seq":28890,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:24.203384+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:24.203384+0000","last_peered":"2024-09-16T10:33:24.203384+0000","last_clean":"2024-09-16T10:33:24.203384+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:24.203384+0000","last_undegraded":"2024-09-16T10:33:24.203384+0000","last_fullsized":"2024-09-16T10:33:24.203384+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20422,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20369","reported_seq":20533,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.585914+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:22.585914+0000","last_peered":"2024-09-16T10:33:22.585914+0000","last_clean":"2024-09-16T10:33:22.585914+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:22.585914+0000","last_undegraded":"2024-09-16T10:33:22.585914+0000","last_fullsized":"2024-09-16T10:33:22.585914+0000","mapping_epoch":34,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380928,"num_objects":93,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16582,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20582","reported_seq":20839,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:22.603507+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:22.603507+0000","last_peered":"2024-09-16T10:33:22.603507+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:22.603507+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331800,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16639,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4221}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20376","reported_seq":20439,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.661496+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:22.661496+0000","last_peered":"2024-09-16T10:33:22.661496+0000","last_clean":"2024-09-16T10:33:22.661496+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:22.661496+0000","last_undegraded":"2024-09-16T10:33:22.661496+0000","last_fullsized":"2024-09-16T10:33:22.661496+0000","mapping_epoch":34,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352256,"num_objects":86,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20376,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20455","reported_seq":20616,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:24.240434+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:24.240434+0000","last_peered":"2024-09-16T10:33:24.240434+0000","last_clean":"2024-09-16T10:33:24.240434+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:24.240434+0000","last_undegraded":"2024-09-16T10:33:24.240434+0000","last_fullsized":"2024-09-16T10:33:24.240434+0000","mapping_epoch":33,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17097,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20324","reported_seq":20660,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.122915+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:24.122915+0000","last_peered":"2024-09-16T10:33:24.122915+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:24.122915+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192512,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16949,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20878","reported_seq":21289,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.239233+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:24.239233+0000","last_peered":"2024-09-16T10:33:24.239233+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:24.239233+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172032,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17120,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20347","reported_seq":20529,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:24.208372+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:24.208372+0000","last_peered":"2024-09-16T10:33:24.208372+0000","last_clean":"2024-09-16T10:33:24.208372+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:24.208372+0000","last_undegraded":"2024-09-16T10:33:24.208372+0000","last_fullsized":"2024-09-16T10:33:24.208372+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176128,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16150,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20382","reported_seq":28892,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.368925+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:23.368925+0000","last_peered":"2024-09-16T10:33:23.368925+0000","last_clean":"2024-09-16T10:33:23.368925+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:23.368925+0000","last_undegraded":"2024-09-16T10:33:23.368925+0000","last_fullsized":"2024-09-16T10:33:23.368925+0000","mapping_epoch":43,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324550,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20380,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20985","reported_seq":21265,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.199120+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:24.199120+0000","last_peered":"2024-09-16T10:33:24.199120+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:24.199120+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226246,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20983,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4280}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20334","reported_seq":20534,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:22.632314+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:22.632314+0000","last_peered":"2024-09-16T10:33:22.632314+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:22.632314+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349126,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20332,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20476","reported_seq":28965,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.653281+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:22.653281+0000","last_peered":"2024-09-16T10:33:22.653281+0000","last_clean":"2024-09-16T10:33:22.653281+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:22.653281+0000","last_undegraded":"2024-09-16T10:33:22.653281+0000","last_fullsized":"2024-09-16T10:33:22.653281+0000","mapping_epoch":47,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410566,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20474,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19977","reported_seq":20230,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.170935+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:24.170935+0000","last_peered":"2024-09-16T10:33:24.170935+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:24.170935+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233955,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19976,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20867","reported_seq":21147,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.104255+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:24.104255+0000","last_peered":"2024-09-16T10:33:24.104255+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:24.104255+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193961,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20864,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302601+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:29.302601+0000","last_peered":"2024-09-16T10:31:29.302601+0000","last_clean":"2024-09-16T10:31:29.302601+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:29.302601+0000","last_undegraded":"2024-09-16T10:31:29.302601+0000","last_fullsized":"2024-09-16T10:31:29.302601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20102","reported_seq":24391,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.238657+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:24.238657+0000","last_peered":"2024-09-16T10:33:24.238657+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:24.238657+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156614,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20100,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20842","reported_seq":21088,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.201690+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:24.201690+0000","last_peered":"2024-09-16T10:33:24.201690+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:24.201690+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223116,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20839,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4117006,"num_objects":1022,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":305283,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15303,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":219074560,"data_stored":218997769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019011,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479280,"kb_used_data":140840,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93892560,"statfs":{"total":96636764160,"available":96145981440,"internally_reserved":0,"allocated":144220160,"data_stored":143941514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":5.2229999999999999},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":4.2599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691955,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455544,"kb_used_data":4720,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93916296,"statfs":{"total":96636764160,"available":96170287104,"internally_reserved":0,"allocated":4833280,"data_stored":4496826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":4.5110000000000001},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":4.5430000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.7879999999999998},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":5.1920000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215238,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522448,"kb_used_data":71680,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93849392,"statfs":{"total":96636764160,"available":96101777408,"internally_reserved":0,"allocated":73400320,"data_stored":73129360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":461565608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":2.7719999999999998},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.60799999999999998},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71876608,"data_stored":71872995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3661867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143470592,"data_stored":143462907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:26.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:26.895 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:26.895 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:26.907 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-09-16T10:33:26.759146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4117025,"num_objects":1024,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":305285,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15304,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1457272,"kb_used_data":217240,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281658248,"statfs":{"total":289910292480,"available":288418045952,"internally_reserved":0,"allocated":222453760,"data_stored":221567700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,8,11,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19091456,"num_objects":-4661,"num_object_clones":0,"num_object_copies":-9322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4661,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002877"},"pg_stats":[{"pgid":"2.f","version":"48'20422","reported_seq":28890,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:24.203384+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:24.203384+0000","last_peered":"2024-09-16T10:33:24.203384+0000","last_clean":"2024-09-16T10:33:24.203384+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:24.203384+0000","last_undegraded":"2024-09-16T10:33:24.203384+0000","last_fullsized":"2024-09-16T10:33:24.203384+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20422,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20369","reported_seq":20533,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.585914+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:22.585914+0000","last_peered":"2024-09-16T10:33:22.585914+0000","last_clean":"2024-09-16T10:33:22.585914+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:22.585914+0000","last_undegraded":"2024-09-16T10:33:22.585914+0000","last_fullsized":"2024-09-16T10:33:22.585914+0000","mapping_epoch":34,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380928,"num_objects":93,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16582,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20582","reported_seq":20839,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:22.603507+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:22.603507+0000","last_peered":"2024-09-16T10:33:22.603507+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:22.603507+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331800,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16639,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4221}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20376","reported_seq":20439,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.661496+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:22.661496+0000","last_peered":"2024-09-16T10:33:22.661496+0000","last_clean":"2024-09-16T10:33:22.661496+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:22.661496+0000","last_undegraded":"2024-09-16T10:33:22.661496+0000","last_fullsized":"2024-09-16T10:33:22.661496+0000","mapping_epoch":34,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352256,"num_objects":86,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20376,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20455","reported_seq":20616,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:24.240434+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:24.240434+0000","last_peered":"2024-09-16T10:33:24.240434+0000","last_clean":"2024-09-16T10:33:24.240434+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:24.240434+0000","last_undegraded":"2024-09-16T10:33:24.240434+0000","last_fullsized":"2024-09-16T10:33:24.240434+0000","mapping_epoch":33,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17097,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20324","reported_seq":20660,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.122915+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:24.122915+0000","last_peered":"2024-09-16T10:33:24.122915+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:24.122915+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192512,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16949,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20878","reported_seq":21289,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.239233+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:24.239233+0000","last_peered":"2024-09-16T10:33:24.239233+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:24.239233+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172032,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17120,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20347","reported_seq":20529,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:24.208372+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:24.208372+0000","last_peered":"2024-09-16T10:33:24.208372+0000","last_clean":"2024-09-16T10:33:24.208372+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:24.208372+0000","last_undegraded":"2024-09-16T10:33:24.208372+0000","last_fullsized":"2024-09-16T10:33:24.208372+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176128,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16150,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20382","reported_seq":28892,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.368925+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:23.368925+0000","last_peered":"2024-09-16T10:33:23.368925+0000","last_clean":"2024-09-16T10:33:23.368925+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:23.368925+0000","last_undegraded":"2024-09-16T10:33:23.368925+0000","last_fullsized":"2024-09-16T10:33:23.368925+0000","mapping_epoch":43,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324550,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20380,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20985","reported_seq":21265,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.199120+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:24.199120+0000","last_peered":"2024-09-16T10:33:24.199120+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:24.199120+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226246,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20983,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4280}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20334","reported_seq":20534,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:22.632314+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:22.632314+0000","last_peered":"2024-09-16T10:33:22.632314+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:22.632314+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349126,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20332,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20476","reported_seq":28965,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.653281+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:22.653281+0000","last_peered":"2024-09-16T10:33:22.653281+0000","last_clean":"2024-09-16T10:33:22.653281+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:22.653281+0000","last_undegraded":"2024-09-16T10:33:22.653281+0000","last_fullsized":"2024-09-16T10:33:22.653281+0000","mapping_epoch":47,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410566,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20474,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19977","reported_seq":20230,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.170935+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:24.170935+0000","last_peered":"2024-09-16T10:33:24.170935+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:24.170935+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233955,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19976,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20867","reported_seq":21147,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.104255+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:24.104255+0000","last_peered":"2024-09-16T10:33:24.104255+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:24.104255+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193961,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20864,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302601+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:29.302601+0000","last_peered":"2024-09-16T10:31:29.302601+0000","last_clean":"2024-09-16T10:31:29.302601+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:29.302601+0000","last_undegraded":"2024-09-16T10:31:29.302601+0000","last_fullsized":"2024-09-16T10:31:29.302601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20102","reported_seq":24391,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.238657+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:24.238657+0000","last_peered":"2024-09-16T10:33:24.238657+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:24.238657+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156614,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20100,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20842","reported_seq":21088,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.201690+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:24.201690+0000","last_peered":"2024-09-16T10:33:24.201690+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:24.201690+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223116,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20839,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4117006,"num_objects":1022,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":305283,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15303,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":219074560,"data_stored":218997769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019011,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479280,"kb_used_data":140840,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93892560,"statfs":{"total":96636764160,"available":96145981440,"internally_reserved":0,"allocated":144220160,"data_stored":143941514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":5.2229999999999999},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":4.2599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691955,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455544,"kb_used_data":4720,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93916296,"statfs":{"total":96636764160,"available":96170287104,"internally_reserved":0,"allocated":4833280,"data_stored":4496826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":4.5110000000000001},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":4.5430000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.7879999999999998},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":5.1920000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215238,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522448,"kb_used_data":71680,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93849392,"statfs":{"total":96636764160,"available":96101777408,"internally_reserved":0,"allocated":73400320,"data_stored":73129360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":461565608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":2.7719999999999998},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.60799999999999998},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71876608,"data_stored":71872995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3661867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143470592,"data_stored":143462907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:26.909 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:26.909 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20582", 'reported_seq': 20839, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:22.603507+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:33:22.603507+0000', 'last_peered': '2024-09-16T10:33:22.603507+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:22.603507+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 331800, 'num_objects': 82, 'num_object_clones': 0, 'num_object_copies': 164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 82, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16639, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4221}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:26.909 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:26.909 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20324", 'reported_seq': 20660, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:24.122915+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:24.122915+0000', 'last_peered': '2024-09-16T10:33:24.122915+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:24.122915+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 192512, 'num_objects': 47, 'num_object_clones': 0, 'num_object_copies': 94, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 47, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16949, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4087}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:26.909 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:26.909 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20878", 'reported_seq': 21289, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:24.239233+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:24.239233+0000', 'last_peered': '2024-09-16T10:33:24.239233+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:24.239233+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 172032, 'num_objects': 42, 'num_object_clones': 0, 'num_object_copies': 84, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 42, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17120, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4245}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'20985", 'reported_seq': 21265, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:24.199120+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:24.199120+0000', 'last_peered': '2024-09-16T10:33:24.199120+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:24.199120+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 226246, 'num_objects': 57, 'num_object_clones': 0, 'num_object_copies': 114, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 57, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20983, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4280}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20334", 'reported_seq': 20534, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:22.632314+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:22.632314+0000', 'last_peered': '2024-09-16T10:33:22.632314+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:22.632314+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 349126, 'num_objects': 87, 'num_object_clones': 0, 'num_object_copies': 174, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 87, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20332, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4146}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'19977", 'reported_seq': 20230, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:24.170935+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:24.170935+0000', 'last_peered': '2024-09-16T10:33:24.170935+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:24.170935+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'9900", 'ondisk_log_start': "44'9900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 233955, 'num_objects': 58, 'num_object_clones': 0, 'num_object_copies': 116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 58, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 19976, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20867", 'reported_seq': 21147, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:24.104255+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:24.104255+0000', 'last_peered': '2024-09-16T10:33:24.104255+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:24.104255+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 193961, 'num_objects': 50, 'num_object_clones': 0, 'num_object_copies': 100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 50, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20864, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4217}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'20102", 'reported_seq': 24391, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:24.238657+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:33:24.238657+0000', 'last_peered': '2024-09-16T10:33:24.238657+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:33:24.238657+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'10100", 'ondisk_log_start': "44'10100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 156614, 'num_objects': 40, 'num_object_clones': 0, 'num_object_copies': 80, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2069, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 40, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20100, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2102, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:26.910 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:26.911 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20842", 'reported_seq': 21088, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:24.201690+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:24.201690+0000', 'last_peered': '2024-09-16T10:33:24.201690+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:24.201690+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 223116, 'num_objects': 58, 'num_object_clones': 0, 'num_object_copies': 116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 58, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20839, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:26.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:27.324 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:27.324 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36043/2048 objects degraded (1759.912%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":696,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":1024,"data_bytes":4117025,"bytes_used":1492246528,"bytes_avail":288418045952,"bytes_total":289910292480,"degraded_objects":36043,"degraded_total":2048,"degraded_ratio":17.59912109375,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":200,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":388},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:27.337 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:27.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:27.633 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:27.634 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:27.647 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-09-16T10:33:26.759146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4117025,"num_objects":1024,"num_object_clones":0,"num_object_copies":2048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":305285,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15304,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1457272,"kb_used_data":217240,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281658248,"statfs":{"total":289910292480,"available":288418045952,"internally_reserved":0,"allocated":222453760,"data_stored":221567700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,8,11,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19091456,"num_objects":-4661,"num_object_clones":0,"num_object_copies":-9322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4661,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002877"},"pg_stats":[{"pgid":"2.f","version":"48'20422","reported_seq":28890,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:24.203384+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:24.203384+0000","last_peered":"2024-09-16T10:33:24.203384+0000","last_clean":"2024-09-16T10:33:24.203384+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:24.203384+0000","last_undegraded":"2024-09-16T10:33:24.203384+0000","last_fullsized":"2024-09-16T10:33:24.203384+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188416,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20422,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20369","reported_seq":20533,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.585914+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:22.585914+0000","last_peered":"2024-09-16T10:33:22.585914+0000","last_clean":"2024-09-16T10:33:22.585914+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:22.585914+0000","last_undegraded":"2024-09-16T10:33:22.585914+0000","last_fullsized":"2024-09-16T10:33:22.585914+0000","mapping_epoch":34,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380928,"num_objects":93,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16582,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20582","reported_seq":20839,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:22.603507+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:22.603507+0000","last_peered":"2024-09-16T10:33:22.603507+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:22.603507+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331800,"num_objects":82,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16639,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4221}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20376","reported_seq":20439,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.661496+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:22.661496+0000","last_peered":"2024-09-16T10:33:22.661496+0000","last_clean":"2024-09-16T10:33:22.661496+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:22.661496+0000","last_undegraded":"2024-09-16T10:33:22.661496+0000","last_fullsized":"2024-09-16T10:33:22.661496+0000","mapping_epoch":34,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352256,"num_objects":86,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20376,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20455","reported_seq":20616,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:24.240434+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:24.240434+0000","last_peered":"2024-09-16T10:33:24.240434+0000","last_clean":"2024-09-16T10:33:24.240434+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:24.240434+0000","last_undegraded":"2024-09-16T10:33:24.240434+0000","last_fullsized":"2024-09-16T10:33:24.240434+0000","mapping_epoch":33,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204800,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17097,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20324","reported_seq":20660,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.122915+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:24.122915+0000","last_peered":"2024-09-16T10:33:24.122915+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:24.122915+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192512,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16949,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20878","reported_seq":21289,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.239233+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:24.239233+0000","last_peered":"2024-09-16T10:33:24.239233+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:24.239233+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172032,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17120,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4245}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20347","reported_seq":20529,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:24.208372+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:24.208372+0000","last_peered":"2024-09-16T10:33:24.208372+0000","last_clean":"2024-09-16T10:33:24.208372+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:24.208372+0000","last_undegraded":"2024-09-16T10:33:24.208372+0000","last_fullsized":"2024-09-16T10:33:24.208372+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176128,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16150,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20382","reported_seq":28892,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:23.368925+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:23.368925+0000","last_peered":"2024-09-16T10:33:23.368925+0000","last_clean":"2024-09-16T10:33:23.368925+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:23.368925+0000","last_undegraded":"2024-09-16T10:33:23.368925+0000","last_fullsized":"2024-09-16T10:33:23.368925+0000","mapping_epoch":43,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324550,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20380,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'20985","reported_seq":21265,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.199120+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:24.199120+0000","last_peered":"2024-09-16T10:33:24.199120+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:24.199120+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226246,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20983,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4280}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20334","reported_seq":20534,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:22.632314+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:22.632314+0000","last_peered":"2024-09-16T10:33:22.632314+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:22.632314+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349126,"num_objects":87,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20332,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4146}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20476","reported_seq":28965,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:22.653281+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:22.653281+0000","last_peered":"2024-09-16T10:33:22.653281+0000","last_clean":"2024-09-16T10:33:22.653281+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:22.653281+0000","last_undegraded":"2024-09-16T10:33:22.653281+0000","last_fullsized":"2024-09-16T10:33:22.653281+0000","mapping_epoch":47,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410566,"num_objects":102,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20474,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'19977","reported_seq":20230,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.170935+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:24.170935+0000","last_peered":"2024-09-16T10:33:24.170935+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:24.170935+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'9900","ondisk_log_start":"44'9900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233955,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19976,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20867","reported_seq":21147,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.104255+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:24.104255+0000","last_peered":"2024-09-16T10:33:24.104255+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:24.104255+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193961,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20864,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4217}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302601+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:29.302601+0000","last_peered":"2024-09-16T10:31:29.302601+0000","last_clean":"2024-09-16T10:31:29.302601+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:29.302601+0000","last_undegraded":"2024-09-16T10:31:29.302601+0000","last_fullsized":"2024-09-16T10:31:29.302601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20102","reported_seq":24391,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.238657+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:24.238657+0000","last_peered":"2024-09-16T10:33:24.238657+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:24.238657+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156614,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2069,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20100,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2029}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20842","reported_seq":21088,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:24.201690+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:24.201690+0000","last_peered":"2024-09-16T10:33:24.201690+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:24.201690+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223116,"num_objects":58,"num_object_clones":0,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20839,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4117006,"num_objects":1022,"num_object_clones":0,"num_object_copies":2044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1022,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":305283,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15303,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":219074560,"data_stored":218997769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019011,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479280,"kb_used_data":140840,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93892560,"statfs":{"total":96636764160,"available":96145981440,"internally_reserved":0,"allocated":144220160,"data_stored":143941514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":346549936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":5.2229999999999999},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":4.2599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691955,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455544,"kb_used_data":4720,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93916296,"statfs":{"total":96636764160,"available":96170287104,"internally_reserved":0,"allocated":4833280,"data_stored":4496826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,6,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":4.5110000000000001},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":4.5430000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":3.7879999999999998},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":5.1920000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215238,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522448,"kb_used_data":71680,"kb_used_omap":4,"kb_used_meta":450747,"kb_avail":93849392,"statfs":{"total":96636764160,"available":96101777408,"internally_reserved":0,"allocated":73400320,"data_stored":73129360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":461565608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2,6,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":2.7719999999999998},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.47499999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.60799999999999998},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.214}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71876608,"data_stored":71872995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3661867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":143470592,"data_stored":143462907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:30.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:30.957 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:30.957 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:30.971 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-16T10:33:30.759854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15920,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1446504,"kb_used_data":206480,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281669016,"statfs":{"total":289910292480,"available":288429072384,"internally_reserved":0,"allocated":211435520,"data_stored":210557604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19648536,"num_objects":-4798,"num_object_clones":0,"num_object_copies":-9596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4798,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002665"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28936,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.747283+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:25.747283+0000","last_peered":"2024-09-16T10:33:25.747283+0000","last_clean":"2024-09-16T10:33:25.747283+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:25.747283+0000","last_undegraded":"2024-09-16T10:33:25.747283+0000","last_fullsized":"2024-09-16T10:33:25.747283+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20626,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.714969+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:25.714969+0000","last_peered":"2024-09-16T10:33:25.714969+0000","last_clean":"2024-09-16T10:33:25.714969+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:25.714969+0000","last_undegraded":"2024-09-16T10:33:25.714969+0000","last_fullsized":"2024-09-16T10:33:25.714969+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20921,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.681964+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:25.681964+0000","last_peered":"2024-09-16T10:33:25.681964+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:25.681964+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4303}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20525,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.713450+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:25.713450+0000","last_peered":"2024-09-16T10:33:25.713450+0000","last_clean":"2024-09-16T10:33:25.713450+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:25.713450+0000","last_undegraded":"2024-09-16T10:33:25.713450+0000","last_fullsized":"2024-09-16T10:33:25.713450+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20668,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.769984+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:25.769984+0000","last_peered":"2024-09-16T10:33:25.769984+0000","last_clean":"2024-09-16T10:33:25.769984+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:25.769984+0000","last_undegraded":"2024-09-16T10:33:25.769984+0000","last_fullsized":"2024-09-16T10:33:25.769984+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20707,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.798394+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:25.798394+0000","last_peered":"2024-09-16T10:33:25.798394+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:25.798394+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21332,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.684300+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:25.684300+0000","last_peered":"2024-09-16T10:33:25.684300+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:25.684300+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20572,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.712059+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:25.712059+0000","last_peered":"2024-09-16T10:33:25.712059+0000","last_clean":"2024-09-16T10:33:25.712059+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:25.712059+0000","last_undegraded":"2024-09-16T10:33:25.712059+0000","last_fullsized":"2024-09-16T10:33:25.712059+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28971,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.709091+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:25.709091+0000","last_peered":"2024-09-16T10:33:25.709091+0000","last_clean":"2024-09-16T10:33:25.709091+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:25.709091+0000","last_undegraded":"2024-09-16T10:33:25.709091+0000","last_fullsized":"2024-09-16T10:33:25.709091+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21320,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.797215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:25.797215+0000","last_peered":"2024-09-16T10:33:25.797215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:25.797215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.772599+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:25.772599+0000","last_peered":"2024-09-16T10:33:25.772599+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:25.772599+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29065,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.766197+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:25.766197+0000","last_peered":"2024-09-16T10:33:25.766197+0000","last_clean":"2024-09-16T10:33:25.766197+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:25.766197+0000","last_undegraded":"2024-09-16T10:33:25.766197+0000","last_fullsized":"2024-09-16T10:33:25.766197+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20288,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.771018+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:25.771018+0000","last_peered":"2024-09-16T10:33:25.771018+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:25.771018+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21194,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.672406+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:25.672406+0000","last_peered":"2024-09-16T10:33:25.672406+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:25.672406+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302601+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:29.302601+0000","last_peered":"2024-09-16T10:31:29.302601+0000","last_clean":"2024-09-16T10:31:29.302601+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:29.302601+0000","last_undegraded":"2024-09-16T10:31:29.302601+0000","last_fullsized":"2024-09-16T10:31:29.302601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":25661,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:29.277213+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:29.277213+0000","last_peered":"2024-09-16T10:33:29.277213+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:29.277213+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21142,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.759598+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:25.759598+0000","last_peered":"2024-09-16T10:33:25.759598+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:25.759598+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15919,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210223104,"data_stored":210154457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019012,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480152,"kb_used_data":137616,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891688,"statfs":{"total":96636764160,"available":96145088512,"internally_reserved":0,"allocated":140918784,"data_stored":140644210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.251},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23400000000000001},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691956,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443096,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93928744,"statfs":{"total":96636764160,"available":96183033856,"internally_reserved":0,"allocated":475136,"data_stored":142754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":4.1420000000000003},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":15.151}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28899999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215239,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523256,"kb_used_data":68400,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848584,"statfs":{"total":96636764160,"available":96100950016,"internally_reserved":0,"allocated":70041600,"data_stored":69770640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.188},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.17000000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.20000000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":8211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:30.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:31.287 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:31.287 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:31.300 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-16T10:33:30.759854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15920,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1446504,"kb_used_data":206480,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281669016,"statfs":{"total":289910292480,"available":288429072384,"internally_reserved":0,"allocated":211435520,"data_stored":210557604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19648536,"num_objects":-4798,"num_object_clones":0,"num_object_copies":-9596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4798,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002665"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28936,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.747283+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:25.747283+0000","last_peered":"2024-09-16T10:33:25.747283+0000","last_clean":"2024-09-16T10:33:25.747283+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:25.747283+0000","last_undegraded":"2024-09-16T10:33:25.747283+0000","last_fullsized":"2024-09-16T10:33:25.747283+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20626,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.714969+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:25.714969+0000","last_peered":"2024-09-16T10:33:25.714969+0000","last_clean":"2024-09-16T10:33:25.714969+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:25.714969+0000","last_undegraded":"2024-09-16T10:33:25.714969+0000","last_fullsized":"2024-09-16T10:33:25.714969+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20921,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.681964+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:25.681964+0000","last_peered":"2024-09-16T10:33:25.681964+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:25.681964+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4303}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20525,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.713450+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:25.713450+0000","last_peered":"2024-09-16T10:33:25.713450+0000","last_clean":"2024-09-16T10:33:25.713450+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:25.713450+0000","last_undegraded":"2024-09-16T10:33:25.713450+0000","last_fullsized":"2024-09-16T10:33:25.713450+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20668,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.769984+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:25.769984+0000","last_peered":"2024-09-16T10:33:25.769984+0000","last_clean":"2024-09-16T10:33:25.769984+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:25.769984+0000","last_undegraded":"2024-09-16T10:33:25.769984+0000","last_fullsized":"2024-09-16T10:33:25.769984+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20707,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.798394+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:25.798394+0000","last_peered":"2024-09-16T10:33:25.798394+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:25.798394+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21332,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.684300+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:25.684300+0000","last_peered":"2024-09-16T10:33:25.684300+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:25.684300+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20572,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.712059+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:25.712059+0000","last_peered":"2024-09-16T10:33:25.712059+0000","last_clean":"2024-09-16T10:33:25.712059+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:25.712059+0000","last_undegraded":"2024-09-16T10:33:25.712059+0000","last_fullsized":"2024-09-16T10:33:25.712059+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28971,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.709091+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:25.709091+0000","last_peered":"2024-09-16T10:33:25.709091+0000","last_clean":"2024-09-16T10:33:25.709091+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:25.709091+0000","last_undegraded":"2024-09-16T10:33:25.709091+0000","last_fullsized":"2024-09-16T10:33:25.709091+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21320,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.797215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:25.797215+0000","last_peered":"2024-09-16T10:33:25.797215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:25.797215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.772599+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:25.772599+0000","last_peered":"2024-09-16T10:33:25.772599+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:25.772599+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29065,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.766197+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:25.766197+0000","last_peered":"2024-09-16T10:33:25.766197+0000","last_clean":"2024-09-16T10:33:25.766197+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:25.766197+0000","last_undegraded":"2024-09-16T10:33:25.766197+0000","last_fullsized":"2024-09-16T10:33:25.766197+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20288,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.771018+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:25.771018+0000","last_peered":"2024-09-16T10:33:25.771018+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:25.771018+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21194,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.672406+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:25.672406+0000","last_peered":"2024-09-16T10:33:25.672406+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:25.672406+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302601+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:29.302601+0000","last_peered":"2024-09-16T10:31:29.302601+0000","last_clean":"2024-09-16T10:31:29.302601+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:29.302601+0000","last_undegraded":"2024-09-16T10:31:29.302601+0000","last_fullsized":"2024-09-16T10:31:29.302601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":25661,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:29.277213+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:29.277213+0000","last_peered":"2024-09-16T10:33:29.277213+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:29.277213+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21142,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.759598+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:25.759598+0000","last_peered":"2024-09-16T10:33:25.759598+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:25.759598+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15919,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210223104,"data_stored":210154457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019012,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480152,"kb_used_data":137616,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891688,"statfs":{"total":96636764160,"available":96145088512,"internally_reserved":0,"allocated":140918784,"data_stored":140644210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.251},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23400000000000001},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691956,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443096,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93928744,"statfs":{"total":96636764160,"available":96183033856,"internally_reserved":0,"allocated":475136,"data_stored":142754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":4.1420000000000003},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":15.151}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28899999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215239,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523256,"kb_used_data":68400,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848584,"statfs":{"total":96636764160,"available":96100950016,"internally_reserved":0,"allocated":70041600,"data_stored":69770640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.188},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.17000000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.20000000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":8211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:31.301 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:31.301 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20664", 'reported_seq': 20921, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.681964+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:33:25.681964+0000', 'last_peered': '2024-09-16T10:33:25.681964+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:25.681964+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10600", 'ondisk_log_start': "44'10600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16721, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:31.301 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:31.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 20707, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.798394+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:25.798394+0000', 'last_peered': '2024-09-16T10:33:25.798394+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:25.798394+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:31.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:31.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21332, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.684300+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:25.684300+0000', 'last_peered': '2024-09-16T10:33:25.684300+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:25.684300+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:31.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:31.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21320, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.797215+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:25.797215+0000', 'last_peered': '2024-09-16T10:33:25.797215+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:25.797215+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:31.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:31.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.772599+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:25.772599+0000', 'last_peered': '2024-09-16T10:33:25.772599+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:25.772599+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:31.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:31.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20288, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.771018+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:25.771018+0000', 'last_peered': '2024-09-16T10:33:25.771018+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:25.771018+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:31.302 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:31.303 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21194, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.672406+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:25.672406+0000', 'last_peered': '2024-09-16T10:33:25.672406+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:25.672406+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:31.303 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:33:31.303 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'20140", 'reported_seq': 25661, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:29.277213+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:33:29.277213+0000', 'last_peered': '2024-09-16T10:33:29.277213+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:33:29.277213+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'10100", 'ondisk_log_start': "44'10100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1453, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20138, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2718, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1451}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:31.303 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:31.303 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21142, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.759598+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:25.759598+0000', 'last_peered': '2024-09-16T10:33:25.759598+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:25.759598+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:31.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:31.662 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35427/40 objects degraded (88567.500%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":700,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":48,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1481220096,"bytes_avail":288429072384,"bytes_total":289910292480,"degraded_objects":35427,"degraded_total":40,"degraded_ratio":885.67499999999995,"recovering_objects_per_sec":67,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":399},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:31.675 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:31.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:31.974 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:31.975 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:31.987 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2024-09-16T10:33:30.759854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15920,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1446504,"kb_used_data":206480,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281669016,"statfs":{"total":289910292480,"available":288429072384,"internally_reserved":0,"allocated":211435520,"data_stored":210557604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19648536,"num_objects":-4798,"num_object_clones":0,"num_object_copies":-9596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4798,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002665"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28936,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.747283+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:25.747283+0000","last_peered":"2024-09-16T10:33:25.747283+0000","last_clean":"2024-09-16T10:33:25.747283+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:25.747283+0000","last_undegraded":"2024-09-16T10:33:25.747283+0000","last_fullsized":"2024-09-16T10:33:25.747283+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20626,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.714969+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:25.714969+0000","last_peered":"2024-09-16T10:33:25.714969+0000","last_clean":"2024-09-16T10:33:25.714969+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:25.714969+0000","last_undegraded":"2024-09-16T10:33:25.714969+0000","last_fullsized":"2024-09-16T10:33:25.714969+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20921,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.681964+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:25.681964+0000","last_peered":"2024-09-16T10:33:25.681964+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:25.681964+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4303}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20525,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.713450+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:25.713450+0000","last_peered":"2024-09-16T10:33:25.713450+0000","last_clean":"2024-09-16T10:33:25.713450+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:25.713450+0000","last_undegraded":"2024-09-16T10:33:25.713450+0000","last_fullsized":"2024-09-16T10:33:25.713450+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20668,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.769984+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:25.769984+0000","last_peered":"2024-09-16T10:33:25.769984+0000","last_clean":"2024-09-16T10:33:25.769984+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:25.769984+0000","last_undegraded":"2024-09-16T10:33:25.769984+0000","last_fullsized":"2024-09-16T10:33:25.769984+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20707,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.798394+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:25.798394+0000","last_peered":"2024-09-16T10:33:25.798394+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:25.798394+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21332,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.684300+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:25.684300+0000","last_peered":"2024-09-16T10:33:25.684300+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:25.684300+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20572,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.712059+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:25.712059+0000","last_peered":"2024-09-16T10:33:25.712059+0000","last_clean":"2024-09-16T10:33:25.712059+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:25.712059+0000","last_undegraded":"2024-09-16T10:33:25.712059+0000","last_fullsized":"2024-09-16T10:33:25.712059+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28971,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.709091+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:25.709091+0000","last_peered":"2024-09-16T10:33:25.709091+0000","last_clean":"2024-09-16T10:33:25.709091+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:25.709091+0000","last_undegraded":"2024-09-16T10:33:25.709091+0000","last_fullsized":"2024-09-16T10:33:25.709091+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21320,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.797215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:25.797215+0000","last_peered":"2024-09-16T10:33:25.797215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:25.797215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.772599+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:25.772599+0000","last_peered":"2024-09-16T10:33:25.772599+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:25.772599+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29065,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.766197+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:25.766197+0000","last_peered":"2024-09-16T10:33:25.766197+0000","last_clean":"2024-09-16T10:33:25.766197+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:25.766197+0000","last_undegraded":"2024-09-16T10:33:25.766197+0000","last_fullsized":"2024-09-16T10:33:25.766197+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20288,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.771018+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:25.771018+0000","last_peered":"2024-09-16T10:33:25.771018+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:25.771018+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21194,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.672406+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:25.672406+0000","last_peered":"2024-09-16T10:33:25.672406+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:25.672406+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302601+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:29.302601+0000","last_peered":"2024-09-16T10:31:29.302601+0000","last_clean":"2024-09-16T10:31:29.302601+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:29.302601+0000","last_undegraded":"2024-09-16T10:31:29.302601+0000","last_fullsized":"2024-09-16T10:31:29.302601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":25661,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:29.277213+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:29.277213+0000","last_peered":"2024-09-16T10:33:29.277213+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:29.277213+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2718,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21142,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.759598+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:25.759598+0000","last_peered":"2024-09-16T10:33:25.759598+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:25.759598+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15919,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210223104,"data_stored":210154457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019012,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480152,"kb_used_data":137616,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891688,"statfs":{"total":96636764160,"available":96145088512,"internally_reserved":0,"allocated":140918784,"data_stored":140644210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.251},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23400000000000001},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691956,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443096,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93928744,"statfs":{"total":96636764160,"available":96183033856,"internally_reserved":0,"allocated":475136,"data_stored":142754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":4.1420000000000003},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":15.151}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28899999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215239,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523256,"kb_used_data":68400,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848584,"statfs":{"total":96636764160,"available":96100950016,"internally_reserved":0,"allocated":70041600,"data_stored":69770640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.188},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.17000000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.20000000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":8211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:34.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:35.298 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:35.298 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:35.311 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-16T10:33:34.760632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17088,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1454696,"kb_used_data":206480,"kb_used_omap":12,"kb_used_meta":1248179,"kb_avail":281660824,"statfs":{"total":289910292480,"available":288420683776,"internally_reserved":0,"allocated":211435520,"data_stored":210557604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1278135298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10768408,"num_objects":-2630,"num_object_clones":0,"num_object_copies":-5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1881,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002459"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28936,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.747283+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:25.747283+0000","last_peered":"2024-09-16T10:33:25.747283+0000","last_clean":"2024-09-16T10:33:25.747283+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:25.747283+0000","last_undegraded":"2024-09-16T10:33:25.747283+0000","last_fullsized":"2024-09-16T10:33:25.747283+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20626,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.714969+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:25.714969+0000","last_peered":"2024-09-16T10:33:25.714969+0000","last_clean":"2024-09-16T10:33:25.714969+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:25.714969+0000","last_undegraded":"2024-09-16T10:33:25.714969+0000","last_fullsized":"2024-09-16T10:33:25.714969+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20921,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.681964+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:25.681964+0000","last_peered":"2024-09-16T10:33:25.681964+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:25.681964+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4303}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20525,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.713450+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:25.713450+0000","last_peered":"2024-09-16T10:33:25.713450+0000","last_clean":"2024-09-16T10:33:25.713450+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:25.713450+0000","last_undegraded":"2024-09-16T10:33:25.713450+0000","last_fullsized":"2024-09-16T10:33:25.713450+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.770303+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:25.770303+0000","last_peered":"2024-09-16T10:33:25.770303+0000","last_clean":"2024-09-16T10:33:25.770303+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:25.770303+0000","last_undegraded":"2024-09-16T10:33:25.770303+0000","last_fullsized":"2024-09-16T10:33:25.770303+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20707,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.798394+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:25.798394+0000","last_peered":"2024-09-16T10:33:25.798394+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:25.798394+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21332,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.684300+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:25.684300+0000","last_peered":"2024-09-16T10:33:25.684300+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:25.684300+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20572,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.712059+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:25.712059+0000","last_peered":"2024-09-16T10:33:25.712059+0000","last_clean":"2024-09-16T10:33:25.712059+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:25.712059+0000","last_undegraded":"2024-09-16T10:33:25.712059+0000","last_fullsized":"2024-09-16T10:33:25.712059+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28971,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.709091+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:25.709091+0000","last_peered":"2024-09-16T10:33:25.709091+0000","last_clean":"2024-09-16T10:33:25.709091+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:25.709091+0000","last_undegraded":"2024-09-16T10:33:25.709091+0000","last_fullsized":"2024-09-16T10:33:25.709091+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21320,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.797215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:25.797215+0000","last_peered":"2024-09-16T10:33:25.797215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:25.797215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.772599+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:25.772599+0000","last_peered":"2024-09-16T10:33:25.772599+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:25.772599+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29065,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.766197+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:25.766197+0000","last_peered":"2024-09-16T10:33:25.766197+0000","last_clean":"2024-09-16T10:33:25.766197+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:25.766197+0000","last_undegraded":"2024-09-16T10:33:25.766197+0000","last_fullsized":"2024-09-16T10:33:25.766197+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20288,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.771018+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:25.771018+0000","last_peered":"2024-09-16T10:33:25.771018+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:25.771018+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21194,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.672406+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:25.672406+0000","last_peered":"2024-09-16T10:33:25.672406+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:25.672406+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302601+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:29.302601+0000","last_peered":"2024-09-16T10:31:29.302601+0000","last_clean":"2024-09-16T10:31:29.302601+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:29.302601+0000","last_undegraded":"2024-09-16T10:31:29.302601+0000","last_fullsized":"2024-09-16T10:31:29.302601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":27997,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:34.279256+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:34.279256+0000","last_peered":"2024-09-16T10:33:34.279256+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:34.279256+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3886,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21142,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.759598+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:25.759598+0000","last_peered":"2024-09-16T10:33:25.759598+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:25.759598+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17087,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210223104,"data_stored":210154457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019013,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480152,"kb_used_data":137616,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891688,"statfs":{"total":96636764160,"available":96145088512,"internally_reserved":0,"allocated":140918784,"data_stored":140644210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.187},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.20200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.217},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.255}]}]},{"osd":1,"up_from":40,"seq":171798691957,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451288,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93920552,"statfs":{"total":96636764160,"available":96174645248,"internally_reserved":0,"allocated":475136,"data_stored":142754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.16900000000000001},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.20699999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.222}]}]},{"osd":0,"up_from":24,"seq":103079215240,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523256,"kb_used_data":68400,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848584,"statfs":{"total":96636764160,"available":96100950016,"internally_reserved":0,"allocated":70041600,"data_stored":69770640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.223},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.20999999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":8211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:35.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:35.605 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:35.606 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:35.618 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-16T10:33:34.760632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17088,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1454696,"kb_used_data":206480,"kb_used_omap":12,"kb_used_meta":1248179,"kb_avail":281660824,"statfs":{"total":289910292480,"available":288420683776,"internally_reserved":0,"allocated":211435520,"data_stored":210557604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1278135298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10768408,"num_objects":-2630,"num_object_clones":0,"num_object_copies":-5260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1881,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1881,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002459"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28936,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.747283+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:25.747283+0000","last_peered":"2024-09-16T10:33:25.747283+0000","last_clean":"2024-09-16T10:33:25.747283+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:25.747283+0000","last_undegraded":"2024-09-16T10:33:25.747283+0000","last_fullsized":"2024-09-16T10:33:25.747283+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20626,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.714969+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:25.714969+0000","last_peered":"2024-09-16T10:33:25.714969+0000","last_clean":"2024-09-16T10:33:25.714969+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:25.714969+0000","last_undegraded":"2024-09-16T10:33:25.714969+0000","last_fullsized":"2024-09-16T10:33:25.714969+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20921,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.681964+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:25.681964+0000","last_peered":"2024-09-16T10:33:25.681964+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:25.681964+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4303}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20525,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.713450+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:25.713450+0000","last_peered":"2024-09-16T10:33:25.713450+0000","last_clean":"2024-09-16T10:33:25.713450+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:25.713450+0000","last_undegraded":"2024-09-16T10:33:25.713450+0000","last_fullsized":"2024-09-16T10:33:25.713450+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.770303+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:25.770303+0000","last_peered":"2024-09-16T10:33:25.770303+0000","last_clean":"2024-09-16T10:33:25.770303+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:25.770303+0000","last_undegraded":"2024-09-16T10:33:25.770303+0000","last_fullsized":"2024-09-16T10:33:25.770303+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20707,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.798394+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:25.798394+0000","last_peered":"2024-09-16T10:33:25.798394+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:25.798394+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21332,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.684300+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:25.684300+0000","last_peered":"2024-09-16T10:33:25.684300+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:25.684300+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20572,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.712059+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:25.712059+0000","last_peered":"2024-09-16T10:33:25.712059+0000","last_clean":"2024-09-16T10:33:25.712059+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:25.712059+0000","last_undegraded":"2024-09-16T10:33:25.712059+0000","last_fullsized":"2024-09-16T10:33:25.712059+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28971,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.709091+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:25.709091+0000","last_peered":"2024-09-16T10:33:25.709091+0000","last_clean":"2024-09-16T10:33:25.709091+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:25.709091+0000","last_undegraded":"2024-09-16T10:33:25.709091+0000","last_fullsized":"2024-09-16T10:33:25.709091+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21320,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.797215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:25.797215+0000","last_peered":"2024-09-16T10:33:25.797215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:25.797215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.772599+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:25.772599+0000","last_peered":"2024-09-16T10:33:25.772599+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:25.772599+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29065,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.766197+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:25.766197+0000","last_peered":"2024-09-16T10:33:25.766197+0000","last_clean":"2024-09-16T10:33:25.766197+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:25.766197+0000","last_undegraded":"2024-09-16T10:33:25.766197+0000","last_fullsized":"2024-09-16T10:33:25.766197+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20288,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.771018+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:25.771018+0000","last_peered":"2024-09-16T10:33:25.771018+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:25.771018+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21194,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.672406+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:25.672406+0000","last_peered":"2024-09-16T10:33:25.672406+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:25.672406+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302601+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:29.302601+0000","last_peered":"2024-09-16T10:31:29.302601+0000","last_clean":"2024-09-16T10:31:29.302601+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:29.302601+0000","last_undegraded":"2024-09-16T10:31:29.302601+0000","last_fullsized":"2024-09-16T10:31:29.302601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":27997,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:34.279256+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:34.279256+0000","last_peered":"2024-09-16T10:33:34.279256+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:34.279256+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3886,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21142,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.759598+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:25.759598+0000","last_peered":"2024-09-16T10:33:25.759598+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:25.759598+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17087,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210223104,"data_stored":210154457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019013,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480152,"kb_used_data":137616,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891688,"statfs":{"total":96636764160,"available":96145088512,"internally_reserved":0,"allocated":140918784,"data_stored":140644210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.187},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.20200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.217},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.255}]}]},{"osd":1,"up_from":40,"seq":171798691957,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451288,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93920552,"statfs":{"total":96636764160,"available":96174645248,"internally_reserved":0,"allocated":475136,"data_stored":142754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.16900000000000001},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.20699999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.222}]}]},{"osd":0,"up_from":24,"seq":103079215240,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523256,"kb_used_data":68400,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848584,"statfs":{"total":96636764160,"available":96100950016,"internally_reserved":0,"allocated":70041600,"data_stored":69770640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.223},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.20999999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":8211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:35.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:35.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20664", 'reported_seq': 20921, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.681964+0000', 'last_change': '2024-09-16T10:23:56.881966+0000', 'last_active': '2024-09-16T10:33:25.681964+0000', 'last_peered': '2024-09-16T10:33:25.681964+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:25.681964+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10600", 'ondisk_log_start': "44'10600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4304, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16721, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 32, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:35.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:35.619 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 20707, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.798394+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:25.798394+0000', 'last_peered': '2024-09-16T10:33:25.798394+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:25.798394+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21332, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.684300+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:25.684300+0000', 'last_peered': '2024-09-16T10:33:25.684300+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:25.684300+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21320, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.797215+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:25.797215+0000', 'last_peered': '2024-09-16T10:33:25.797215+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:25.797215+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20620, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.772599+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:25.772599+0000', 'last_peered': '2024-09-16T10:33:25.772599+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:25.772599+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20288, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.771018+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:25.771018+0000', 'last_peered': '2024-09-16T10:33:25.771018+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:25.771018+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21194, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.672406+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:25.672406+0000', 'last_peered': '2024-09-16T10:33:25.672406+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:25.672406+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:35.620 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.5 is not active+clean 2024-09-16T10:33:35.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.5', 'version': "48'20140", 'reported_seq': 27997, 'reported_epoch': 48, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:34.279256+0000', 'last_change': '2024-09-16T10:31:27.294379+0000', 'last_active': '2024-09-16T10:33:34.279256+0000', 'last_peered': '2024-09-16T10:33:34.279256+0000', 'last_clean': '2024-09-16T10:23:54.331548+0000', 'last_became_active': '2024-09-16T10:23:56.894775+0000', 'last_became_peered': '2024-09-16T10:23:56.894775+0000', 'last_unstale': '2024-09-16T10:33:34.279256+0000', 'last_undegraded': '2024-09-16T10:23:56.433192+0000', 'last_fullsized': '2024-09-16T10:23:56.366848+0000', 'mapping_epoch': 41, 'log_start': "44'10100", 'ondisk_log_start': "44'10100", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20138, 'num_write_kb': 40274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3886, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -283}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:35.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:35.621 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21142, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.759598+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:25.759598+0000', 'last_peered': '2024-09-16T10:33:25.759598+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:25.759598+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:35.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:36.240 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:36.240 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34259/40 objects degraded (85647.500%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":705,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":49,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1489608704,"bytes_avail":288420683776,"bytes_total":289910292480,"degraded_objects":34259,"degraded_total":40,"degraded_ratio":856.47500000000002,"recovering_objects_per_sec":156,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1881,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":219},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:36.254 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:36.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:36.580 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:36.580 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:36.594 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-16T10:33:35.896185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17088,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":39,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1454696,"kb_used_data":206480,"kb_used_omap":12,"kb_used_meta":1248179,"kb_avail":281660824,"statfs":{"total":289910292480,"available":288420683776,"internally_reserved":0,"allocated":211435520,"data_stored":210557604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1278135298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4108312,"num_objects":-1004,"num_object_clones":0,"num_object_copies":-2008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1784,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1004,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1784,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.137536"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28936,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.747283+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:25.747283+0000","last_peered":"2024-09-16T10:33:25.747283+0000","last_clean":"2024-09-16T10:33:25.747283+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:25.747283+0000","last_undegraded":"2024-09-16T10:33:25.747283+0000","last_fullsized":"2024-09-16T10:33:25.747283+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20626,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.714969+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:25.714969+0000","last_peered":"2024-09-16T10:33:25.714969+0000","last_clean":"2024-09-16T10:33:25.714969+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:25.714969+0000","last_undegraded":"2024-09-16T10:33:25.714969+0000","last_fullsized":"2024-09-16T10:33:25.714969+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20921,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.681964+0000","last_change":"2024-09-16T10:23:56.881966+0000","last_active":"2024-09-16T10:33:25.681964+0000","last_peered":"2024-09-16T10:33:25.681964+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:25.681964+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4304,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4303}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20525,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.713450+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:25.713450+0000","last_peered":"2024-09-16T10:33:25.713450+0000","last_clean":"2024-09-16T10:33:25.713450+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:25.713450+0000","last_undegraded":"2024-09-16T10:33:25.713450+0000","last_fullsized":"2024-09-16T10:33:25.713450+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.770303+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:25.770303+0000","last_peered":"2024-09-16T10:33:25.770303+0000","last_clean":"2024-09-16T10:33:25.770303+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:25.770303+0000","last_undegraded":"2024-09-16T10:33:25.770303+0000","last_fullsized":"2024-09-16T10:33:25.770303+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20707,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.798394+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:25.798394+0000","last_peered":"2024-09-16T10:33:25.798394+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:25.798394+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21332,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.684300+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:25.684300+0000","last_peered":"2024-09-16T10:33:25.684300+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:25.684300+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20572,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.712059+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:25.712059+0000","last_peered":"2024-09-16T10:33:25.712059+0000","last_clean":"2024-09-16T10:33:25.712059+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:25.712059+0000","last_undegraded":"2024-09-16T10:33:25.712059+0000","last_fullsized":"2024-09-16T10:33:25.712059+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302730+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:31:29.302730+0000","last_peered":"2024-09-16T10:31:29.302730+0000","last_clean":"2024-09-16T10:31:29.302730+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:31:29.302730+0000","last_undegraded":"2024-09-16T10:31:29.302730+0000","last_fullsized":"2024-09-16T10:31:29.302730+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28971,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.709091+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:25.709091+0000","last_peered":"2024-09-16T10:33:25.709091+0000","last_clean":"2024-09-16T10:33:25.709091+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:25.709091+0000","last_undegraded":"2024-09-16T10:33:25.709091+0000","last_fullsized":"2024-09-16T10:33:25.709091+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.301170+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:31:29.301170+0000","last_peered":"2024-09-16T10:31:29.301170+0000","last_clean":"2024-09-16T10:31:29.301170+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:31:29.301170+0000","last_undegraded":"2024-09-16T10:31:29.301170+0000","last_fullsized":"2024-09-16T10:31:29.301170+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21320,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.797215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:25.797215+0000","last_peered":"2024-09-16T10:33:25.797215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:25.797215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20620,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.772599+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:25.772599+0000","last_peered":"2024-09-16T10:33:25.772599+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:25.772599+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":75,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302755+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:31:29.302755+0000","last_peered":"2024-09-16T10:31:29.302755+0000","last_clean":"2024-09-16T10:31:29.302755+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:31:29.302755+0000","last_undegraded":"2024-09-16T10:31:29.302755+0000","last_fullsized":"2024-09-16T10:31:29.302755+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29065,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.766197+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:25.766197+0000","last_peered":"2024-09-16T10:33:25.766197+0000","last_clean":"2024-09-16T10:33:25.766197+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:25.766197+0000","last_undegraded":"2024-09-16T10:33:25.766197+0000","last_fullsized":"2024-09-16T10:33:25.766197+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20288,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.771018+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:25.771018+0000","last_peered":"2024-09-16T10:33:25.771018+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:25.771018+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303068+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:31:29.303068+0000","last_peered":"2024-09-16T10:31:29.303068+0000","last_clean":"2024-09-16T10:31:29.303068+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:31:29.303068+0000","last_undegraded":"2024-09-16T10:31:29.303068+0000","last_fullsized":"2024-09-16T10:31:29.303068+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21194,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.672406+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:25.672406+0000","last_peered":"2024-09-16T10:33:25.672406+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:25.672406+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":64,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302601+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:31:29.302601+0000","last_peered":"2024-09-16T10:31:29.302601+0000","last_clean":"2024-09-16T10:31:29.302601+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:31:29.302601+0000","last_undegraded":"2024-09-16T10:31:29.302601+0000","last_fullsized":"2024-09-16T10:31:29.302601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":27997,"reported_epoch":48,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:34.279256+0000","last_change":"2024-09-16T10:31:27.294379+0000","last_active":"2024-09-16T10:33:34.279256+0000","last_peered":"2024-09-16T10:33:34.279256+0000","last_clean":"2024-09-16T10:23:54.331548+0000","last_became_active":"2024-09-16T10:23:56.894775+0000","last_became_peered":"2024-09-16T10:23:56.894775+0000","last_unstale":"2024-09-16T10:33:34.279256+0000","last_undegraded":"2024-09-16T10:23:56.433192+0000","last_fullsized":"2024-09-16T10:23:56.366848+0000","mapping_epoch":41,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3886,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":61,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.302890+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:31:29.302890+0000","last_peered":"2024-09-16T10:31:29.302890+0000","last_clean":"2024-09-16T10:31:29.302890+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:31:29.302890+0000","last_undegraded":"2024-09-16T10:31:29.302890+0000","last_fullsized":"2024-09-16T10:31:29.302890+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":70,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.303074+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:31:29.303074+0000","last_peered":"2024-09-16T10:31:29.303074+0000","last_clean":"2024-09-16T10:31:29.303074+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:31:29.303074+0000","last_undegraded":"2024-09-16T10:31:29.303074+0000","last_fullsized":"2024-09-16T10:31:29.303074+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21142,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.759598+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:25.759598+0000","last_peered":"2024-09-16T10:33:25.759598+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:25.759598+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17087,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210223104,"data_stored":210154457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":23,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019013,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480152,"kb_used_data":137616,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891688,"statfs":{"total":96636764160,"available":96145088512,"internally_reserved":0,"allocated":140918784,"data_stored":140644210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.187},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.20200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.217},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.255}]}]},{"osd":1,"up_from":40,"seq":171798691957,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451288,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93920552,"statfs":{"total":96636764160,"available":96174645248,"internally_reserved":0,"allocated":475136,"data_stored":142754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.16900000000000001},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.20699999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.222}]}]},{"osd":0,"up_from":24,"seq":103079215240,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523256,"kb_used_data":68400,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848584,"statfs":{"total":96636764160,"available":96100950016,"internally_reserved":0,"allocated":70041600,"data_stored":69770640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.223},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.20999999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":8211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:39.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:39.916 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:39.917 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:39.930 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-16T10:33:38.761362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17751,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1446536,"kb_used_data":206508,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281668984,"statfs":{"total":289910292480,"available":288429039616,"internally_reserved":0,"allocated":211464192,"data_stored":210568799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001507"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28936,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.747283+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:25.747283+0000","last_peered":"2024-09-16T10:33:25.747283+0000","last_clean":"2024-09-16T10:33:25.747283+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:25.747283+0000","last_undegraded":"2024-09-16T10:33:25.747283+0000","last_fullsized":"2024-09-16T10:33:25.747283+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":21680,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:37.683188+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:37.683188+0000","last_peered":"2024-09-16T10:33:37.683188+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:37.683188+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3926}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.770303+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:25.770303+0000","last_peered":"2024-09-16T10:33:25.770303+0000","last_clean":"2024-09-16T10:33:25.770303+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:25.770303+0000","last_undegraded":"2024-09-16T10:33:25.770303+0000","last_fullsized":"2024-09-16T10:33:25.770303+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20707,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.798394+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:25.798394+0000","last_peered":"2024-09-16T10:33:25.798394+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:25.798394+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21332,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.684300+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:25.684300+0000","last_peered":"2024-09-16T10:33:25.684300+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:25.684300+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20572,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.712059+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:25.712059+0000","last_peered":"2024-09-16T10:33:25.712059+0000","last_clean":"2024-09-16T10:33:25.712059+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:25.712059+0000","last_undegraded":"2024-09-16T10:33:25.712059+0000","last_fullsized":"2024-09-16T10:33:25.712059+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21320,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.797215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:25.797215+0000","last_peered":"2024-09-16T10:33:25.797215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:25.797215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20288,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.771018+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:25.771018+0000","last_peered":"2024-09-16T10:33:25.771018+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:25.771018+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21194,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.672406+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:25.672406+0000","last_peered":"2024-09-16T10:33:25.672406+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:25.672406+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21142,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.759598+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:25.759598+0000","last_peered":"2024-09-16T10:33:25.759598+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:25.759598+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17750,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019014,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.23300000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.221}]}]},{"osd":1,"up_from":40,"seq":171798691958,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443108,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93928732,"statfs":{"total":96636764160,"available":96183021568,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.187},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.223},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.252}]}]},{"osd":0,"up_from":24,"seq":103079215240,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523256,"kb_used_data":68400,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848584,"statfs":{"total":96636764160,"available":96100950016,"internally_reserved":0,"allocated":70041600,"data_stored":69770640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.223},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.20999999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:39.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:40.273 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:40.274 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:40.286 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-16T10:33:38.761362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17751,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1446536,"kb_used_data":206508,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281668984,"statfs":{"total":289910292480,"available":288429039616,"internally_reserved":0,"allocated":211464192,"data_stored":210568799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001507"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28936,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.747283+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:25.747283+0000","last_peered":"2024-09-16T10:33:25.747283+0000","last_clean":"2024-09-16T10:33:25.747283+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:25.747283+0000","last_undegraded":"2024-09-16T10:33:25.747283+0000","last_fullsized":"2024-09-16T10:33:25.747283+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":21680,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:37.683188+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:37.683188+0000","last_peered":"2024-09-16T10:33:37.683188+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:37.683188+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3926}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20669,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.770303+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:25.770303+0000","last_peered":"2024-09-16T10:33:25.770303+0000","last_clean":"2024-09-16T10:33:25.770303+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:25.770303+0000","last_undegraded":"2024-09-16T10:33:25.770303+0000","last_fullsized":"2024-09-16T10:33:25.770303+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20707,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.798394+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:25.798394+0000","last_peered":"2024-09-16T10:33:25.798394+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:25.798394+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21332,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.684300+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:25.684300+0000","last_peered":"2024-09-16T10:33:25.684300+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:25.684300+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20572,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:33:25.712059+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:25.712059+0000","last_peered":"2024-09-16T10:33:25.712059+0000","last_clean":"2024-09-16T10:33:25.712059+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:25.712059+0000","last_undegraded":"2024-09-16T10:33:25.712059+0000","last_fullsized":"2024-09-16T10:33:25.712059+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21320,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.797215+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:25.797215+0000","last_peered":"2024-09-16T10:33:25.797215+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:25.797215+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":95,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-16T10:31:29.309265+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:31:29.309265+0000","last_peered":"2024-09-16T10:31:29.309265+0000","last_clean":"2024-09-16T10:31:29.309265+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:31:29.309265+0000","last_undegraded":"2024-09-16T10:31:29.309265+0000","last_fullsized":"2024-09-16T10:31:29.309265+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20288,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.771018+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:25.771018+0000","last_peered":"2024-09-16T10:33:25.771018+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:25.771018+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21194,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.672406+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:25.672406+0000","last_peered":"2024-09-16T10:33:25.672406+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:25.672406+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21142,"reported_epoch":48,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:25.759598+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:25.759598+0000","last_peered":"2024-09-16T10:33:25.759598+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:25.759598+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17750,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019014,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.23300000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.221}]}]},{"osd":1,"up_from":40,"seq":171798691958,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443108,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93928732,"statfs":{"total":96636764160,"available":96183021568,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.187},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.223},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.252}]}]},{"osd":0,"up_from":24,"seq":103079215240,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523256,"kb_used_data":68400,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848584,"statfs":{"total":96636764160,"available":96100950016,"internally_reserved":0,"allocated":70041600,"data_stored":69770640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.223},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.20999999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.255}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:40.288 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:40.288 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20664", 'reported_seq': 21680, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:37.683188+0000', 'last_change': '2024-09-16T10:33:35.213051+0000', 'last_active': '2024-09-16T10:33:37.683188+0000', 'last_peered': '2024-09-16T10:33:37.683188+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:37.683188+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10600", 'ondisk_log_start': "44'10600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3926, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16721, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 410, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3926}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:40.288 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:40.288 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 20707, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.798394+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:25.798394+0000', 'last_peered': '2024-09-16T10:33:25.798394+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:25.798394+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:40.288 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:40.288 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21332, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.684300+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:25.684300+0000', 'last_peered': '2024-09-16T10:33:25.684300+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:25.684300+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:40.289 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:40.289 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21320, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.797215+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:25.797215+0000', 'last_peered': '2024-09-16T10:33:25.797215+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:25.797215+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:40.289 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:40.289 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20622, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.897671+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:35.897671+0000', 'last_peered': '2024-09-16T10:33:35.897671+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:35.897671+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:40.289 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:40.289 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20288, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.771018+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:25.771018+0000', 'last_peered': '2024-09-16T10:33:25.771018+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:25.771018+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:40.289 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:40.289 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21194, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.672406+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:25.672406+0000', 'last_peered': '2024-09-16T10:33:25.672406+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:25.672406+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:40.289 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:40.290 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21142, 'reported_epoch': 48, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:25.759598+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:25.759598+0000', 'last_peered': '2024-09-16T10:33:25.759598+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:25.759598+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:40.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:40.735 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:40.736 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33596/40 objects degraded (83990.000%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":709,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":50,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1481252864,"bytes_avail":288429039616,"bytes_total":289910292480,"degraded_objects":33596,"degraded_total":40,"degraded_ratio":839.89999999999998,"recovering_objects_per_sec":228,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1831,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:40.751 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:40.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:41.049 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:41.049 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:41.061 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-16T10:33:40.761730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17751,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1446556,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281668964,"statfs":{"total":289910292480,"available":288429019136,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1831,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001437"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28938,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.901377+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:35.901377+0000","last_peered":"2024-09-16T10:33:35.901377+0000","last_clean":"2024-09-16T10:33:35.901377+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:35.901377+0000","last_undegraded":"2024-09-16T10:33:35.901377+0000","last_fullsized":"2024-09-16T10:33:35.901377+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":21680,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:37.683188+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:37.683188+0000","last_peered":"2024-09-16T10:33:37.683188+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:37.683188+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3926,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":410,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3926}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20709,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900456+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:35.900456+0000","last_peered":"2024-09-16T10:33:35.900456+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:35.900456+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21334,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900002+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:35.900002+0000","last_peered":"2024-09-16T10:33:35.900002+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:35.900002+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20290,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900506+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:35.900506+0000","last_peered":"2024-09-16T10:33:35.900506+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:35.900506+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17750,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019014,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.23300000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.221}]}]},{"osd":1,"up_from":40,"seq":171798691958,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443108,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93928732,"statfs":{"total":96636764160,"available":96183021568,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.187},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.223},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.252}]}]},{"osd":0,"up_from":24,"seq":103079215241,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.183},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.26600000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.222}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:44.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:44.354 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:44.355 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:44.367 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-16T10:33:42.762175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18541,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1446556,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281668964,"statfs":{"total":289910292480,"available":288429019136,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001543"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28938,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.901377+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:35.901377+0000","last_peered":"2024-09-16T10:33:35.901377+0000","last_clean":"2024-09-16T10:33:35.901377+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:35.901377+0000","last_undegraded":"2024-09-16T10:33:35.901377+0000","last_fullsized":"2024-09-16T10:33:35.901377+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":23260,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:42.686043+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:42.686043+0000","last_peered":"2024-09-16T10:33:42.686043+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:42.686043+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3136}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20709,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900456+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:35.900456+0000","last_peered":"2024-09-16T10:33:35.900456+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:35.900456+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21334,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900002+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:35.900002+0000","last_peered":"2024-09-16T10:33:35.900002+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:35.900002+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20290,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900506+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:35.900506+0000","last_peered":"2024-09-16T10:33:35.900506+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:35.900506+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18540,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019015,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.34200000000000003},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.377},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.36799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691958,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443108,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93928732,"statfs":{"total":96636764160,"available":96183021568,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.187},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.223},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.252}]}]},{"osd":0,"up_from":24,"seq":103079215241,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.183},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.26600000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.222}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:44.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:44.671 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:44.671 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:44.683 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-16T10:33:42.762175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18541,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1446556,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1239987,"kb_avail":281668964,"statfs":{"total":289910292480,"available":288429019136,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1269746690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001543"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28938,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.901377+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:35.901377+0000","last_peered":"2024-09-16T10:33:35.901377+0000","last_clean":"2024-09-16T10:33:35.901377+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:35.901377+0000","last_undegraded":"2024-09-16T10:33:35.901377+0000","last_fullsized":"2024-09-16T10:33:35.901377+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":23260,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:42.686043+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:42.686043+0000","last_peered":"2024-09-16T10:33:42.686043+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:42.686043+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3136}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20709,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900456+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:35.900456+0000","last_peered":"2024-09-16T10:33:35.900456+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:35.900456+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21334,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900002+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:35.900002+0000","last_peered":"2024-09-16T10:33:35.900002+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:35.900002+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20290,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900506+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:35.900506+0000","last_peered":"2024-09-16T10:33:35.900506+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:35.900506+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18540,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019015,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.34200000000000003},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.377},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.36799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691958,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":443108,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93928732,"statfs":{"total":96636764160,"available":96183021568,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":453242538},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.187},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.223},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.252}]}]},{"osd":0,"up_from":24,"seq":103079215241,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.183},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.26600000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.222}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:44.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:44.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20664", 'reported_seq': 23260, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:42.686043+0000', 'last_change': '2024-09-16T10:33:35.213051+0000', 'last_active': '2024-09-16T10:33:42.686043+0000', 'last_peered': '2024-09-16T10:33:42.686043+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:42.686043+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10600", 'ondisk_log_start': "44'10600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16721, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1200, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3136}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:44.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:44.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 20709, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.900456+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:35.900456+0000', 'last_peered': '2024-09-16T10:33:35.900456+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:35.900456+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:44.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:44.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21334, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.900002+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:35.900002+0000', 'last_peered': '2024-09-16T10:33:35.900002+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:35.900002+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:44.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:44.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21322, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.901594+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:35.901594+0000', 'last_peered': '2024-09-16T10:33:35.901594+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:35.901594+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:44.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:44.685 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20622, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.897671+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:35.897671+0000', 'last_peered': '2024-09-16T10:33:35.897671+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:35.897671+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:44.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:44.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20290, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.900506+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:35.900506+0000', 'last_peered': '2024-09-16T10:33:35.900506+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:35.900506+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:44.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:44.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21196, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.902142+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:35.902142+0000', 'last_peered': '2024-09-16T10:33:35.902142+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:35.902142+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:44.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:44.686 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21144, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.900441+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:35.900441+0000', 'last_peered': '2024-09-16T10:33:35.900441+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:35.900441+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:44.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:45.054 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:45.054 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32806/40 objects degraded (82015.000%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":713,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":50,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1489661952,"bytes_avail":288420630528,"bytes_total":289910292480,"degraded_objects":32806,"degraded_total":40,"degraded_ratio":820.14999999999998,"recovering_objects_per_sec":163,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1453,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:45.066 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:45.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:45.414 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:45.415 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:45.427 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-09-16T10:33:44.762517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18541,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1454748,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1248179,"kb_avail":281660772,"statfs":{"total":289910292480,"available":288420630528,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1278135298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.866332"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28938,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.901377+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:35.901377+0000","last_peered":"2024-09-16T10:33:35.901377+0000","last_clean":"2024-09-16T10:33:35.901377+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:35.901377+0000","last_undegraded":"2024-09-16T10:33:35.901377+0000","last_fullsized":"2024-09-16T10:33:35.901377+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":23260,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:42.686043+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:42.686043+0000","last_peered":"2024-09-16T10:33:42.686043+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:42.686043+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3136}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18540,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019015,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.34200000000000003},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.377},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.36799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691959,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451300,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93920540,"statfs":{"total":96636764160,"available":96174632960,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.31}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28999999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215242,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.28199999999999997},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:48.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:48.729 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:48.729 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:48.741 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-16T10:33:46.762826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18541,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1454748,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1248179,"kb_avail":281660772,"statfs":{"total":289910292480,"available":288420630528,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1278135298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001833"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28938,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.901377+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:35.901377+0000","last_peered":"2024-09-16T10:33:35.901377+0000","last_clean":"2024-09-16T10:33:35.901377+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:35.901377+0000","last_undegraded":"2024-09-16T10:33:35.901377+0000","last_fullsized":"2024-09-16T10:33:35.901377+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":23260,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:42.686043+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:42.686043+0000","last_peered":"2024-09-16T10:33:42.686043+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:42.686043+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1200,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3136}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32806,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18540,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019015,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.34200000000000003},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.35399999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.377},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.36799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691959,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":451300,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":450811,"kb_avail":93920540,"statfs":{"total":96636764160,"available":96174632960,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":461631146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.31}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28999999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215242,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.28199999999999997},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:48.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:49.088 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:49.088 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:49.101 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-09-16T10:33:48.763140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19182,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1458844,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1252275,"kb_avail":281656676,"statfs":{"total":289910292480,"available":288416436224,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1282329602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.863013"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28938,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.901377+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:35.901377+0000","last_peered":"2024-09-16T10:33:35.901377+0000","last_clean":"2024-09-16T10:33:35.901377+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:35.901377+0000","last_undegraded":"2024-09-16T10:33:35.901377+0000","last_fullsized":"2024-09-16T10:33:35.901377+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":24542,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:47.688370+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:47.688370+0000","last_peered":"2024-09-16T10:33:47.688370+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:47.688370+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1841,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2495}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19181,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019016,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.245},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.23200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.217},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.187}]}]},{"osd":1,"up_from":40,"seq":171798691960,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455396,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93916444,"statfs":{"total":96636764160,"available":96170438656,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.252},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28199999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215242,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.28199999999999997},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:49.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:49.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20664", 'reported_seq': 24542, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:47.688370+0000', 'last_change': '2024-09-16T10:33:35.213051+0000', 'last_active': '2024-09-16T10:33:47.688370+0000', 'last_peered': '2024-09-16T10:33:47.688370+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:47.688370+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10600", 'ondisk_log_start': "44'10600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2495, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16721, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1841, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -2495}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:49.102 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 20710, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.904813+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:36.904813+0000', 'last_peered': '2024-09-16T10:33:36.904813+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:36.904813+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21335, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.905582+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:36.905582+0000', 'last_peered': '2024-09-16T10:33:36.905582+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:36.905582+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21322, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.901594+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:35.901594+0000', 'last_peered': '2024-09-16T10:33:35.901594+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:35.901594+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20622, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.897671+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:35.897671+0000', 'last_peered': '2024-09-16T10:33:35.897671+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:35.897671+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20291, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.907426+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:36.907426+0000', 'last_peered': '2024-09-16T10:33:36.907426+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:36.907426+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:49.103 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21196, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.902142+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:35.902142+0000', 'last_peered': '2024-09-16T10:33:35.902142+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:35.902142+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:49.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:49.104 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21144, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.900441+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:35.900441+0000', 'last_peered': '2024-09-16T10:33:35.900441+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:35.900441+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:49.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:49.475 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:49.475 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32165/40 objects degraded (80412.500%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":718,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":50,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1493856256,"bytes_avail":288416436224,"bytes_total":289910292480,"degraded_objects":32165,"degraded_total":40,"degraded_ratio":804.125,"recovering_objects_per_sec":176,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2094,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:49.488 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:49.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:49.784 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:49.785 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:49.798 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-09-16T10:33:48.763140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19182,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1458844,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1252275,"kb_avail":281656676,"statfs":{"total":289910292480,"available":288416436224,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1282329602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2094,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.863013"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28938,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.901377+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:35.901377+0000","last_peered":"2024-09-16T10:33:35.901377+0000","last_clean":"2024-09-16T10:33:35.901377+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:35.901377+0000","last_undegraded":"2024-09-16T10:33:35.901377+0000","last_fullsized":"2024-09-16T10:33:35.901377+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":24542,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:47.688370+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:47.688370+0000","last_peered":"2024-09-16T10:33:47.688370+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:47.688370+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1841,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-2495}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19181,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019016,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.245},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.23200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.217},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.187}]}]},{"osd":1,"up_from":40,"seq":171798691960,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455396,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93916444,"statfs":{"total":96636764160,"available":96170438656,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.252},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28199999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215242,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.28199999999999997},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:52.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:53.103 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:53.103 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:53.116 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-16T10:33:52.764004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19957,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1458844,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1252275,"kb_avail":281656676,"statfs":{"total":289910292480,"available":288416436224,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1282329602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002274"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28938,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.901377+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:35.901377+0000","last_peered":"2024-09-16T10:33:35.901377+0000","last_clean":"2024-09-16T10:33:35.901377+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:35.901377+0000","last_undegraded":"2024-09-16T10:33:35.901377+0000","last_fullsized":"2024-09-16T10:33:35.901377+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":26092,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:52.687783+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:52.687783+0000","last_peered":"2024-09-16T10:33:52.687783+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:52.687783+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2616,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1720}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19956,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019017,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.20100000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.22600000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.17599999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691960,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455396,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93916444,"statfs":{"total":96636764160,"available":96170438656,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.252},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28199999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215243,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.191},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20699999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.22900000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:53.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:53.411 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:53.411 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:53.424 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-16T10:33:52.764004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19957,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1458844,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1252275,"kb_avail":281656676,"statfs":{"total":289910292480,"available":288416436224,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1282329602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002274"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28938,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.901377+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:35.901377+0000","last_peered":"2024-09-16T10:33:35.901377+0000","last_clean":"2024-09-16T10:33:35.901377+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:35.901377+0000","last_undegraded":"2024-09-16T10:33:35.901377+0000","last_fullsized":"2024-09-16T10:33:35.901377+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":26092,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:52.687783+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:52.687783+0000","last_peered":"2024-09-16T10:33:52.687783+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:52.687783+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2616,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1720}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19956,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019017,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.20100000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.22600000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.17599999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691960,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455396,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93916444,"statfs":{"total":96636764160,"available":96170438656,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.252},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28199999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215243,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.191},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20699999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.22900000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:53.425 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:53.426 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20664", 'reported_seq': 26092, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:52.687783+0000', 'last_change': '2024-09-16T10:33:35.213051+0000', 'last_active': '2024-09-16T10:33:52.687783+0000', 'last_peered': '2024-09-16T10:33:52.687783+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:52.687783+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10600", 'ondisk_log_start': "44'10600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1720, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16721, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2616, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1720}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:53.426 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:53.426 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 20710, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.904813+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:36.904813+0000', 'last_peered': '2024-09-16T10:33:36.904813+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:36.904813+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:53.426 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:53.426 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21335, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.905582+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:36.905582+0000', 'last_peered': '2024-09-16T10:33:36.905582+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:36.905582+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:53.426 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:53.426 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21322, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.901594+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:35.901594+0000', 'last_peered': '2024-09-16T10:33:35.901594+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:35.901594+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:53.426 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:53.426 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20622, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.897671+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:35.897671+0000', 'last_peered': '2024-09-16T10:33:35.897671+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:35.897671+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:53.427 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:53.427 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20291, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.907426+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:36.907426+0000', 'last_peered': '2024-09-16T10:33:36.907426+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:36.907426+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:53.427 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:53.427 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21196, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.902142+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:35.902142+0000', 'last_peered': '2024-09-16T10:33:35.902142+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:35.902142+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:53.427 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:53.427 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21144, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.900441+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:35.900441+0000', 'last_peered': '2024-09-16T10:33:35.900441+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:35.900441+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:53.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:53.826 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:53.826 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31390/40 objects degraded (78475.000%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":722,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":50,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1493856256,"bytes_avail":288416436224,"bytes_total":289910292480,"degraded_objects":31390,"degraded_total":40,"degraded_ratio":784.75,"recovering_objects_per_sec":183,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2206,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:53.839 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:53.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:54.133 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:54.133 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:54.145 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-16T10:33:52.764004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19957,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1458844,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1252275,"kb_avail":281656676,"statfs":{"total":289910292480,"available":288416436224,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1282329602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2206,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002274"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28938,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.901377+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:35.901377+0000","last_peered":"2024-09-16T10:33:35.901377+0000","last_clean":"2024-09-16T10:33:35.901377+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:35.901377+0000","last_undegraded":"2024-09-16T10:33:35.901377+0000","last_fullsized":"2024-09-16T10:33:35.901377+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":26092,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:52.687783+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:52.687783+0000","last_peered":"2024-09-16T10:33:52.687783+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:52.687783+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2616,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1720}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19956,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019017,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.20100000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.22600000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.17599999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691960,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455396,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":454907,"kb_avail":93916444,"statfs":{"total":96636764160,"available":96170438656,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":465825450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.252},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.21299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.22800000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28199999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215243,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.191},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20699999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.22900000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:57.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:57.442 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:57.443 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:57.455 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-16T10:33:56.764724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19957,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464604,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650916,"statfs":{"total":289910292480,"available":288410537984,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002207"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28939,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.907177+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:36.907177+0000","last_peered":"2024-09-16T10:33:36.907177+0000","last_clean":"2024-09-16T10:33:36.907177+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:36.907177+0000","last_undegraded":"2024-09-16T10:33:36.907177+0000","last_fullsized":"2024-09-16T10:33:36.907177+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":26092,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:52.687783+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:52.687783+0000","last_peered":"2024-09-16T10:33:52.687783+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:52.687783+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2616,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1720}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19956,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019017,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.20100000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.22600000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.17599999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691961,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461156,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910684,"statfs":{"total":96636764160,"available":96164540416,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23999999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.19700000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215244,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.186},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20699999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23599999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.22500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:57.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:57.769 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:57.769 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:57.782 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-16T10:33:56.764724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19957,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464604,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650916,"statfs":{"total":289910292480,"available":288410537984,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002207"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28939,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.907177+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:36.907177+0000","last_peered":"2024-09-16T10:33:36.907177+0000","last_clean":"2024-09-16T10:33:36.907177+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:36.907177+0000","last_undegraded":"2024-09-16T10:33:36.907177+0000","last_fullsized":"2024-09-16T10:33:36.907177+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":26092,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:52.687783+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:52.687783+0000","last_peered":"2024-09-16T10:33:52.687783+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:52.687783+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2616,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1720}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19956,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019017,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.20100000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.22600000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.17599999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691961,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461156,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910684,"statfs":{"total":96636764160,"available":96164540416,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23999999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.19700000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215244,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.186},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20699999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23599999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.22500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:33:57.784 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:33:57.784 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20664", 'reported_seq': 26092, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:52.687783+0000', 'last_change': '2024-09-16T10:33:35.213051+0000', 'last_active': '2024-09-16T10:33:52.687783+0000', 'last_peered': '2024-09-16T10:33:52.687783+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:52.687783+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10600", 'ondisk_log_start': "44'10600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1720, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16721, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2616, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1720}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:57.784 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:33:57.784 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 20710, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.904813+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:36.904813+0000', 'last_peered': '2024-09-16T10:33:36.904813+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:36.904813+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:57.784 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:33:57.784 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21335, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.905582+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:36.905582+0000', 'last_peered': '2024-09-16T10:33:36.905582+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:36.905582+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:57.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:33:57.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21322, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.901594+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:35.901594+0000', 'last_peered': '2024-09-16T10:33:35.901594+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:35.901594+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:57.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:33:57.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20622, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.897671+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:35.897671+0000', 'last_peered': '2024-09-16T10:33:35.897671+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:35.897671+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:33:57.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:33:57.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20291, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.907426+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:36.907426+0000', 'last_peered': '2024-09-16T10:33:36.907426+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:36.907426+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:57.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:33:57.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21196, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.902142+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:35.902142+0000', 'last_peered': '2024-09-16T10:33:35.902142+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:35.902142+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:57.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:33:57.785 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21144, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.900441+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:35.900441+0000', 'last_peered': '2024-09-16T10:33:35.900441+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:35.900441+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:33:57.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:33:58.189 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:58.190 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31390/40 objects degraded (78475.000%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":726,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":50,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1499754496,"bytes_avail":288410537984,"bytes_total":289910292480,"degraded_objects":31390,"degraded_total":40,"degraded_ratio":784.75,"recovering_objects_per_sec":117,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1416,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:33:58.202 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:33:58.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:33:58.522 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:33:58.522 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:33:58.535 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-16T10:33:56.764724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19957,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464604,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650916,"statfs":{"total":289910292480,"available":288410537984,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002207"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28939,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.907177+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:36.907177+0000","last_peered":"2024-09-16T10:33:36.907177+0000","last_clean":"2024-09-16T10:33:36.907177+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:36.907177+0000","last_undegraded":"2024-09-16T10:33:36.907177+0000","last_fullsized":"2024-09-16T10:33:36.907177+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":26092,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:52.687783+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:52.687783+0000","last_peered":"2024-09-16T10:33:52.687783+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:52.687783+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2616,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1720}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898528+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:35.898528+0000","last_peered":"2024-09-16T10:33:35.898528+0000","last_clean":"2024-09-16T10:33:35.898528+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:35.898528+0000","last_undegraded":"2024-09-16T10:33:35.898528+0000","last_fullsized":"2024-09-16T10:33:35.898528+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31390,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19956,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019017,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.20100000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.22600000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.17599999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691961,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461156,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910684,"statfs":{"total":96636764160,"available":96164540416,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23999999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.19700000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215244,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.186},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20699999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23599999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.22500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:01.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:01.862 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:01.862 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:01.876 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-16T10:34:01.104984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20950,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472796,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281642724,"statfs":{"total":289910292480,"available":288402149376,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1768,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.341400"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28939,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.907177+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:36.907177+0000","last_peered":"2024-09-16T10:33:36.907177+0000","last_clean":"2024-09-16T10:33:36.907177+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:36.907177+0000","last_undegraded":"2024-09-16T10:33:36.907177+0000","last_fullsized":"2024-09-16T10:33:36.907177+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":28078,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:57.690946+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:57.690946+0000","last_peered":"2024-09-16T10:33:57.690946+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:57.690946+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3609,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-727}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.902158+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:36.902158+0000","last_peered":"2024-09-16T10:33:36.902158+0000","last_clean":"2024-09-16T10:33:36.902158+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:36.902158+0000","last_undegraded":"2024-09-16T10:33:36.902158+0000","last_fullsized":"2024-09-16T10:33:36.902158+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20949,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019018,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.20699999999999999},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.219}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.189},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.17199999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691962,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469348,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902492,"statfs":{"total":96636764160,"available":96156151808,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.19},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.248}]}]},{"osd":0,"up_from":24,"seq":103079215245,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17799999999999999},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.22},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:01.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:02.208 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:02.209 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:02.225 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-16T10:34:02.106019+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20950,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472796,"kb_used_data":206524,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281642724,"statfs":{"total":289910292480,"available":288402149376,"internally_reserved":0,"allocated":211480576,"data_stored":210574638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.342014"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28939,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.907177+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:36.907177+0000","last_peered":"2024-09-16T10:33:36.907177+0000","last_clean":"2024-09-16T10:33:36.907177+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:36.907177+0000","last_undegraded":"2024-09-16T10:33:36.907177+0000","last_fullsized":"2024-09-16T10:33:36.907177+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20628,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898189+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:33:35.898189+0000","last_peered":"2024-09-16T10:33:35.898189+0000","last_clean":"2024-09-16T10:33:35.898189+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:33:35.898189+0000","last_undegraded":"2024-09-16T10:33:35.898189+0000","last_fullsized":"2024-09-16T10:33:35.898189+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":28078,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:57.690946+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:57.690946+0000","last_peered":"2024-09-16T10:33:57.690946+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:57.690946+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3609,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-727}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20527,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897967+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:33:35.897967+0000","last_peered":"2024-09-16T10:33:35.897967+0000","last_clean":"2024-09-16T10:33:35.897967+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:33:35.897967+0000","last_undegraded":"2024-09-16T10:33:35.897967+0000","last_fullsized":"2024-09-16T10:33:35.897967+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.902158+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:36.902158+0000","last_peered":"2024-09-16T10:33:36.902158+0000","last_clean":"2024-09-16T10:33:36.902158+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:36.902158+0000","last_undegraded":"2024-09-16T10:33:36.902158+0000","last_fullsized":"2024-09-16T10:33:36.902158+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897253+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:33:35.897253+0000","last_peered":"2024-09-16T10:33:35.897253+0000","last_clean":"2024-09-16T10:33:35.897253+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:33:35.897253+0000","last_undegraded":"2024-09-16T10:33:35.897253+0000","last_fullsized":"2024-09-16T10:33:35.897253+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20622,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.897671+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:33:35.897671+0000","last_peered":"2024-09-16T10:33:35.897671+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:33:35.897671+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29067,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.897333+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:33:35.897333+0000","last_peered":"2024-09-16T10:33:35.897333+0000","last_clean":"2024-09-16T10:33:35.897333+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:33:35.897333+0000","last_undegraded":"2024-09-16T10:33:35.897333+0000","last_fullsized":"2024-09-16T10:33:35.897333+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899734+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:33:35.899734+0000","last_peered":"2024-09-16T10:33:35.899734+0000","last_clean":"2024-09-16T10:33:35.899734+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:33:35.899734+0000","last_undegraded":"2024-09-16T10:33:35.899734+0000","last_fullsized":"2024-09-16T10:33:35.899734+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20949,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019018,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480172,"kb_used_data":137632,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891668,"statfs":{"total":96636764160,"available":96145068032,"internally_reserved":0,"allocated":140935168,"data_stored":140650049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.20699999999999999},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.219}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.189},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.17199999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691962,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469348,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902492,"statfs":{"total":96636764160,"available":96156151808,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.19},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.248}]}]},{"osd":0,"up_from":24,"seq":103079215245,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17799999999999999},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.22},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:02.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.d is not active+clean 2024-09-16T10:34:02.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.d', 'version': "48'20664", 'reported_seq': 28078, 'reported_epoch': 50, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:57.690946+0000', 'last_change': '2024-09-16T10:33:35.213051+0000', 'last_active': '2024-09-16T10:33:57.690946+0000', 'last_peered': '2024-09-16T10:33:57.690946+0000', 'last_clean': '2024-09-16T10:23:54.392083+0000', 'last_became_active': '2024-09-16T10:23:56.874397+0000', 'last_became_peered': '2024-09-16T10:23:56.874397+0000', 'last_unstale': '2024-09-16T10:33:57.690946+0000', 'last_undegraded': '2024-09-16T10:23:56.410408+0000', 'last_fullsized': '2024-09-16T10:23:56.366400+0000', 'mapping_epoch': 41, 'log_start': "44'10600", 'ondisk_log_start': "44'10600", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'4014", 'last_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:47.680381+0000', 'objects_scrubbed': 3939, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000', 'scrub_duration': 1.621791945, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 727, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 16721, 'num_write_kb': 25561, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3609, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -727}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:02.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:34:02.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 20710, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.904813+0000', 'last_change': '2024-09-16T10:23:56.914961+0000', 'last_active': '2024-09-16T10:33:36.904813+0000', 'last_peered': '2024-09-16T10:33:36.904813+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:33:36.904813+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4136, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 83, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:02.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:02.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21335, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.905582+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:33:36.905582+0000', 'last_peered': '2024-09-16T10:33:36.905582+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:33:36.905582+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:02.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:02.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21322, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.901594+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:33:35.901594+0000', 'last_peered': '2024-09-16T10:33:35.901594+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:33:35.901594+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:02.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:02.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20622, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.897671+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:33:35.897671+0000', 'last_peered': '2024-09-16T10:33:35.897671+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:33:35.897671+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:02.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:02.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20291, 'reported_epoch': 50, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:36.907426+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:33:36.907426+0000', 'last_peered': '2024-09-16T10:33:36.907426+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:33:36.907426+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:02.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:02.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21196, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.902142+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:33:35.902142+0000', 'last_peered': '2024-09-16T10:33:35.902142+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:33:35.902142+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:02.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:02.228 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21144, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:33:35.900441+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:33:35.900441+0000', 'last_peered': '2024-09-16T10:33:35.900441+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:33:35.900441+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:02.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:02.621 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:02.622 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30397/40 objects degraded (75992.500%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":731,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":52,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":16},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1508143104,"bytes_avail":288402149376,"bytes_total":289910292480,"degraded_objects":30397,"degraded_total":40,"degraded_ratio":759.92499999999995,"recovering_objects_per_sec":147,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1768,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:02.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:02.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:02.991 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:02.992 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:03.009 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-09-16T10:34:02.765883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20950,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":40,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472816,"kb_used_data":206540,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281642704,"statfs":{"total":289910292480,"available":288402128896,"internally_reserved":0,"allocated":211496960,"data_stored":210580427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001506"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28939,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.907177+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:33:36.907177+0000","last_peered":"2024-09-16T10:33:36.907177+0000","last_clean":"2024-09-16T10:33:36.907177+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:33:36.907177+0000","last_undegraded":"2024-09-16T10:33:36.907177+0000","last_fullsized":"2024-09-16T10:33:36.907177+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20630,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107014+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:01.107014+0000","last_peered":"2024-09-16T10:34:01.107014+0000","last_clean":"2024-09-16T10:34:01.107014+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:01.107014+0000","last_undegraded":"2024-09-16T10:34:01.107014+0000","last_fullsized":"2024-09-16T10:34:01.107014+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":28078,"reported_epoch":50,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:57.690946+0000","last_change":"2024-09-16T10:33:35.213051+0000","last_active":"2024-09-16T10:33:57.690946+0000","last_peered":"2024-09-16T10:33:57.690946+0000","last_clean":"2024-09-16T10:23:54.392083+0000","last_became_active":"2024-09-16T10:23:56.874397+0000","last_became_peered":"2024-09-16T10:23:56.874397+0000","last_unstale":"2024-09-16T10:33:57.690946+0000","last_undegraded":"2024-09-16T10:23:56.410408+0000","last_fullsized":"2024-09-16T10:23:56.366400+0000","mapping_epoch":41,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:59:03.345967+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3609,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-727}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20529,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106457+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:01.106457+0000","last_peered":"2024-09-16T10:34:01.106457+0000","last_clean":"2024-09-16T10:34:01.106457+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:01.106457+0000","last_undegraded":"2024-09-16T10:34:01.106457+0000","last_fullsized":"2024-09-16T10:34:01.106457+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20671,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899839+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:33:35.899839+0000","last_peered":"2024-09-16T10:33:35.899839+0000","last_clean":"2024-09-16T10:33:35.899839+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:33:35.899839+0000","last_undegraded":"2024-09-16T10:33:35.899839+0000","last_fullsized":"2024-09-16T10:33:35.899839+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20710,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.904813+0000","last_change":"2024-09-16T10:23:56.914961+0000","last_active":"2024-09-16T10:33:36.904813+0000","last_peered":"2024-09-16T10:33:36.904813+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:33:36.904813+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4134}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21335,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.905582+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:33:36.905582+0000","last_peered":"2024-09-16T10:33:36.905582+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:33:36.905582+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20574,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899476+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:33:35.899476+0000","last_peered":"2024-09-16T10:33:35.899476+0000","last_clean":"2024-09-16T10:33:35.899476+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:33:35.899476+0000","last_undegraded":"2024-09-16T10:33:35.899476+0000","last_fullsized":"2024-09-16T10:33:35.899476+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.902158+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:33:36.902158+0000","last_peered":"2024-09-16T10:33:36.902158+0000","last_clean":"2024-09-16T10:33:36.902158+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:33:36.902158+0000","last_undegraded":"2024-09-16T10:33:36.902158+0000","last_fullsized":"2024-09-16T10:33:36.902158+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28973,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898972+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:33:35.898972+0000","last_peered":"2024-09-16T10:33:35.898972+0000","last_clean":"2024-09-16T10:33:35.898972+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:33:35.898972+0000","last_undegraded":"2024-09-16T10:33:35.898972+0000","last_fullsized":"2024-09-16T10:33:35.898972+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.105943+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:01.105943+0000","last_peered":"2024-09-16T10:34:01.105943+0000","last_clean":"2024-09-16T10:34:01.105943+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:01.105943+0000","last_undegraded":"2024-09-16T10:34:01.105943+0000","last_fullsized":"2024-09-16T10:34:01.105943+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21322,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.901594+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:33:35.901594+0000","last_peered":"2024-09-16T10:33:35.901594+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:33:35.901594+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899635+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:33:35.899635+0000","last_peered":"2024-09-16T10:33:35.899635+0000","last_clean":"2024-09-16T10:33:35.899635+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:33:35.899635+0000","last_undegraded":"2024-09-16T10:33:35.899635+0000","last_fullsized":"2024-09-16T10:33:35.899635+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20624,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.106309+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:34:01.106309+0000","last_peered":"2024-09-16T10:34:01.106309+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:01.106309+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":77,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.898887+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:33:35.898887+0000","last_peered":"2024-09-16T10:33:35.898887+0000","last_clean":"2024-09-16T10:33:35.898887+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:33:35.898887+0000","last_undegraded":"2024-09-16T10:33:35.898887+0000","last_fullsized":"2024-09-16T10:33:35.898887+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29069,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106133+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:01.106133+0000","last_peered":"2024-09-16T10:34:01.106133+0000","last_clean":"2024-09-16T10:34:01.106133+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:01.106133+0000","last_undegraded":"2024-09-16T10:34:01.106133+0000","last_fullsized":"2024-09-16T10:34:01.106133+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20291,"reported_epoch":50,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:36.907426+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:33:36.907426+0000","last_peered":"2024-09-16T10:33:36.907426+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:33:36.907426+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899295+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:33:35.899295+0000","last_peered":"2024-09-16T10:33:35.899295+0000","last_clean":"2024-09-16T10:33:35.899295+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:33:35.899295+0000","last_undegraded":"2024-09-16T10:33:35.899295+0000","last_fullsized":"2024-09-16T10:33:35.899295+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21196,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.902142+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:33:35.902142+0000","last_peered":"2024-09-16T10:33:35.902142+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:33:35.902142+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106625+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:01.106625+0000","last_peered":"2024-09-16T10:34:01.106625+0000","last_clean":"2024-09-16T10:34:01.106625+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:01.106625+0000","last_undegraded":"2024-09-16T10:34:01.106625+0000","last_fullsized":"2024-09-16T10:34:01.106625+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24505,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T10:33:36.981835+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:33:36.981835+0000","last_peered":"2024-09-16T10:33:36.981835+0000","last_clean":"2024-09-16T10:33:36.981835+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:33:36.981835+0000","last_undegraded":"2024-09-16T10:33:36.981835+0000","last_fullsized":"2024-09-16T10:33:36.981835+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":63,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899318+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:33:35.899318+0000","last_peered":"2024-09-16T10:33:35.899318+0000","last_clean":"2024-09-16T10:33:35.899318+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:33:35.899318+0000","last_undegraded":"2024-09-16T10:33:35.899318+0000","last_fullsized":"2024-09-16T10:33:35.899318+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":72,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T10:33:35.899218+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:33:35.899218+0000","last_peered":"2024-09-16T10:33:35.899218+0000","last_clean":"2024-09-16T10:33:35.899218+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:33:35.899218+0000","last_undegraded":"2024-09-16T10:33:35.899218+0000","last_fullsized":"2024-09-16T10:33:35.899218+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21144,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:33:35.900441+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:33:35.900441+0000","last_peered":"2024-09-16T10:33:35.900441+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:33:35.900441+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20949,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210219008,"data_stored":210153974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019019,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480192,"kb_used_data":137648,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891648,"statfs":{"total":96636764160,"available":96145047552,"internally_reserved":0,"allocated":140951552,"data_stored":140655838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.31900000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.26300000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.27600000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691962,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469348,"kb_used_data":476,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902492,"statfs":{"total":96636764160,"available":96156151808,"internally_reserved":0,"allocated":487424,"data_stored":148110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.19},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.248}]}]},{"osd":0,"up_from":24,"seq":103079215245,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523276,"kb_used_data":68416,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848564,"statfs":{"total":96636764160,"available":96100929536,"internally_reserved":0,"allocated":70057984,"data_stored":69776479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17799999999999999},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.22},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:06.011 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:06.337 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:06.353 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-16T10:34:04.766242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22351,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464660,"kb_used_data":206564,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650860,"statfs":{"total":289910292480,"available":288410480640,"internally_reserved":0,"allocated":211521536,"data_stored":210591039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2394,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001517"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20630,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107014+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:01.107014+0000","last_peered":"2024-09-16T10:34:01.107014+0000","last_clean":"2024-09-16T10:34:01.107014+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:01.107014+0000","last_undegraded":"2024-09-16T10:34:01.107014+0000","last_fullsized":"2024-09-16T10:34:01.107014+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20933,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.253954+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:02.253954+0000","last_peered":"2024-09-16T10:34:02.253954+0000","last_clean":"2024-09-16T10:34:02.253954+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:02.253954+0000","last_undegraded":"2024-09-16T10:34:02.253954+0000","last_fullsized":"2024-09-16T10:34:02.253954+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20529,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106457+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:01.106457+0000","last_peered":"2024-09-16T10:34:01.106457+0000","last_clean":"2024-09-16T10:34:01.106457+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:01.106457+0000","last_undegraded":"2024-09-16T10:34:01.106457+0000","last_fullsized":"2024-09-16T10:34:01.106457+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":22061,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:04.284611+0000","last_change":"2024-09-16T10:34:00.273743+0000","last_active":"2024-09-16T10:34:04.284611+0000","last_peered":"2024-09-16T10:34:04.284611+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:34:04.284611+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107447+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:01.107447+0000","last_peered":"2024-09-16T10:34:01.107447+0000","last_clean":"2024-09-16T10:34:01.107447+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:01.107447+0000","last_undegraded":"2024-09-16T10:34:01.107447+0000","last_fullsized":"2024-09-16T10:34:01.107447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28975,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107688+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:01.107688+0000","last_peered":"2024-09-16T10:34:01.107688+0000","last_clean":"2024-09-16T10:34:01.107688+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:01.107688+0000","last_undegraded":"2024-09-16T10:34:01.107688+0000","last_fullsized":"2024-09-16T10:34:01.107688+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.105943+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:01.105943+0000","last_peered":"2024-09-16T10:34:01.105943+0000","last_clean":"2024-09-16T10:34:01.105943+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:01.105943+0000","last_undegraded":"2024-09-16T10:34:01.105943+0000","last_fullsized":"2024-09-16T10:34:01.105943+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20624,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.106309+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:34:01.106309+0000","last_peered":"2024-09-16T10:34:01.106309+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:01.106309+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":79,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107605+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:01.107605+0000","last_peered":"2024-09-16T10:34:01.107605+0000","last_clean":"2024-09-16T10:34:01.107605+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:01.107605+0000","last_undegraded":"2024-09-16T10:34:01.107605+0000","last_fullsized":"2024-09-16T10:34:01.107605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29069,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106133+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:01.106133+0000","last_peered":"2024-09-16T10:34:01.106133+0000","last_clean":"2024-09-16T10:34:01.106133+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:01.106133+0000","last_undegraded":"2024-09-16T10:34:01.106133+0000","last_fullsized":"2024-09-16T10:34:01.106133+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107546+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:01.107546+0000","last_peered":"2024-09-16T10:34:01.107546+0000","last_clean":"2024-09-16T10:34:01.107546+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:01.107546+0000","last_undegraded":"2024-09-16T10:34:01.107546+0000","last_fullsized":"2024-09-16T10:34:01.107546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106625+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:01.106625+0000","last_peered":"2024-09-16T10:34:01.106625+0000","last_clean":"2024-09-16T10:34:01.106625+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:01.106625+0000","last_undegraded":"2024-09-16T10:34:01.106625+0000","last_fullsized":"2024-09-16T10:34:01.106625+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24507,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107708+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:01.107708+0000","last_peered":"2024-09-16T10:34:01.107708+0000","last_clean":"2024-09-16T10:34:01.107708+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:01.107708+0000","last_undegraded":"2024-09-16T10:34:01.107708+0000","last_fullsized":"2024-09-16T10:34:01.107708+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107625+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:01.107625+0000","last_peered":"2024-09-16T10:34:01.107625+0000","last_clean":"2024-09-16T10:34:01.107625+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:01.107625+0000","last_undegraded":"2024-09-16T10:34:01.107625+0000","last_fullsized":"2024-09-16T10:34:01.107625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107986+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:01.107986+0000","last_peered":"2024-09-16T10:34:01.107986+0000","last_clean":"2024-09-16T10:34:01.107986+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:01.107986+0000","last_undegraded":"2024-09-16T10:34:01.107986+0000","last_fullsized":"2024-09-16T10:34:01.107986+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22350,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210210816,"data_stored":210153008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019019,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480192,"kb_used_data":137648,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891648,"statfs":{"total":96636764160,"available":96145047552,"internally_reserved":0,"allocated":140951552,"data_stored":140655838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.31900000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.26300000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.27600000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691963,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910668,"statfs":{"total":96636764160,"available":96164524032,"internally_reserved":0,"allocated":495616,"data_stored":152933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.245}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.23400000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.30099999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215246,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.188},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.25800000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:06.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:06.739 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:06.739 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:06.752 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-16T10:34:04.766242+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22351,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464660,"kb_used_data":206564,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650860,"statfs":{"total":289910292480,"available":288410480640,"internally_reserved":0,"allocated":211521536,"data_stored":210591039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2394,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001517"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20630,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107014+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:01.107014+0000","last_peered":"2024-09-16T10:34:01.107014+0000","last_clean":"2024-09-16T10:34:01.107014+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:01.107014+0000","last_undegraded":"2024-09-16T10:34:01.107014+0000","last_fullsized":"2024-09-16T10:34:01.107014+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20933,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.253954+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:02.253954+0000","last_peered":"2024-09-16T10:34:02.253954+0000","last_clean":"2024-09-16T10:34:02.253954+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:02.253954+0000","last_undegraded":"2024-09-16T10:34:02.253954+0000","last_fullsized":"2024-09-16T10:34:02.253954+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20529,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106457+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:01.106457+0000","last_peered":"2024-09-16T10:34:01.106457+0000","last_clean":"2024-09-16T10:34:01.106457+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:01.106457+0000","last_undegraded":"2024-09-16T10:34:01.106457+0000","last_fullsized":"2024-09-16T10:34:01.106457+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":22061,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:04.284611+0000","last_change":"2024-09-16T10:34:00.273743+0000","last_active":"2024-09-16T10:34:04.284611+0000","last_peered":"2024-09-16T10:34:04.284611+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:34:04.284611+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107447+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:01.107447+0000","last_peered":"2024-09-16T10:34:01.107447+0000","last_clean":"2024-09-16T10:34:01.107447+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:01.107447+0000","last_undegraded":"2024-09-16T10:34:01.107447+0000","last_fullsized":"2024-09-16T10:34:01.107447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28975,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107688+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:01.107688+0000","last_peered":"2024-09-16T10:34:01.107688+0000","last_clean":"2024-09-16T10:34:01.107688+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:01.107688+0000","last_undegraded":"2024-09-16T10:34:01.107688+0000","last_fullsized":"2024-09-16T10:34:01.107688+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.105943+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:01.105943+0000","last_peered":"2024-09-16T10:34:01.105943+0000","last_clean":"2024-09-16T10:34:01.105943+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:01.105943+0000","last_undegraded":"2024-09-16T10:34:01.105943+0000","last_fullsized":"2024-09-16T10:34:01.105943+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20624,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.106309+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:34:01.106309+0000","last_peered":"2024-09-16T10:34:01.106309+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:01.106309+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":79,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107605+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:01.107605+0000","last_peered":"2024-09-16T10:34:01.107605+0000","last_clean":"2024-09-16T10:34:01.107605+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:01.107605+0000","last_undegraded":"2024-09-16T10:34:01.107605+0000","last_fullsized":"2024-09-16T10:34:01.107605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29069,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106133+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:01.106133+0000","last_peered":"2024-09-16T10:34:01.106133+0000","last_clean":"2024-09-16T10:34:01.106133+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:01.106133+0000","last_undegraded":"2024-09-16T10:34:01.106133+0000","last_fullsized":"2024-09-16T10:34:01.106133+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107546+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:01.107546+0000","last_peered":"2024-09-16T10:34:01.107546+0000","last_clean":"2024-09-16T10:34:01.107546+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:01.107546+0000","last_undegraded":"2024-09-16T10:34:01.107546+0000","last_fullsized":"2024-09-16T10:34:01.107546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106625+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:01.106625+0000","last_peered":"2024-09-16T10:34:01.106625+0000","last_clean":"2024-09-16T10:34:01.106625+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:01.106625+0000","last_undegraded":"2024-09-16T10:34:01.106625+0000","last_fullsized":"2024-09-16T10:34:01.106625+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24507,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107708+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:01.107708+0000","last_peered":"2024-09-16T10:34:01.107708+0000","last_clean":"2024-09-16T10:34:01.107708+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:01.107708+0000","last_undegraded":"2024-09-16T10:34:01.107708+0000","last_fullsized":"2024-09-16T10:34:01.107708+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107625+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:01.107625+0000","last_peered":"2024-09-16T10:34:01.107625+0000","last_clean":"2024-09-16T10:34:01.107625+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:01.107625+0000","last_undegraded":"2024-09-16T10:34:01.107625+0000","last_fullsized":"2024-09-16T10:34:01.107625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107986+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:01.107986+0000","last_peered":"2024-09-16T10:34:01.107986+0000","last_clean":"2024-09-16T10:34:01.107986+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:01.107986+0000","last_undegraded":"2024-09-16T10:34:01.107986+0000","last_fullsized":"2024-09-16T10:34:01.107986+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22350,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210210816,"data_stored":210153008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019019,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480192,"kb_used_data":137648,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891648,"statfs":{"total":96636764160,"available":96145047552,"internally_reserved":0,"allocated":140951552,"data_stored":140655838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.31900000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.26300000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.27600000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691963,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910668,"statfs":{"total":96636764160,"available":96164524032,"internally_reserved":0,"allocated":495616,"data_stored":152933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.245}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.23400000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.30099999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215246,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.188},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.25800000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:06.754 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:34:06.754 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 22061, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:04.284611+0000', 'last_change': '2024-09-16T10:34:00.273743+0000', 'last_active': '2024-09-16T10:34:04.284611+0000', 'last_peered': '2024-09-16T10:34:04.284611+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:34:04.284611+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3462, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 757, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:06.754 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:06.754 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21337, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110253+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:01.110253+0000', 'last_peered': '2024-09-16T10:34:01.110253+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:01.110253+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:06.754 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:06.754 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21324, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.113489+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:01.113489+0000', 'last_peered': '2024-09-16T10:34:01.113489+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:01.113489+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:06.754 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:06.754 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20624, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.106309+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:34:01.106309+0000', 'last_peered': '2024-09-16T10:34:01.106309+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:34:01.106309+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:06.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:06.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20293, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110488+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:01.110488+0000', 'last_peered': '2024-09-16T10:34:01.110488+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:01.110488+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:06.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:06.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21198, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110537+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:34:01.110537+0000', 'last_peered': '2024-09-16T10:34:01.110537+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:01.110537+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:06.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:06.755 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21146, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110361+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:01.110361+0000', 'last_peered': '2024-09-16T10:34:01.110361+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:01.110361+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:06.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:07.248 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:07.248 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28996/40 objects degraded (72490.000%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":736,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":52,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1499811840,"bytes_avail":288410480640,"bytes_total":289910292480,"degraded_objects":28996,"degraded_total":40,"degraded_ratio":724.89999999999998,"recovering_objects_per_sec":175,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1401,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:07.263 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:07.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:07.610 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:07.610 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:07.622 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2024-09-16T10:34:06.766611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22351,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464660,"kb_used_data":206564,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650860,"statfs":{"total":289910292480,"available":288410480640,"internally_reserved":0,"allocated":211521536,"data_stored":210591039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1401,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001523"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20630,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107014+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:01.107014+0000","last_peered":"2024-09-16T10:34:01.107014+0000","last_clean":"2024-09-16T10:34:01.107014+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:01.107014+0000","last_undegraded":"2024-09-16T10:34:01.107014+0000","last_fullsized":"2024-09-16T10:34:01.107014+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20933,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.253954+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:02.253954+0000","last_peered":"2024-09-16T10:34:02.253954+0000","last_clean":"2024-09-16T10:34:02.253954+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:02.253954+0000","last_undegraded":"2024-09-16T10:34:02.253954+0000","last_fullsized":"2024-09-16T10:34:02.253954+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20529,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106457+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:01.106457+0000","last_peered":"2024-09-16T10:34:01.106457+0000","last_clean":"2024-09-16T10:34:01.106457+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:01.106457+0000","last_undegraded":"2024-09-16T10:34:01.106457+0000","last_fullsized":"2024-09-16T10:34:01.106457+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":22061,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:04.284611+0000","last_change":"2024-09-16T10:34:00.273743+0000","last_active":"2024-09-16T10:34:04.284611+0000","last_peered":"2024-09-16T10:34:04.284611+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:34:04.284611+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107447+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:01.107447+0000","last_peered":"2024-09-16T10:34:01.107447+0000","last_clean":"2024-09-16T10:34:01.107447+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:01.107447+0000","last_undegraded":"2024-09-16T10:34:01.107447+0000","last_fullsized":"2024-09-16T10:34:01.107447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28975,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107688+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:01.107688+0000","last_peered":"2024-09-16T10:34:01.107688+0000","last_clean":"2024-09-16T10:34:01.107688+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:01.107688+0000","last_undegraded":"2024-09-16T10:34:01.107688+0000","last_fullsized":"2024-09-16T10:34:01.107688+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.105943+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:01.105943+0000","last_peered":"2024-09-16T10:34:01.105943+0000","last_clean":"2024-09-16T10:34:01.105943+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:01.105943+0000","last_undegraded":"2024-09-16T10:34:01.105943+0000","last_fullsized":"2024-09-16T10:34:01.105943+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20624,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.106309+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:34:01.106309+0000","last_peered":"2024-09-16T10:34:01.106309+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:01.106309+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":79,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107605+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:01.107605+0000","last_peered":"2024-09-16T10:34:01.107605+0000","last_clean":"2024-09-16T10:34:01.107605+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:01.107605+0000","last_undegraded":"2024-09-16T10:34:01.107605+0000","last_fullsized":"2024-09-16T10:34:01.107605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29069,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106133+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:01.106133+0000","last_peered":"2024-09-16T10:34:01.106133+0000","last_clean":"2024-09-16T10:34:01.106133+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:01.106133+0000","last_undegraded":"2024-09-16T10:34:01.106133+0000","last_fullsized":"2024-09-16T10:34:01.106133+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107546+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:01.107546+0000","last_peered":"2024-09-16T10:34:01.107546+0000","last_clean":"2024-09-16T10:34:01.107546+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:01.107546+0000","last_undegraded":"2024-09-16T10:34:01.107546+0000","last_fullsized":"2024-09-16T10:34:01.107546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106625+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:01.106625+0000","last_peered":"2024-09-16T10:34:01.106625+0000","last_clean":"2024-09-16T10:34:01.106625+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:01.106625+0000","last_undegraded":"2024-09-16T10:34:01.106625+0000","last_fullsized":"2024-09-16T10:34:01.106625+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24507,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107708+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:01.107708+0000","last_peered":"2024-09-16T10:34:01.107708+0000","last_clean":"2024-09-16T10:34:01.107708+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:01.107708+0000","last_undegraded":"2024-09-16T10:34:01.107708+0000","last_fullsized":"2024-09-16T10:34:01.107708+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107625+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:01.107625+0000","last_peered":"2024-09-16T10:34:01.107625+0000","last_clean":"2024-09-16T10:34:01.107625+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:01.107625+0000","last_undegraded":"2024-09-16T10:34:01.107625+0000","last_fullsized":"2024-09-16T10:34:01.107625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107986+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:01.107986+0000","last_peered":"2024-09-16T10:34:01.107986+0000","last_clean":"2024-09-16T10:34:01.107986+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:01.107986+0000","last_undegraded":"2024-09-16T10:34:01.107986+0000","last_fullsized":"2024-09-16T10:34:01.107986+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22350,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210210816,"data_stored":210153008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019019,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480192,"kb_used_data":137648,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891648,"statfs":{"total":96636764160,"available":96145047552,"internally_reserved":0,"allocated":140951552,"data_stored":140655838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.31900000000000001},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.26300000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.27600000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691963,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461172,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910668,"statfs":{"total":96636764160,"available":96164524032,"internally_reserved":0,"allocated":495616,"data_stored":152933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.245}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.23400000000000001},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.30099999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215246,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.188},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.25800000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.29399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:10.625 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:10.973 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:10.974 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:10.989 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-16T10:34:10.767313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23457,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472844,"kb_used_data":206564,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281642676,"statfs":{"total":289910292480,"available":288402100224,"internally_reserved":0,"allocated":211521536,"data_stored":210591039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2507,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.662329"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20630,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107014+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:01.107014+0000","last_peered":"2024-09-16T10:34:01.107014+0000","last_clean":"2024-09-16T10:34:01.107014+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:01.107014+0000","last_undegraded":"2024-09-16T10:34:01.107014+0000","last_fullsized":"2024-09-16T10:34:01.107014+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20933,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.253954+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:02.253954+0000","last_peered":"2024-09-16T10:34:02.253954+0000","last_clean":"2024-09-16T10:34:02.253954+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:02.253954+0000","last_undegraded":"2024-09-16T10:34:02.253954+0000","last_fullsized":"2024-09-16T10:34:02.253954+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20529,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106457+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:01.106457+0000","last_peered":"2024-09-16T10:34:01.106457+0000","last_clean":"2024-09-16T10:34:01.106457+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:01.106457+0000","last_undegraded":"2024-09-16T10:34:01.106457+0000","last_fullsized":"2024-09-16T10:34:01.106457+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":24273,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:09.285905+0000","last_change":"2024-09-16T10:34:00.273743+0000","last_active":"2024-09-16T10:34:09.285905+0000","last_peered":"2024-09-16T10:34:09.285905+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:34:09.285905+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1863,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107447+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:01.107447+0000","last_peered":"2024-09-16T10:34:01.107447+0000","last_clean":"2024-09-16T10:34:01.107447+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:01.107447+0000","last_undegraded":"2024-09-16T10:34:01.107447+0000","last_fullsized":"2024-09-16T10:34:01.107447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28975,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107688+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:01.107688+0000","last_peered":"2024-09-16T10:34:01.107688+0000","last_clean":"2024-09-16T10:34:01.107688+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:01.107688+0000","last_undegraded":"2024-09-16T10:34:01.107688+0000","last_fullsized":"2024-09-16T10:34:01.107688+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.105943+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:01.105943+0000","last_peered":"2024-09-16T10:34:01.105943+0000","last_clean":"2024-09-16T10:34:01.105943+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:01.105943+0000","last_undegraded":"2024-09-16T10:34:01.105943+0000","last_fullsized":"2024-09-16T10:34:01.105943+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20624,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.106309+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:34:01.106309+0000","last_peered":"2024-09-16T10:34:01.106309+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:01.106309+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":79,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107605+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:01.107605+0000","last_peered":"2024-09-16T10:34:01.107605+0000","last_clean":"2024-09-16T10:34:01.107605+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:01.107605+0000","last_undegraded":"2024-09-16T10:34:01.107605+0000","last_fullsized":"2024-09-16T10:34:01.107605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29069,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106133+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:01.106133+0000","last_peered":"2024-09-16T10:34:01.106133+0000","last_clean":"2024-09-16T10:34:01.106133+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:01.106133+0000","last_undegraded":"2024-09-16T10:34:01.106133+0000","last_fullsized":"2024-09-16T10:34:01.106133+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107546+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:01.107546+0000","last_peered":"2024-09-16T10:34:01.107546+0000","last_clean":"2024-09-16T10:34:01.107546+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:01.107546+0000","last_undegraded":"2024-09-16T10:34:01.107546+0000","last_fullsized":"2024-09-16T10:34:01.107546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106625+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:01.106625+0000","last_peered":"2024-09-16T10:34:01.106625+0000","last_clean":"2024-09-16T10:34:01.106625+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:01.106625+0000","last_undegraded":"2024-09-16T10:34:01.106625+0000","last_fullsized":"2024-09-16T10:34:01.106625+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24507,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107708+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:01.107708+0000","last_peered":"2024-09-16T10:34:01.107708+0000","last_clean":"2024-09-16T10:34:01.107708+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:01.107708+0000","last_undegraded":"2024-09-16T10:34:01.107708+0000","last_fullsized":"2024-09-16T10:34:01.107708+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107625+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:01.107625+0000","last_peered":"2024-09-16T10:34:01.107625+0000","last_clean":"2024-09-16T10:34:01.107625+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:01.107625+0000","last_undegraded":"2024-09-16T10:34:01.107625+0000","last_fullsized":"2024-09-16T10:34:01.107625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107986+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:01.107986+0000","last_peered":"2024-09-16T10:34:01.107986+0000","last_clean":"2024-09-16T10:34:01.107986+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:01.107986+0000","last_undegraded":"2024-09-16T10:34:01.107986+0000","last_fullsized":"2024-09-16T10:34:01.107986+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23456,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210210816,"data_stored":210153008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019020,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480192,"kb_used_data":137648,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891648,"statfs":{"total":96636764160,"available":96145047552,"internally_reserved":0,"allocated":140951552,"data_stored":140655838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.183},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.20200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.224},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.28599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691964,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469356,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902484,"statfs":{"total":96636764160,"available":96156143616,"internally_reserved":0,"allocated":495616,"data_stored":152933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23200000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.24399999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215247,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.214}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.24299999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:10.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:11.325 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:11.326 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:11.339 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-16T10:34:10.767313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23457,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472844,"kb_used_data":206564,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281642676,"statfs":{"total":289910292480,"available":288402100224,"internally_reserved":0,"allocated":211521536,"data_stored":210591039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2507,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.662329"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20630,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107014+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:01.107014+0000","last_peered":"2024-09-16T10:34:01.107014+0000","last_clean":"2024-09-16T10:34:01.107014+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:01.107014+0000","last_undegraded":"2024-09-16T10:34:01.107014+0000","last_fullsized":"2024-09-16T10:34:01.107014+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20933,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.253954+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:02.253954+0000","last_peered":"2024-09-16T10:34:02.253954+0000","last_clean":"2024-09-16T10:34:02.253954+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:02.253954+0000","last_undegraded":"2024-09-16T10:34:02.253954+0000","last_fullsized":"2024-09-16T10:34:02.253954+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20529,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106457+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:01.106457+0000","last_peered":"2024-09-16T10:34:01.106457+0000","last_clean":"2024-09-16T10:34:01.106457+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:01.106457+0000","last_undegraded":"2024-09-16T10:34:01.106457+0000","last_fullsized":"2024-09-16T10:34:01.106457+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":24273,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:09.285905+0000","last_change":"2024-09-16T10:34:00.273743+0000","last_active":"2024-09-16T10:34:09.285905+0000","last_peered":"2024-09-16T10:34:09.285905+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:34:09.285905+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1863,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107447+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:01.107447+0000","last_peered":"2024-09-16T10:34:01.107447+0000","last_clean":"2024-09-16T10:34:01.107447+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:01.107447+0000","last_undegraded":"2024-09-16T10:34:01.107447+0000","last_fullsized":"2024-09-16T10:34:01.107447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28975,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107688+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:01.107688+0000","last_peered":"2024-09-16T10:34:01.107688+0000","last_clean":"2024-09-16T10:34:01.107688+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:01.107688+0000","last_undegraded":"2024-09-16T10:34:01.107688+0000","last_fullsized":"2024-09-16T10:34:01.107688+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.105943+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:01.105943+0000","last_peered":"2024-09-16T10:34:01.105943+0000","last_clean":"2024-09-16T10:34:01.105943+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:01.105943+0000","last_undegraded":"2024-09-16T10:34:01.105943+0000","last_fullsized":"2024-09-16T10:34:01.105943+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20624,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.106309+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:34:01.106309+0000","last_peered":"2024-09-16T10:34:01.106309+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:01.106309+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":79,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107605+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:01.107605+0000","last_peered":"2024-09-16T10:34:01.107605+0000","last_clean":"2024-09-16T10:34:01.107605+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:01.107605+0000","last_undegraded":"2024-09-16T10:34:01.107605+0000","last_fullsized":"2024-09-16T10:34:01.107605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29069,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106133+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:01.106133+0000","last_peered":"2024-09-16T10:34:01.106133+0000","last_clean":"2024-09-16T10:34:01.106133+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:01.106133+0000","last_undegraded":"2024-09-16T10:34:01.106133+0000","last_fullsized":"2024-09-16T10:34:01.106133+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107546+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:01.107546+0000","last_peered":"2024-09-16T10:34:01.107546+0000","last_clean":"2024-09-16T10:34:01.107546+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:01.107546+0000","last_undegraded":"2024-09-16T10:34:01.107546+0000","last_fullsized":"2024-09-16T10:34:01.107546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106625+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:01.106625+0000","last_peered":"2024-09-16T10:34:01.106625+0000","last_clean":"2024-09-16T10:34:01.106625+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:01.106625+0000","last_undegraded":"2024-09-16T10:34:01.106625+0000","last_fullsized":"2024-09-16T10:34:01.106625+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24507,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107708+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:01.107708+0000","last_peered":"2024-09-16T10:34:01.107708+0000","last_clean":"2024-09-16T10:34:01.107708+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:01.107708+0000","last_undegraded":"2024-09-16T10:34:01.107708+0000","last_fullsized":"2024-09-16T10:34:01.107708+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107625+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:01.107625+0000","last_peered":"2024-09-16T10:34:01.107625+0000","last_clean":"2024-09-16T10:34:01.107625+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:01.107625+0000","last_undegraded":"2024-09-16T10:34:01.107625+0000","last_fullsized":"2024-09-16T10:34:01.107625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107986+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:01.107986+0000","last_peered":"2024-09-16T10:34:01.107986+0000","last_clean":"2024-09-16T10:34:01.107986+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:01.107986+0000","last_undegraded":"2024-09-16T10:34:01.107986+0000","last_fullsized":"2024-09-16T10:34:01.107986+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23456,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210210816,"data_stored":210153008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019020,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480192,"kb_used_data":137648,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891648,"statfs":{"total":96636764160,"available":96145047552,"internally_reserved":0,"allocated":140951552,"data_stored":140655838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.183},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.20200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.224},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.28599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691964,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469356,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902484,"statfs":{"total":96636764160,"available":96156143616,"internally_reserved":0,"allocated":495616,"data_stored":152933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23200000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.24399999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215247,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.214}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.24299999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:11.340 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:34:11.340 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 24273, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:09.285905+0000', 'last_change': '2024-09-16T10:34:00.273743+0000', 'last_active': '2024-09-16T10:34:09.285905+0000', 'last_peered': '2024-09-16T10:34:09.285905+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:34:09.285905+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2356, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1863, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2356}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:11.340 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:11.340 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21337, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110253+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:01.110253+0000', 'last_peered': '2024-09-16T10:34:01.110253+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:01.110253+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:11.340 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:11.340 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21324, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.113489+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:01.113489+0000', 'last_peered': '2024-09-16T10:34:01.113489+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:01.113489+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:11.341 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:11.341 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20624, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.106309+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:34:01.106309+0000', 'last_peered': '2024-09-16T10:34:01.106309+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:34:01.106309+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:11.341 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:11.341 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20293, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110488+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:01.110488+0000', 'last_peered': '2024-09-16T10:34:01.110488+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:01.110488+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:11.341 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:11.341 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21198, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110537+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:34:01.110537+0000', 'last_peered': '2024-09-16T10:34:01.110537+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:01.110537+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:11.341 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:11.341 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21146, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110361+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:01.110361+0000', 'last_peered': '2024-09-16T10:34:01.110361+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:01.110361+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:11.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:11.762 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:11.762 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27890/40 objects degraded (69725.000%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":740,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":52,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1508192256,"bytes_avail":288402100224,"bytes_total":289910292480,"degraded_objects":27890,"degraded_total":40,"degraded_ratio":697.25,"recovering_objects_per_sec":259,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2507,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:11.776 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:11.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:12.113 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:12.113 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:12.126 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-16T10:34:10.767313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23457,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472844,"kb_used_data":206564,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281642676,"statfs":{"total":289910292480,"available":288402100224,"internally_reserved":0,"allocated":211521536,"data_stored":210591039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2507,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2507,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.662329"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20630,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107014+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:01.107014+0000","last_peered":"2024-09-16T10:34:01.107014+0000","last_clean":"2024-09-16T10:34:01.107014+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:01.107014+0000","last_undegraded":"2024-09-16T10:34:01.107014+0000","last_fullsized":"2024-09-16T10:34:01.107014+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20933,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.253954+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:02.253954+0000","last_peered":"2024-09-16T10:34:02.253954+0000","last_clean":"2024-09-16T10:34:02.253954+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:02.253954+0000","last_undegraded":"2024-09-16T10:34:02.253954+0000","last_fullsized":"2024-09-16T10:34:02.253954+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20529,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106457+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:01.106457+0000","last_peered":"2024-09-16T10:34:01.106457+0000","last_clean":"2024-09-16T10:34:01.106457+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:01.106457+0000","last_undegraded":"2024-09-16T10:34:01.106457+0000","last_fullsized":"2024-09-16T10:34:01.106457+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":24273,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:09.285905+0000","last_change":"2024-09-16T10:34:00.273743+0000","last_active":"2024-09-16T10:34:09.285905+0000","last_peered":"2024-09-16T10:34:09.285905+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:34:09.285905+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2356,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1863,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2356}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107447+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:01.107447+0000","last_peered":"2024-09-16T10:34:01.107447+0000","last_clean":"2024-09-16T10:34:01.107447+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:01.107447+0000","last_undegraded":"2024-09-16T10:34:01.107447+0000","last_fullsized":"2024-09-16T10:34:01.107447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28975,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107688+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:01.107688+0000","last_peered":"2024-09-16T10:34:01.107688+0000","last_clean":"2024-09-16T10:34:01.107688+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:01.107688+0000","last_undegraded":"2024-09-16T10:34:01.107688+0000","last_fullsized":"2024-09-16T10:34:01.107688+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.105943+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:01.105943+0000","last_peered":"2024-09-16T10:34:01.105943+0000","last_clean":"2024-09-16T10:34:01.105943+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:01.105943+0000","last_undegraded":"2024-09-16T10:34:01.105943+0000","last_fullsized":"2024-09-16T10:34:01.105943+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20624,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.106309+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:34:01.106309+0000","last_peered":"2024-09-16T10:34:01.106309+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:01.106309+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":79,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107605+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:01.107605+0000","last_peered":"2024-09-16T10:34:01.107605+0000","last_clean":"2024-09-16T10:34:01.107605+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:01.107605+0000","last_undegraded":"2024-09-16T10:34:01.107605+0000","last_fullsized":"2024-09-16T10:34:01.107605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29069,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106133+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:01.106133+0000","last_peered":"2024-09-16T10:34:01.106133+0000","last_clean":"2024-09-16T10:34:01.106133+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:01.106133+0000","last_undegraded":"2024-09-16T10:34:01.106133+0000","last_fullsized":"2024-09-16T10:34:01.106133+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107546+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:01.107546+0000","last_peered":"2024-09-16T10:34:01.107546+0000","last_clean":"2024-09-16T10:34:01.107546+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:01.107546+0000","last_undegraded":"2024-09-16T10:34:01.107546+0000","last_fullsized":"2024-09-16T10:34:01.107546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106625+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:01.106625+0000","last_peered":"2024-09-16T10:34:01.106625+0000","last_clean":"2024-09-16T10:34:01.106625+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:01.106625+0000","last_undegraded":"2024-09-16T10:34:01.106625+0000","last_fullsized":"2024-09-16T10:34:01.106625+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24507,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107708+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:01.107708+0000","last_peered":"2024-09-16T10:34:01.107708+0000","last_clean":"2024-09-16T10:34:01.107708+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:01.107708+0000","last_undegraded":"2024-09-16T10:34:01.107708+0000","last_fullsized":"2024-09-16T10:34:01.107708+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107625+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:01.107625+0000","last_peered":"2024-09-16T10:34:01.107625+0000","last_clean":"2024-09-16T10:34:01.107625+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:01.107625+0000","last_undegraded":"2024-09-16T10:34:01.107625+0000","last_fullsized":"2024-09-16T10:34:01.107625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107986+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:01.107986+0000","last_peered":"2024-09-16T10:34:01.107986+0000","last_clean":"2024-09-16T10:34:01.107986+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:01.107986+0000","last_undegraded":"2024-09-16T10:34:01.107986+0000","last_fullsized":"2024-09-16T10:34:01.107986+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23456,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210210816,"data_stored":210153008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019020,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480192,"kb_used_data":137648,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891648,"statfs":{"total":96636764160,"available":96145047552,"internally_reserved":0,"allocated":140951552,"data_stored":140655838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":2.8719999999999999,"5min":2.556,"15min":1.627},"min":{"1min":0.35799999999999998,"5min":0.19,"15min":0.184},"max":{"1min":11.718999999999999,"5min":13.74,"15min":13.74},"last":0.183},{"interface":"front","average":{"1min":2.738,"5min":2.0209999999999999,"15min":1.752},"min":{"1min":0.24199999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":12.265000000000001,"5min":13.917,"15min":13.917},"last":0.20200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:33:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6230000000000002,"5min":3.2869999999999999,"15min":3.0169999999999999},"min":{"1min":0.224,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":19.518999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.224},{"interface":"front","average":{"1min":3.423,"5min":3.5579999999999998,"15min":2.7850000000000001},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":11.766999999999999,"5min":17.914000000000001,"15min":17.914000000000001},"last":0.28599999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691964,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469356,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902484,"statfs":{"total":96636764160,"available":96156143616,"internally_reserved":0,"allocated":495616,"data_stored":152933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.20599999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.23200000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.24399999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215247,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.214}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.24299999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:15.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:15.459 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:15.459 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:15.471 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-16T10:34:14.768073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24873,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1468748,"kb_used_data":206564,"kb_used_omap":12,"kb_used_meta":1262131,"kb_avail":281646772,"statfs":{"total":289910292480,"available":288406294528,"internally_reserved":0,"allocated":211521536,"data_stored":210591039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1292422146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3923,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20630,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107014+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:01.107014+0000","last_peered":"2024-09-16T10:34:01.107014+0000","last_clean":"2024-09-16T10:34:01.107014+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:01.107014+0000","last_undegraded":"2024-09-16T10:34:01.107014+0000","last_fullsized":"2024-09-16T10:34:01.107014+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20933,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.253954+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:02.253954+0000","last_peered":"2024-09-16T10:34:02.253954+0000","last_clean":"2024-09-16T10:34:02.253954+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:02.253954+0000","last_undegraded":"2024-09-16T10:34:02.253954+0000","last_fullsized":"2024-09-16T10:34:02.253954+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20530,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.108178+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:02.108178+0000","last_peered":"2024-09-16T10:34:02.108178+0000","last_clean":"2024-09-16T10:34:02.108178+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:02.108178+0000","last_undegraded":"2024-09-16T10:34:02.108178+0000","last_fullsized":"2024-09-16T10:34:02.108178+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":27105,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:14.288850+0000","last_change":"2024-09-16T10:34:00.273743+0000","last_active":"2024-09-16T10:34:14.288850+0000","last_peered":"2024-09-16T10:34:14.288850+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:34:14.288850+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107447+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:01.107447+0000","last_peered":"2024-09-16T10:34:01.107447+0000","last_clean":"2024-09-16T10:34:01.107447+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:01.107447+0000","last_undegraded":"2024-09-16T10:34:01.107447+0000","last_fullsized":"2024-09-16T10:34:01.107447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28975,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107688+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:01.107688+0000","last_peered":"2024-09-16T10:34:01.107688+0000","last_clean":"2024-09-16T10:34:01.107688+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:01.107688+0000","last_undegraded":"2024-09-16T10:34:01.107688+0000","last_fullsized":"2024-09-16T10:34:01.107688+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.105943+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:01.105943+0000","last_peered":"2024-09-16T10:34:01.105943+0000","last_clean":"2024-09-16T10:34:01.105943+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:01.105943+0000","last_undegraded":"2024-09-16T10:34:01.105943+0000","last_fullsized":"2024-09-16T10:34:01.105943+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20624,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.106309+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:34:01.106309+0000","last_peered":"2024-09-16T10:34:01.106309+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:01.106309+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":79,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107605+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:01.107605+0000","last_peered":"2024-09-16T10:34:01.107605+0000","last_clean":"2024-09-16T10:34:01.107605+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:01.107605+0000","last_undegraded":"2024-09-16T10:34:01.107605+0000","last_fullsized":"2024-09-16T10:34:01.107605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29069,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106133+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:01.106133+0000","last_peered":"2024-09-16T10:34:01.106133+0000","last_clean":"2024-09-16T10:34:01.106133+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:01.106133+0000","last_undegraded":"2024-09-16T10:34:01.106133+0000","last_fullsized":"2024-09-16T10:34:01.106133+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107546+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:01.107546+0000","last_peered":"2024-09-16T10:34:01.107546+0000","last_clean":"2024-09-16T10:34:01.107546+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:01.107546+0000","last_undegraded":"2024-09-16T10:34:01.107546+0000","last_fullsized":"2024-09-16T10:34:01.107546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106625+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:01.106625+0000","last_peered":"2024-09-16T10:34:01.106625+0000","last_clean":"2024-09-16T10:34:01.106625+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:01.106625+0000","last_undegraded":"2024-09-16T10:34:01.106625+0000","last_fullsized":"2024-09-16T10:34:01.106625+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24507,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107708+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:01.107708+0000","last_peered":"2024-09-16T10:34:01.107708+0000","last_clean":"2024-09-16T10:34:01.107708+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:01.107708+0000","last_undegraded":"2024-09-16T10:34:01.107708+0000","last_fullsized":"2024-09-16T10:34:01.107708+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107625+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:01.107625+0000","last_peered":"2024-09-16T10:34:01.107625+0000","last_clean":"2024-09-16T10:34:01.107625+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:01.107625+0000","last_undegraded":"2024-09-16T10:34:01.107625+0000","last_fullsized":"2024-09-16T10:34:01.107625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107986+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:01.107986+0000","last_peered":"2024-09-16T10:34:01.107986+0000","last_clean":"2024-09-16T10:34:01.107986+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:01.107986+0000","last_undegraded":"2024-09-16T10:34:01.107986+0000","last_fullsized":"2024-09-16T10:34:01.107986+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24872,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210210816,"data_stored":210153008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019021,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480192,"kb_used_data":137648,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891648,"statfs":{"total":96636764160,"available":96145047552,"internally_reserved":0,"allocated":140951552,"data_stored":140655838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.215}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.23200000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691965,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465260,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93906580,"statfs":{"total":96636764160,"available":96160337920,"internally_reserved":0,"allocated":495616,"data_stored":152933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":475917994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.17999999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.218}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215248,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17699999999999999},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:15.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:15.812 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:15.813 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:15.825 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-16T10:34:14.768073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24873,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1468748,"kb_used_data":206564,"kb_used_omap":12,"kb_used_meta":1262131,"kb_avail":281646772,"statfs":{"total":289910292480,"available":288406294528,"internally_reserved":0,"allocated":211521536,"data_stored":210591039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1292422146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3923,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20630,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107014+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:01.107014+0000","last_peered":"2024-09-16T10:34:01.107014+0000","last_clean":"2024-09-16T10:34:01.107014+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:01.107014+0000","last_undegraded":"2024-09-16T10:34:01.107014+0000","last_fullsized":"2024-09-16T10:34:01.107014+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20933,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.253954+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:02.253954+0000","last_peered":"2024-09-16T10:34:02.253954+0000","last_clean":"2024-09-16T10:34:02.253954+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:02.253954+0000","last_undegraded":"2024-09-16T10:34:02.253954+0000","last_fullsized":"2024-09-16T10:34:02.253954+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20530,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.108178+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:02.108178+0000","last_peered":"2024-09-16T10:34:02.108178+0000","last_clean":"2024-09-16T10:34:02.108178+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:02.108178+0000","last_undegraded":"2024-09-16T10:34:02.108178+0000","last_fullsized":"2024-09-16T10:34:02.108178+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":27105,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:14.288850+0000","last_change":"2024-09-16T10:34:00.273743+0000","last_active":"2024-09-16T10:34:14.288850+0000","last_peered":"2024-09-16T10:34:14.288850+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:34:14.288850+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107447+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:01.107447+0000","last_peered":"2024-09-16T10:34:01.107447+0000","last_clean":"2024-09-16T10:34:01.107447+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:01.107447+0000","last_undegraded":"2024-09-16T10:34:01.107447+0000","last_fullsized":"2024-09-16T10:34:01.107447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28975,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107688+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:01.107688+0000","last_peered":"2024-09-16T10:34:01.107688+0000","last_clean":"2024-09-16T10:34:01.107688+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:01.107688+0000","last_undegraded":"2024-09-16T10:34:01.107688+0000","last_fullsized":"2024-09-16T10:34:01.107688+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.105943+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:01.105943+0000","last_peered":"2024-09-16T10:34:01.105943+0000","last_clean":"2024-09-16T10:34:01.105943+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:01.105943+0000","last_undegraded":"2024-09-16T10:34:01.105943+0000","last_fullsized":"2024-09-16T10:34:01.105943+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20624,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.106309+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:34:01.106309+0000","last_peered":"2024-09-16T10:34:01.106309+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:01.106309+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":79,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107605+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:01.107605+0000","last_peered":"2024-09-16T10:34:01.107605+0000","last_clean":"2024-09-16T10:34:01.107605+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:01.107605+0000","last_undegraded":"2024-09-16T10:34:01.107605+0000","last_fullsized":"2024-09-16T10:34:01.107605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29069,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106133+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:01.106133+0000","last_peered":"2024-09-16T10:34:01.106133+0000","last_clean":"2024-09-16T10:34:01.106133+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:01.106133+0000","last_undegraded":"2024-09-16T10:34:01.106133+0000","last_fullsized":"2024-09-16T10:34:01.106133+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107546+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:01.107546+0000","last_peered":"2024-09-16T10:34:01.107546+0000","last_clean":"2024-09-16T10:34:01.107546+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:01.107546+0000","last_undegraded":"2024-09-16T10:34:01.107546+0000","last_fullsized":"2024-09-16T10:34:01.107546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106625+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:01.106625+0000","last_peered":"2024-09-16T10:34:01.106625+0000","last_clean":"2024-09-16T10:34:01.106625+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:01.106625+0000","last_undegraded":"2024-09-16T10:34:01.106625+0000","last_fullsized":"2024-09-16T10:34:01.106625+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24507,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107708+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:01.107708+0000","last_peered":"2024-09-16T10:34:01.107708+0000","last_clean":"2024-09-16T10:34:01.107708+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:01.107708+0000","last_undegraded":"2024-09-16T10:34:01.107708+0000","last_fullsized":"2024-09-16T10:34:01.107708+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107625+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:01.107625+0000","last_peered":"2024-09-16T10:34:01.107625+0000","last_clean":"2024-09-16T10:34:01.107625+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:01.107625+0000","last_undegraded":"2024-09-16T10:34:01.107625+0000","last_fullsized":"2024-09-16T10:34:01.107625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107986+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:01.107986+0000","last_peered":"2024-09-16T10:34:01.107986+0000","last_clean":"2024-09-16T10:34:01.107986+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:01.107986+0000","last_undegraded":"2024-09-16T10:34:01.107986+0000","last_fullsized":"2024-09-16T10:34:01.107986+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24872,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210210816,"data_stored":210153008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019021,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480192,"kb_used_data":137648,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891648,"statfs":{"total":96636764160,"available":96145047552,"internally_reserved":0,"allocated":140951552,"data_stored":140655838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.215}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.23200000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691965,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465260,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93906580,"statfs":{"total":96636764160,"available":96160337920,"internally_reserved":0,"allocated":495616,"data_stored":152933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":475917994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.17999999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.218}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215248,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17699999999999999},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:15.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.a is not active+clean 2024-09-16T10:34:15.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.a', 'version': "48'20371", 'reported_seq': 27105, 'reported_epoch': 52, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:14.288850+0000', 'last_change': '2024-09-16T10:34:00.273743+0000', 'last_active': '2024-09-16T10:34:14.288850+0000', 'last_peered': '2024-09-16T10:34:14.288850+0000', 'last_clean': '2024-09-16T10:23:54.368191+0000', 'last_became_active': '2024-09-16T10:23:56.900833+0000', 'last_became_peered': '2024-09-16T10:23:56.900833+0000', 'last_unstale': '2024-09-16T10:34:14.288850+0000', 'last_undegraded': '2024-09-16T10:23:56.482906+0000', 'last_fullsized': '2024-09-16T10:23:56.369849+0000', 'mapping_epoch': 41, 'log_start': "44'10300", 'ondisk_log_start': "44'10300", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3438", 'last_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:41.819370+0000', 'objects_scrubbed': 3370, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000', 'scrub_duration': 1.35400025, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 940, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 16996, 'num_write_kb': 27252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3279, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:15.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:15.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21337, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110253+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:01.110253+0000', 'last_peered': '2024-09-16T10:34:01.110253+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:01.110253+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:15.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:15.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21324, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.113489+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:01.113489+0000', 'last_peered': '2024-09-16T10:34:01.113489+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:01.113489+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:15.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:15.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20624, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.106309+0000', 'last_change': '2024-09-16T10:23:56.908635+0000', 'last_active': '2024-09-16T10:34:01.106309+0000', 'last_peered': '2024-09-16T10:34:01.106309+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:34:01.106309+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4235, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:15.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:15.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20293, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110488+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:01.110488+0000', 'last_peered': '2024-09-16T10:34:01.110488+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:01.110488+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:15.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:15.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21198, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110537+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:34:01.110537+0000', 'last_peered': '2024-09-16T10:34:01.110537+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:01.110537+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:15.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:15.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21146, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110361+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:01.110361+0000', 'last_peered': '2024-09-16T10:34:01.110361+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:01.110361+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:15.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:16.213 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:16.213 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26474/40 objects degraded (66185.000%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":745,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":52,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":17},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1503997952,"bytes_avail":288406294528,"bytes_total":289910292480,"degraded_objects":26474,"degraded_total":40,"degraded_ratio":661.85000000000002,"recovering_objects_per_sec":326,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":3923,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:16.227 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:16.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:16.555 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:16.555 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:16.568 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":417,"stamp":"2024-09-16T10:34:14.768073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24873,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":41,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1468748,"kb_used_data":206564,"kb_used_omap":12,"kb_used_meta":1262131,"kb_avail":281646772,"statfs":{"total":289910292480,"available":288406294528,"internally_reserved":0,"allocated":211521536,"data_stored":210591039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1292422146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3923,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3923,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002189"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20630,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107014+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:01.107014+0000","last_peered":"2024-09-16T10:34:01.107014+0000","last_clean":"2024-09-16T10:34:01.107014+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:01.107014+0000","last_undegraded":"2024-09-16T10:34:01.107014+0000","last_fullsized":"2024-09-16T10:34:01.107014+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20933,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.253954+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:02.253954+0000","last_peered":"2024-09-16T10:34:02.253954+0000","last_clean":"2024-09-16T10:34:02.253954+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:02.253954+0000","last_undegraded":"2024-09-16T10:34:02.253954+0000","last_fullsized":"2024-09-16T10:34:02.253954+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20530,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.108178+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:02.108178+0000","last_peered":"2024-09-16T10:34:02.108178+0000","last_clean":"2024-09-16T10:34:02.108178+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:02.108178+0000","last_undegraded":"2024-09-16T10:34:02.108178+0000","last_fullsized":"2024-09-16T10:34:02.108178+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":27105,"reported_epoch":52,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:14.288850+0000","last_change":"2024-09-16T10:34:00.273743+0000","last_active":"2024-09-16T10:34:14.288850+0000","last_peered":"2024-09-16T10:34:14.288850+0000","last_clean":"2024-09-16T10:23:54.368191+0000","last_became_active":"2024-09-16T10:23:56.900833+0000","last_became_peered":"2024-09-16T10:23:56.900833+0000","last_unstale":"2024-09-16T10:34:14.288850+0000","last_undegraded":"2024-09-16T10:23:56.482906+0000","last_fullsized":"2024-09-16T10:23:56.369849+0000","mapping_epoch":41,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:53:21.196737+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107447+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:01.107447+0000","last_peered":"2024-09-16T10:34:01.107447+0000","last_clean":"2024-09-16T10:34:01.107447+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:01.107447+0000","last_undegraded":"2024-09-16T10:34:01.107447+0000","last_fullsized":"2024-09-16T10:34:01.107447+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28975,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107688+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:01.107688+0000","last_peered":"2024-09-16T10:34:01.107688+0000","last_clean":"2024-09-16T10:34:01.107688+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:01.107688+0000","last_undegraded":"2024-09-16T10:34:01.107688+0000","last_fullsized":"2024-09-16T10:34:01.107688+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.105943+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:01.105943+0000","last_peered":"2024-09-16T10:34:01.105943+0000","last_clean":"2024-09-16T10:34:01.105943+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:01.105943+0000","last_undegraded":"2024-09-16T10:34:01.105943+0000","last_fullsized":"2024-09-16T10:34:01.105943+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20624,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.106309+0000","last_change":"2024-09-16T10:23:56.908635+0000","last_active":"2024-09-16T10:34:01.106309+0000","last_peered":"2024-09-16T10:34:01.106309+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:01.106309+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":79,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107605+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:01.107605+0000","last_peered":"2024-09-16T10:34:01.107605+0000","last_clean":"2024-09-16T10:34:01.107605+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:01.107605+0000","last_undegraded":"2024-09-16T10:34:01.107605+0000","last_fullsized":"2024-09-16T10:34:01.107605+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29069,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106133+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:01.106133+0000","last_peered":"2024-09-16T10:34:01.106133+0000","last_clean":"2024-09-16T10:34:01.106133+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:01.106133+0000","last_undegraded":"2024-09-16T10:34:01.106133+0000","last_fullsized":"2024-09-16T10:34:01.106133+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107546+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:01.107546+0000","last_peered":"2024-09-16T10:34:01.107546+0000","last_clean":"2024-09-16T10:34:01.107546+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:01.107546+0000","last_undegraded":"2024-09-16T10:34:01.107546+0000","last_fullsized":"2024-09-16T10:34:01.107546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":68,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.106625+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:01.106625+0000","last_peered":"2024-09-16T10:34:01.106625+0000","last_clean":"2024-09-16T10:34:01.106625+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:01.106625+0000","last_undegraded":"2024-09-16T10:34:01.106625+0000","last_fullsized":"2024-09-16T10:34:01.106625+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24507,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107708+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:01.107708+0000","last_peered":"2024-09-16T10:34:01.107708+0000","last_clean":"2024-09-16T10:34:01.107708+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:01.107708+0000","last_undegraded":"2024-09-16T10:34:01.107708+0000","last_fullsized":"2024-09-16T10:34:01.107708+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107625+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:01.107625+0000","last_peered":"2024-09-16T10:34:01.107625+0000","last_clean":"2024-09-16T10:34:01.107625+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:01.107625+0000","last_undegraded":"2024-09-16T10:34:01.107625+0000","last_fullsized":"2024-09-16T10:34:01.107625+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":74,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.107986+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:01.107986+0000","last_peered":"2024-09-16T10:34:01.107986+0000","last_clean":"2024-09-16T10:34:01.107986+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:01.107986+0000","last_undegraded":"2024-09-16T10:34:01.107986+0000","last_fullsized":"2024-09-16T10:34:01.107986+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26474,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24872,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210210816,"data_stored":210153008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":25,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019021,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480192,"kb_used_data":137648,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891648,"statfs":{"total":96636764160,"available":96145047552,"internally_reserved":0,"allocated":140951552,"data_stored":140655838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17699999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.215}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.23200000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691965,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":465260,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93906580,"statfs":{"total":96636764160,"available":96160337920,"internally_reserved":0,"allocated":495616,"data_stored":152933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":475917994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.17999999999999999},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.218}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.26800000000000002},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215248,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17699999999999999},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":6762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:19.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:19.883 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:19.883 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:19.897 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-16T10:34:18.768739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25937,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464660,"kb_used_data":206572,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650860,"statfs":{"total":289910292480,"available":288410480640,"internally_reserved":0,"allocated":211529728,"data_stored":210595833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001425"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20631,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.109324+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:02.109324+0000","last_peered":"2024-09-16T10:34:02.109324+0000","last_clean":"2024-09-16T10:34:02.109324+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:02.109324+0000","last_undegraded":"2024-09-16T10:34:02.109324+0000","last_fullsized":"2024-09-16T10:34:02.109324+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20531,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:12.693230+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:12.693230+0000","last_peered":"2024-09-16T10:34:12.693230+0000","last_clean":"2024-09-16T10:34:12.693230+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:12.693230+0000","last_undegraded":"2024-09-16T10:34:12.693230+0000","last_fullsized":"2024-09-16T10:34:12.693230+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28977,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197214+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:17.197214+0000","last_peered":"2024-09-16T10:34:17.197214+0000","last_clean":"2024-09-16T10:34:17.197214+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:17.197214+0000","last_undegraded":"2024-09-16T10:34:17.197214+0000","last_fullsized":"2024-09-16T10:34:17.197214+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.107835+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:02.107835+0000","last_peered":"2024-09-16T10:34:02.107835+0000","last_clean":"2024-09-16T10:34:02.107835+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:02.107835+0000","last_undegraded":"2024-09-16T10:34:02.107835+0000","last_fullsized":"2024-09-16T10:34:02.107835+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20874,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.696425+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:17.696425+0000","last_peered":"2024-09-16T10:34:17.696425+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:17.696425+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":81,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196738+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:17.196738+0000","last_peered":"2024-09-16T10:34:17.196738+0000","last_clean":"2024-09-16T10:34:17.196738+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:17.196738+0000","last_undegraded":"2024-09-16T10:34:17.196738+0000","last_fullsized":"2024-09-16T10:34:17.196738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29070,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.107650+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:02.107650+0000","last_peered":"2024-09-16T10:34:02.107650+0000","last_clean":"2024-09-16T10:34:02.107650+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:02.107650+0000","last_undegraded":"2024-09-16T10:34:02.107650+0000","last_fullsized":"2024-09-16T10:34:02.107650+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197113+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:17.197113+0000","last_peered":"2024-09-16T10:34:17.197113+0000","last_clean":"2024-09-16T10:34:17.197113+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:17.197113+0000","last_undegraded":"2024-09-16T10:34:17.197113+0000","last_fullsized":"2024-09-16T10:34:17.197113+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.109284+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:02.109284+0000","last_peered":"2024-09-16T10:34:02.109284+0000","last_clean":"2024-09-16T10:34:02.109284+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:02.109284+0000","last_undegraded":"2024-09-16T10:34:02.109284+0000","last_fullsized":"2024-09-16T10:34:02.109284+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197071+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:17.197071+0000","last_peered":"2024-09-16T10:34:17.197071+0000","last_clean":"2024-09-16T10:34:17.197071+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:17.197071+0000","last_undegraded":"2024-09-16T10:34:17.197071+0000","last_fullsized":"2024-09-16T10:34:17.197071+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25936,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019022,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480196,"kb_used_data":137656,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891644,"statfs":{"total":96636764160,"available":96145043456,"internally_reserved":0,"allocated":140959744,"data_stored":140658718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.221},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.252}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.27000000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691966,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461168,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910672,"statfs":{"total":96636764160,"available":96164528128,"internally_reserved":0,"allocated":495616,"data_stored":154847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.156},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.185}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.26900000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215248,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17699999999999999},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:19.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:20.293 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:20.294 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:20.307 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-09-16T10:34:18.768739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25937,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464660,"kb_used_data":206572,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650860,"statfs":{"total":289910292480,"available":288410480640,"internally_reserved":0,"allocated":211529728,"data_stored":210595833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001425"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28941,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.110985+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:01.110985+0000","last_peered":"2024-09-16T10:34:01.110985+0000","last_clean":"2024-09-16T10:34:01.110985+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:01.110985+0000","last_undegraded":"2024-09-16T10:34:01.110985+0000","last_fullsized":"2024-09-16T10:34:01.110985+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20631,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.109324+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:02.109324+0000","last_peered":"2024-09-16T10:34:02.109324+0000","last_clean":"2024-09-16T10:34:02.109324+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:02.109324+0000","last_undegraded":"2024-09-16T10:34:02.109324+0000","last_fullsized":"2024-09-16T10:34:02.109324+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20531,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:12.693230+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:12.693230+0000","last_peered":"2024-09-16T10:34:12.693230+0000","last_clean":"2024-09-16T10:34:12.693230+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:12.693230+0000","last_undegraded":"2024-09-16T10:34:12.693230+0000","last_fullsized":"2024-09-16T10:34:12.693230+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20673,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109945+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:01.109945+0000","last_peered":"2024-09-16T10:34:01.109945+0000","last_clean":"2024-09-16T10:34:01.109945+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:01.109945+0000","last_undegraded":"2024-09-16T10:34:01.109945+0000","last_fullsized":"2024-09-16T10:34:01.109945+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21337,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110253+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:01.110253+0000","last_peered":"2024-09-16T10:34:01.110253+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:01.110253+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20576,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109808+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:01.109808+0000","last_peered":"2024-09-16T10:34:01.109808+0000","last_clean":"2024-09-16T10:34:01.109808+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:01.109808+0000","last_undegraded":"2024-09-16T10:34:01.109808+0000","last_fullsized":"2024-09-16T10:34:01.109808+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28977,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197214+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:17.197214+0000","last_peered":"2024-09-16T10:34:17.197214+0000","last_clean":"2024-09-16T10:34:17.197214+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:17.197214+0000","last_undegraded":"2024-09-16T10:34:17.197214+0000","last_fullsized":"2024-09-16T10:34:17.197214+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.107835+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:02.107835+0000","last_peered":"2024-09-16T10:34:02.107835+0000","last_clean":"2024-09-16T10:34:02.107835+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:02.107835+0000","last_undegraded":"2024-09-16T10:34:02.107835+0000","last_fullsized":"2024-09-16T10:34:02.107835+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21324,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.113489+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:01.113489+0000","last_peered":"2024-09-16T10:34:01.113489+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:01.113489+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":99,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-16T10:34:01.109866+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:01.109866+0000","last_peered":"2024-09-16T10:34:01.109866+0000","last_clean":"2024-09-16T10:34:01.109866+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:01.109866+0000","last_undegraded":"2024-09-16T10:34:01.109866+0000","last_fullsized":"2024-09-16T10:34:01.109866+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20874,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.696425+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:17.696425+0000","last_peered":"2024-09-16T10:34:17.696425+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:17.696425+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":81,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196738+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:17.196738+0000","last_peered":"2024-09-16T10:34:17.196738+0000","last_clean":"2024-09-16T10:34:17.196738+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:17.196738+0000","last_undegraded":"2024-09-16T10:34:17.196738+0000","last_fullsized":"2024-09-16T10:34:17.196738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29070,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.107650+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:02.107650+0000","last_peered":"2024-09-16T10:34:02.107650+0000","last_clean":"2024-09-16T10:34:02.107650+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:02.107650+0000","last_undegraded":"2024-09-16T10:34:02.107650+0000","last_fullsized":"2024-09-16T10:34:02.107650+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20293,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110488+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:01.110488+0000","last_peered":"2024-09-16T10:34:01.110488+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:01.110488+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197113+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:17.197113+0000","last_peered":"2024-09-16T10:34:17.197113+0000","last_clean":"2024-09-16T10:34:17.197113+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:17.197113+0000","last_undegraded":"2024-09-16T10:34:17.197113+0000","last_fullsized":"2024-09-16T10:34:17.197113+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21198,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110537+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:01.110537+0000","last_peered":"2024-09-16T10:34:01.110537+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:01.110537+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.109284+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:02.109284+0000","last_peered":"2024-09-16T10:34:02.109284+0000","last_clean":"2024-09-16T10:34:02.109284+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:02.109284+0000","last_undegraded":"2024-09-16T10:34:02.109284+0000","last_fullsized":"2024-09-16T10:34:02.109284+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197071+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:17.197071+0000","last_peered":"2024-09-16T10:34:17.197071+0000","last_clean":"2024-09-16T10:34:17.197071+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:17.197071+0000","last_undegraded":"2024-09-16T10:34:17.197071+0000","last_fullsized":"2024-09-16T10:34:17.197071+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21146,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:01.110361+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:01.110361+0000","last_peered":"2024-09-16T10:34:01.110361+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:01.110361+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25936,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019022,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480196,"kb_used_data":137656,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891644,"statfs":{"total":96636764160,"available":96145043456,"internally_reserved":0,"allocated":140959744,"data_stored":140658718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.221},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.252}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.27000000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691966,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461168,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910672,"statfs":{"total":96636764160,"available":96164528128,"internally_reserved":0,"allocated":495616,"data_stored":154847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.156},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.185}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.26900000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215248,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523296,"kb_used_data":68432,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848544,"statfs":{"total":96636764160,"available":96100909056,"internally_reserved":0,"allocated":70074368,"data_stored":69782268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17699999999999999},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.25600000000000001},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:20.308 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:20.308 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21337, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110253+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:01.110253+0000', 'last_peered': '2024-09-16T10:34:01.110253+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:01.110253+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:20.308 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:20.308 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21324, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.113489+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:01.113489+0000', 'last_peered': '2024-09-16T10:34:01.113489+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:01.113489+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:20.308 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:20.308 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 20874, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.696425+0000', 'last_change': '2024-09-16T10:34:17.012403+0000', 'last_active': '2024-09-16T10:34:17.696425+0000', 'last_peered': '2024-09-16T10:34:17.696425+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:34:17.696425+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4111, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 170, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -4109}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:20.309 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:20.309 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20293, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110488+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:01.110488+0000', 'last_peered': '2024-09-16T10:34:01.110488+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:01.110488+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:20.309 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:20.309 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21198, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110537+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:34:01.110537+0000', 'last_peered': '2024-09-16T10:34:01.110537+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:01.110537+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:20.309 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:20.309 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21146, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:01.110361+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:01.110361+0000', 'last_peered': '2024-09-16T10:34:01.110361+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:01.110361+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:20.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:20.702 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:20.702 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 25410/40 objects degraded (63525.000%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":749,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":54,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1499811840,"bytes_avail":288410480640,"bytes_total":289910292480,"degraded_objects":25410,"degraded_total":40,"degraded_ratio":635.25,"recovering_objects_per_sec":309,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2480,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:20.716 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:20.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:21.076 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:21.076 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:21.088 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-16T10:34:20.769075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25937,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464676,"kb_used_data":206588,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650844,"statfs":{"total":289910292480,"available":288410464256,"internally_reserved":0,"allocated":211546112,"data_stored":210601572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2480,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2480,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001337"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28943,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.200232+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:17.200232+0000","last_peered":"2024-09-16T10:34:17.200232+0000","last_clean":"2024-09-16T10:34:17.200232+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:17.200232+0000","last_undegraded":"2024-09-16T10:34:17.200232+0000","last_fullsized":"2024-09-16T10:34:17.200232+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20631,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.109324+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:02.109324+0000","last_peered":"2024-09-16T10:34:02.109324+0000","last_clean":"2024-09-16T10:34:02.109324+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:02.109324+0000","last_undegraded":"2024-09-16T10:34:02.109324+0000","last_fullsized":"2024-09-16T10:34:02.109324+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20531,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:12.693230+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:12.693230+0000","last_peered":"2024-09-16T10:34:12.693230+0000","last_clean":"2024-09-16T10:34:12.693230+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:12.693230+0000","last_undegraded":"2024-09-16T10:34:12.693230+0000","last_fullsized":"2024-09-16T10:34:12.693230+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199693+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:17.199693+0000","last_peered":"2024-09-16T10:34:17.199693+0000","last_clean":"2024-09-16T10:34:17.199693+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:17.199693+0000","last_undegraded":"2024-09-16T10:34:17.199693+0000","last_fullsized":"2024-09-16T10:34:17.199693+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.199768+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:17.199768+0000","last_peered":"2024-09-16T10:34:17.199768+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:17.199768+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199363+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:17.199363+0000","last_peered":"2024-09-16T10:34:17.199363+0000","last_clean":"2024-09-16T10:34:17.199363+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:17.199363+0000","last_undegraded":"2024-09-16T10:34:17.199363+0000","last_fullsized":"2024-09-16T10:34:17.199363+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28977,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197214+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:17.197214+0000","last_peered":"2024-09-16T10:34:17.197214+0000","last_clean":"2024-09-16T10:34:17.197214+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:17.197214+0000","last_undegraded":"2024-09-16T10:34:17.197214+0000","last_fullsized":"2024-09-16T10:34:17.197214+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.107835+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:02.107835+0000","last_peered":"2024-09-16T10:34:02.107835+0000","last_clean":"2024-09-16T10:34:02.107835+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:02.107835+0000","last_undegraded":"2024-09-16T10:34:02.107835+0000","last_fullsized":"2024-09-16T10:34:02.107835+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21326,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.206563+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:17.206563+0000","last_peered":"2024-09-16T10:34:17.206563+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:17.206563+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":101,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199411+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:17.199411+0000","last_peered":"2024-09-16T10:34:17.199411+0000","last_clean":"2024-09-16T10:34:17.199411+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:17.199411+0000","last_undegraded":"2024-09-16T10:34:17.199411+0000","last_fullsized":"2024-09-16T10:34:17.199411+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":20874,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.696425+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:17.696425+0000","last_peered":"2024-09-16T10:34:17.696425+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:17.696425+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-4109}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":81,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196738+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:17.196738+0000","last_peered":"2024-09-16T10:34:17.196738+0000","last_clean":"2024-09-16T10:34:17.196738+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:17.196738+0000","last_undegraded":"2024-09-16T10:34:17.196738+0000","last_fullsized":"2024-09-16T10:34:17.196738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29070,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.107650+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:02.107650+0000","last_peered":"2024-09-16T10:34:02.107650+0000","last_clean":"2024-09-16T10:34:02.107650+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:02.107650+0000","last_undegraded":"2024-09-16T10:34:02.107650+0000","last_fullsized":"2024-09-16T10:34:02.107650+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20295,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200596+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:17.200596+0000","last_peered":"2024-09-16T10:34:17.200596+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:17.200596+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197113+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:17.197113+0000","last_peered":"2024-09-16T10:34:17.197113+0000","last_clean":"2024-09-16T10:34:17.197113+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:17.197113+0000","last_undegraded":"2024-09-16T10:34:17.197113+0000","last_fullsized":"2024-09-16T10:34:17.197113+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21200,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200322+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:17.200322+0000","last_peered":"2024-09-16T10:34:17.200322+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:17.200322+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":69,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-16T10:34:02.109284+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:02.109284+0000","last_peered":"2024-09-16T10:34:02.109284+0000","last_clean":"2024-09-16T10:34:02.109284+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:02.109284+0000","last_undegraded":"2024-09-16T10:34:02.109284+0000","last_fullsized":"2024-09-16T10:34:02.109284+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197071+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:17.197071+0000","last_peered":"2024-09-16T10:34:17.197071+0000","last_clean":"2024-09-16T10:34:17.197071+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:17.197071+0000","last_undegraded":"2024-09-16T10:34:17.197071+0000","last_fullsized":"2024-09-16T10:34:17.197071+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21148,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200145+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:17.200145+0000","last_peered":"2024-09-16T10:34:17.200145+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:17.200145+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25936,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019022,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480196,"kb_used_data":137656,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891644,"statfs":{"total":96636764160,"available":96145043456,"internally_reserved":0,"allocated":140959744,"data_stored":140658718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.221},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.252}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23699999999999999},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.27000000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691966,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461168,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910672,"statfs":{"total":96636764160,"available":96164528128,"internally_reserved":0,"allocated":495616,"data_stored":154847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.156},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.185}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.26900000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215249,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17100000000000001},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.223},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:24.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:24.438 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:24.438 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:24.453 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-16T10:34:22.769485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26918,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464688,"kb_used_data":206596,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650832,"statfs":{"total":289910292480,"available":288410451968,"internally_reserved":0,"allocated":211554304,"data_stored":210604431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001412"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28943,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.200232+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:17.200232+0000","last_peered":"2024-09-16T10:34:17.200232+0000","last_clean":"2024-09-16T10:34:17.200232+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:17.200232+0000","last_undegraded":"2024-09-16T10:34:17.200232+0000","last_fullsized":"2024-09-16T10:34:17.200232+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199693+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:17.199693+0000","last_peered":"2024-09-16T10:34:17.199693+0000","last_clean":"2024-09-16T10:34:17.199693+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:17.199693+0000","last_undegraded":"2024-09-16T10:34:17.199693+0000","last_fullsized":"2024-09-16T10:34:17.199693+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.199768+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:17.199768+0000","last_peered":"2024-09-16T10:34:17.199768+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:17.199768+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199363+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:17.199363+0000","last_peered":"2024-09-16T10:34:17.199363+0000","last_clean":"2024-09-16T10:34:17.199363+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:17.199363+0000","last_undegraded":"2024-09-16T10:34:17.199363+0000","last_fullsized":"2024-09-16T10:34:17.199363+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28977,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197214+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:17.197214+0000","last_peered":"2024-09-16T10:34:17.197214+0000","last_clean":"2024-09-16T10:34:17.197214+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:17.197214+0000","last_undegraded":"2024-09-16T10:34:17.197214+0000","last_fullsized":"2024-09-16T10:34:17.197214+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21326,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.206563+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:17.206563+0000","last_peered":"2024-09-16T10:34:17.206563+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:17.206563+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":101,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199411+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:17.199411+0000","last_peered":"2024-09-16T10:34:17.199411+0000","last_clean":"2024-09-16T10:34:17.199411+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:17.199411+0000","last_undegraded":"2024-09-16T10:34:17.199411+0000","last_fullsized":"2024-09-16T10:34:17.199411+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":22837,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:22.695774+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:22.695774+0000","last_peered":"2024-09-16T10:34:22.695774+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:22.695774+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3128}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":81,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196738+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:17.196738+0000","last_peered":"2024-09-16T10:34:17.196738+0000","last_clean":"2024-09-16T10:34:17.196738+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:17.196738+0000","last_undegraded":"2024-09-16T10:34:17.196738+0000","last_fullsized":"2024-09-16T10:34:17.196738+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20295,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200596+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:17.200596+0000","last_peered":"2024-09-16T10:34:17.200596+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:17.200596+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197113+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:17.197113+0000","last_peered":"2024-09-16T10:34:17.197113+0000","last_clean":"2024-09-16T10:34:17.197113+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:17.197113+0000","last_undegraded":"2024-09-16T10:34:17.197113+0000","last_fullsized":"2024-09-16T10:34:17.197113+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21200,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200322+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:17.200322+0000","last_peered":"2024-09-16T10:34:17.200322+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:17.200322+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197071+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:17.197071+0000","last_peered":"2024-09-16T10:34:17.197071+0000","last_clean":"2024-09-16T10:34:17.197071+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:17.197071+0000","last_undegraded":"2024-09-16T10:34:17.197071+0000","last_fullsized":"2024-09-16T10:34:17.197071+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21148,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200145+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:17.200145+0000","last_peered":"2024-09-16T10:34:17.200145+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:17.200145+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26917,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019023,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.247}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.23300000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691966,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461168,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910672,"statfs":{"total":96636764160,"available":96164528128,"internally_reserved":0,"allocated":495616,"data_stored":154847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":3.5859999999999999,"5min":3.181,"15min":3.4430000000000001},"min":{"1min":0.26500000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":12.798,"5min":12.798,"15min":17.675000000000001},"last":0.156},{"interface":"front","average":{"1min":3.7839999999999998,"5min":2.887,"15min":2.8319999999999999},"min":{"1min":0.29299999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.0410000000000004,"5min":12.132,"15min":16.850999999999999},"last":0.185}]},{"osd":2,"last update":"Mon Sep 16 10:33:19 2024","interfaces":[{"interface":"back","average":{"1min":4.0919999999999996,"5min":2.891,"15min":2.6030000000000002},"min":{"1min":0.34300000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":14.089,"5min":18.638999999999999,"15min":18.638999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":3.3490000000000002,"5min":2.7879999999999998,"15min":2.8809999999999998},"min":{"1min":0.28100000000000003,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":10.656000000000001,"5min":11.074,"15min":11.412000000000001},"last":0.26900000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215249,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":2.6890000000000001,"15min":2.7450000000000001},"min":{"1min":0.20000000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":9.7560000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17100000000000001},{"interface":"front","average":{"1min":3.6720000000000002,"5min":2.7850000000000001,"15min":2.3849999999999998},"min":{"1min":0.30099999999999999,"5min":0.22600000000000001,"15min":0.22500000000000001},"max":{"1min":12.691000000000001,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:33:15 2024","interfaces":[{"interface":"back","average":{"1min":4.2690000000000001,"5min":2.911,"15min":1.9339999999999999},"min":{"1min":0.20300000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":15.534000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.223},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.4449999999999998,"15min":1.9399999999999999},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":8.2579999999999991,"5min":13.752000000000001,"15min":13.752000000000001},"last":0.24299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:24.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:24.811 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:24.812 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:24.824 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-16T10:34:24.769828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26918,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472884,"kb_used_data":206604,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281642636,"statfs":{"total":289910292480,"available":288402059264,"internally_reserved":0,"allocated":211562496,"data_stored":210607290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001418"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28943,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.200232+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:17.200232+0000","last_peered":"2024-09-16T10:34:17.200232+0000","last_clean":"2024-09-16T10:34:17.200232+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:17.200232+0000","last_undegraded":"2024-09-16T10:34:17.200232+0000","last_fullsized":"2024-09-16T10:34:17.200232+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199693+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:17.199693+0000","last_peered":"2024-09-16T10:34:17.199693+0000","last_clean":"2024-09-16T10:34:17.199693+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:17.199693+0000","last_undegraded":"2024-09-16T10:34:17.199693+0000","last_fullsized":"2024-09-16T10:34:17.199693+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.199768+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:17.199768+0000","last_peered":"2024-09-16T10:34:17.199768+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:17.199768+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199363+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:17.199363+0000","last_peered":"2024-09-16T10:34:17.199363+0000","last_clean":"2024-09-16T10:34:17.199363+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:17.199363+0000","last_undegraded":"2024-09-16T10:34:17.199363+0000","last_fullsized":"2024-09-16T10:34:17.199363+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28977,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197214+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:17.197214+0000","last_peered":"2024-09-16T10:34:17.197214+0000","last_clean":"2024-09-16T10:34:17.197214+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:17.197214+0000","last_undegraded":"2024-09-16T10:34:17.197214+0000","last_fullsized":"2024-09-16T10:34:17.197214+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21326,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.206563+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:17.206563+0000","last_peered":"2024-09-16T10:34:17.206563+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:17.206563+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":101,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199411+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:17.199411+0000","last_peered":"2024-09-16T10:34:17.199411+0000","last_clean":"2024-09-16T10:34:17.199411+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:17.199411+0000","last_undegraded":"2024-09-16T10:34:17.199411+0000","last_fullsized":"2024-09-16T10:34:17.199411+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":22837,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:22.695774+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:22.695774+0000","last_peered":"2024-09-16T10:34:22.695774+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:22.695774+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3128}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201878+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:18.201878+0000","last_peered":"2024-09-16T10:34:18.201878+0000","last_clean":"2024-09-16T10:34:18.201878+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:18.201878+0000","last_undegraded":"2024-09-16T10:34:18.201878+0000","last_fullsized":"2024-09-16T10:34:18.201878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20295,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200596+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:17.200596+0000","last_peered":"2024-09-16T10:34:17.200596+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:17.200596+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201982+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:18.201982+0000","last_peered":"2024-09-16T10:34:18.201982+0000","last_clean":"2024-09-16T10:34:18.201982+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:18.201982+0000","last_undegraded":"2024-09-16T10:34:18.201982+0000","last_fullsized":"2024-09-16T10:34:18.201982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21200,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200322+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:17.200322+0000","last_peered":"2024-09-16T10:34:17.200322+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:17.200322+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197071+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:17.197071+0000","last_peered":"2024-09-16T10:34:17.197071+0000","last_clean":"2024-09-16T10:34:17.197071+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:17.197071+0000","last_undegraded":"2024-09-16T10:34:17.197071+0000","last_fullsized":"2024-09-16T10:34:17.197071+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21148,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200145+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:17.200145+0000","last_peered":"2024-09-16T10:34:17.200145+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:17.200145+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26917,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019023,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.247}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.23300000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691967,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469364,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902476,"statfs":{"total":96636764160,"available":96156135424,"internally_reserved":0,"allocated":503808,"data_stored":157706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.23599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.219},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.27200000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215250,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:24.825 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:24.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21339, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.199768+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:17.199768+0000', 'last_peered': '2024-09-16T10:34:17.199768+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:17.199768+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:24.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:24.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21326, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.206563+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:17.206563+0000', 'last_peered': '2024-09-16T10:34:17.206563+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:17.206563+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:24.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:24.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 22837, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:22.695774+0000', 'last_change': '2024-09-16T10:34:17.012403+0000', 'last_active': '2024-09-16T10:34:22.695774+0000', 'last_peered': '2024-09-16T10:34:22.695774+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:34:22.695774+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3130, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1151, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -3128}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:24.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:24.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20295, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.200596+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:17.200596+0000', 'last_peered': '2024-09-16T10:34:17.200596+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:17.200596+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:24.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:24.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21200, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.200322+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:34:17.200322+0000', 'last_peered': '2024-09-16T10:34:17.200322+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:17.200322+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:24.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:24.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21148, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.200145+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:17.200145+0000', 'last_peered': '2024-09-16T10:34:17.200145+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:17.200145+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:24.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:25.288 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:25.288 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24429/40 objects degraded (61072.500%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":754,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":54,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1508233216,"bytes_avail":288402059264,"bytes_total":289910292480,"degraded_objects":24429,"degraded_total":40,"degraded_ratio":610.72500000000002,"recovering_objects_per_sec":255,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2045,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:25.301 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:25.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:25.644 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:25.644 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:25.656 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-16T10:34:24.769828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26918,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1472884,"kb_used_data":206604,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281642636,"statfs":{"total":289910292480,"available":288402059264,"internally_reserved":0,"allocated":211562496,"data_stored":210607290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001418"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28943,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.200232+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:17.200232+0000","last_peered":"2024-09-16T10:34:17.200232+0000","last_clean":"2024-09-16T10:34:17.200232+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:17.200232+0000","last_undegraded":"2024-09-16T10:34:17.200232+0000","last_fullsized":"2024-09-16T10:34:17.200232+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199693+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:17.199693+0000","last_peered":"2024-09-16T10:34:17.199693+0000","last_clean":"2024-09-16T10:34:17.199693+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:17.199693+0000","last_undegraded":"2024-09-16T10:34:17.199693+0000","last_fullsized":"2024-09-16T10:34:17.199693+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.199768+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:17.199768+0000","last_peered":"2024-09-16T10:34:17.199768+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:17.199768+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199363+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:17.199363+0000","last_peered":"2024-09-16T10:34:17.199363+0000","last_clean":"2024-09-16T10:34:17.199363+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:17.199363+0000","last_undegraded":"2024-09-16T10:34:17.199363+0000","last_fullsized":"2024-09-16T10:34:17.199363+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28977,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197214+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:17.197214+0000","last_peered":"2024-09-16T10:34:17.197214+0000","last_clean":"2024-09-16T10:34:17.197214+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:17.197214+0000","last_undegraded":"2024-09-16T10:34:17.197214+0000","last_fullsized":"2024-09-16T10:34:17.197214+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21326,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.206563+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:17.206563+0000","last_peered":"2024-09-16T10:34:17.206563+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:17.206563+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":101,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199411+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:17.199411+0000","last_peered":"2024-09-16T10:34:17.199411+0000","last_clean":"2024-09-16T10:34:17.199411+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:17.199411+0000","last_undegraded":"2024-09-16T10:34:17.199411+0000","last_fullsized":"2024-09-16T10:34:17.199411+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":22837,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:22.695774+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:22.695774+0000","last_peered":"2024-09-16T10:34:22.695774+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:22.695774+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1151,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-3128}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201878+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:18.201878+0000","last_peered":"2024-09-16T10:34:18.201878+0000","last_clean":"2024-09-16T10:34:18.201878+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:18.201878+0000","last_undegraded":"2024-09-16T10:34:18.201878+0000","last_fullsized":"2024-09-16T10:34:18.201878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20295,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200596+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:17.200596+0000","last_peered":"2024-09-16T10:34:17.200596+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:17.200596+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201982+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:18.201982+0000","last_peered":"2024-09-16T10:34:18.201982+0000","last_clean":"2024-09-16T10:34:18.201982+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:18.201982+0000","last_undegraded":"2024-09-16T10:34:18.201982+0000","last_fullsized":"2024-09-16T10:34:18.201982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21200,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200322+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:17.200322+0000","last_peered":"2024-09-16T10:34:17.200322+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:17.200322+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":67,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197071+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:17.197071+0000","last_peered":"2024-09-16T10:34:17.197071+0000","last_clean":"2024-09-16T10:34:17.197071+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:17.197071+0000","last_undegraded":"2024-09-16T10:34:17.197071+0000","last_fullsized":"2024-09-16T10:34:17.197071+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21148,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200145+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:17.200145+0000","last_peered":"2024-09-16T10:34:17.200145+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:17.200145+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24429,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26917,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019023,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.247}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20599999999999999},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.23300000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691967,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469364,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902476,"statfs":{"total":96636764160,"available":96156135424,"internally_reserved":0,"allocated":503808,"data_stored":157706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20399999999999999},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.23599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.219},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.27200000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215250,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:28.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:28.995 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:28.995 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:29.008 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-09-16T10:34:28.770604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28140,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464692,"kb_used_data":206604,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650828,"statfs":{"total":289910292480,"available":288410447872,"internally_reserved":0,"allocated":211562496,"data_stored":210607290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.571183"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28943,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.200232+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:17.200232+0000","last_peered":"2024-09-16T10:34:17.200232+0000","last_clean":"2024-09-16T10:34:17.200232+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:17.200232+0000","last_undegraded":"2024-09-16T10:34:17.200232+0000","last_fullsized":"2024-09-16T10:34:17.200232+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199693+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:17.199693+0000","last_peered":"2024-09-16T10:34:17.199693+0000","last_clean":"2024-09-16T10:34:17.199693+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:17.199693+0000","last_undegraded":"2024-09-16T10:34:17.199693+0000","last_fullsized":"2024-09-16T10:34:17.199693+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.199768+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:17.199768+0000","last_peered":"2024-09-16T10:34:17.199768+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:17.199768+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199363+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:17.199363+0000","last_peered":"2024-09-16T10:34:17.199363+0000","last_clean":"2024-09-16T10:34:17.199363+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:17.199363+0000","last_undegraded":"2024-09-16T10:34:17.199363+0000","last_fullsized":"2024-09-16T10:34:17.199363+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28978,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201909+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:18.201909+0000","last_peered":"2024-09-16T10:34:18.201909+0000","last_clean":"2024-09-16T10:34:18.201909+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:18.201909+0000","last_undegraded":"2024-09-16T10:34:18.201909+0000","last_fullsized":"2024-09-16T10:34:18.201909+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21326,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.206563+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:17.206563+0000","last_peered":"2024-09-16T10:34:17.206563+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:17.206563+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":101,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199411+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:17.199411+0000","last_peered":"2024-09-16T10:34:17.199411+0000","last_clean":"2024-09-16T10:34:17.199411+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:17.199411+0000","last_undegraded":"2024-09-16T10:34:17.199411+0000","last_fullsized":"2024-09-16T10:34:17.199411+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":25281,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:27.696579+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:27.696579+0000","last_peered":"2024-09-16T10:34:27.696579+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:27.696579+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2373,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1906}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201878+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:18.201878+0000","last_peered":"2024-09-16T10:34:18.201878+0000","last_clean":"2024-09-16T10:34:18.201878+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:18.201878+0000","last_undegraded":"2024-09-16T10:34:18.201878+0000","last_fullsized":"2024-09-16T10:34:18.201878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20295,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200596+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:17.200596+0000","last_peered":"2024-09-16T10:34:17.200596+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:17.200596+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201982+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:18.201982+0000","last_peered":"2024-09-16T10:34:18.201982+0000","last_clean":"2024-09-16T10:34:18.201982+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:18.201982+0000","last_undegraded":"2024-09-16T10:34:18.201982+0000","last_fullsized":"2024-09-16T10:34:18.201982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21200,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200322+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:17.200322+0000","last_peered":"2024-09-16T10:34:17.200322+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:17.200322+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201967+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:18.201967+0000","last_peered":"2024-09-16T10:34:18.201967+0000","last_clean":"2024-09-16T10:34:18.201967+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:18.201967+0000","last_undegraded":"2024-09-16T10:34:18.201967+0000","last_fullsized":"2024-09-16T10:34:18.201967+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21148,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200145+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:17.200145+0000","last_peered":"2024-09-16T10:34:17.200145+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:17.200145+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28139,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019024,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.26200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.248}]}]},{"osd":1,"up_from":40,"seq":171798691968,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461172,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910668,"statfs":{"total":96636764160,"available":96164524032,"internally_reserved":0,"allocated":503808,"data_stored":157706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.189},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.254}]}]},{"osd":0,"up_from":24,"seq":103079215250,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:29.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:29.367 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:29.368 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:29.381 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-09-16T10:34:28.770604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28140,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464692,"kb_used_data":206604,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650828,"statfs":{"total":289910292480,"available":288410447872,"internally_reserved":0,"allocated":211562496,"data_stored":210607290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.571183"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28943,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.200232+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:17.200232+0000","last_peered":"2024-09-16T10:34:17.200232+0000","last_clean":"2024-09-16T10:34:17.200232+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:17.200232+0000","last_undegraded":"2024-09-16T10:34:17.200232+0000","last_fullsized":"2024-09-16T10:34:17.200232+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199693+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:17.199693+0000","last_peered":"2024-09-16T10:34:17.199693+0000","last_clean":"2024-09-16T10:34:17.199693+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:17.199693+0000","last_undegraded":"2024-09-16T10:34:17.199693+0000","last_fullsized":"2024-09-16T10:34:17.199693+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.199768+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:17.199768+0000","last_peered":"2024-09-16T10:34:17.199768+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:17.199768+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199363+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:17.199363+0000","last_peered":"2024-09-16T10:34:17.199363+0000","last_clean":"2024-09-16T10:34:17.199363+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:17.199363+0000","last_undegraded":"2024-09-16T10:34:17.199363+0000","last_fullsized":"2024-09-16T10:34:17.199363+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28978,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201909+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:18.201909+0000","last_peered":"2024-09-16T10:34:18.201909+0000","last_clean":"2024-09-16T10:34:18.201909+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:18.201909+0000","last_undegraded":"2024-09-16T10:34:18.201909+0000","last_fullsized":"2024-09-16T10:34:18.201909+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21326,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.206563+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:17.206563+0000","last_peered":"2024-09-16T10:34:17.206563+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:17.206563+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":101,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199411+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:17.199411+0000","last_peered":"2024-09-16T10:34:17.199411+0000","last_clean":"2024-09-16T10:34:17.199411+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:17.199411+0000","last_undegraded":"2024-09-16T10:34:17.199411+0000","last_fullsized":"2024-09-16T10:34:17.199411+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":25281,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:27.696579+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:27.696579+0000","last_peered":"2024-09-16T10:34:27.696579+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:27.696579+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2373,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1906}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201878+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:18.201878+0000","last_peered":"2024-09-16T10:34:18.201878+0000","last_clean":"2024-09-16T10:34:18.201878+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:18.201878+0000","last_undegraded":"2024-09-16T10:34:18.201878+0000","last_fullsized":"2024-09-16T10:34:18.201878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20295,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200596+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:17.200596+0000","last_peered":"2024-09-16T10:34:17.200596+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:17.200596+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201982+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:18.201982+0000","last_peered":"2024-09-16T10:34:18.201982+0000","last_clean":"2024-09-16T10:34:18.201982+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:18.201982+0000","last_undegraded":"2024-09-16T10:34:18.201982+0000","last_fullsized":"2024-09-16T10:34:18.201982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21200,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200322+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:17.200322+0000","last_peered":"2024-09-16T10:34:17.200322+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:17.200322+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201967+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:18.201967+0000","last_peered":"2024-09-16T10:34:18.201967+0000","last_clean":"2024-09-16T10:34:18.201967+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:18.201967+0000","last_undegraded":"2024-09-16T10:34:18.201967+0000","last_fullsized":"2024-09-16T10:34:18.201967+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21148,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200145+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:17.200145+0000","last_peered":"2024-09-16T10:34:17.200145+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:17.200145+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28139,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019024,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.26200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.248}]}]},{"osd":1,"up_from":40,"seq":171798691968,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461172,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910668,"statfs":{"total":96636764160,"available":96164524032,"internally_reserved":0,"allocated":503808,"data_stored":157706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.189},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.254}]}]},{"osd":0,"up_from":24,"seq":103079215250,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:29.382 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:29.382 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21339, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.199768+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:17.199768+0000', 'last_peered': '2024-09-16T10:34:17.199768+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:17.199768+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:29.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:29.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21326, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.206563+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:17.206563+0000', 'last_peered': '2024-09-16T10:34:17.206563+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:17.206563+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:29.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:29.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 25281, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:27.696579+0000', 'last_change': '2024-09-16T10:34:17.012403+0000', 'last_active': '2024-09-16T10:34:27.696579+0000', 'last_peered': '2024-09-16T10:34:27.696579+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:34:27.696579+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1908, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2373, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -1906}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:29.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:29.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20295, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.200596+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:17.200596+0000', 'last_peered': '2024-09-16T10:34:17.200596+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:17.200596+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:29.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:29.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21200, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.200322+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:34:17.200322+0000', 'last_peered': '2024-09-16T10:34:17.200322+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:17.200322+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:29.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:29.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21148, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.200145+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:17.200145+0000', 'last_peered': '2024-09-16T10:34:17.200145+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:17.200145+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:29.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:29.801 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:29.802 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23207/40 objects degraded (58017.500%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":758,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":54,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1499844608,"bytes_avail":288410447872,"bytes_total":289910292480,"degraded_objects":23207,"degraded_total":40,"degraded_ratio":580.17499999999995,"recovering_objects_per_sec":297,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":3143,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:29.815 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:29.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:30.145 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:30.158 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-09-16T10:34:28.770604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28140,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464692,"kb_used_data":206604,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650828,"statfs":{"total":289910292480,"available":288410447872,"internally_reserved":0,"allocated":211562496,"data_stored":210607290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.571183"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28943,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.200232+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:17.200232+0000","last_peered":"2024-09-16T10:34:17.200232+0000","last_clean":"2024-09-16T10:34:17.200232+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:17.200232+0000","last_undegraded":"2024-09-16T10:34:17.200232+0000","last_fullsized":"2024-09-16T10:34:17.200232+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199693+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:17.199693+0000","last_peered":"2024-09-16T10:34:17.199693+0000","last_clean":"2024-09-16T10:34:17.199693+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:17.199693+0000","last_undegraded":"2024-09-16T10:34:17.199693+0000","last_fullsized":"2024-09-16T10:34:17.199693+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.199768+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:17.199768+0000","last_peered":"2024-09-16T10:34:17.199768+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:17.199768+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199363+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:17.199363+0000","last_peered":"2024-09-16T10:34:17.199363+0000","last_clean":"2024-09-16T10:34:17.199363+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:17.199363+0000","last_undegraded":"2024-09-16T10:34:17.199363+0000","last_fullsized":"2024-09-16T10:34:17.199363+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28978,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201909+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:18.201909+0000","last_peered":"2024-09-16T10:34:18.201909+0000","last_clean":"2024-09-16T10:34:18.201909+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:18.201909+0000","last_undegraded":"2024-09-16T10:34:18.201909+0000","last_fullsized":"2024-09-16T10:34:18.201909+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21326,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.206563+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:17.206563+0000","last_peered":"2024-09-16T10:34:17.206563+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:17.206563+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":101,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199411+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:17.199411+0000","last_peered":"2024-09-16T10:34:17.199411+0000","last_clean":"2024-09-16T10:34:17.199411+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:17.199411+0000","last_undegraded":"2024-09-16T10:34:17.199411+0000","last_fullsized":"2024-09-16T10:34:17.199411+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":25281,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:27.696579+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:27.696579+0000","last_peered":"2024-09-16T10:34:27.696579+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:27.696579+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1908,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2373,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-1906}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201878+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:18.201878+0000","last_peered":"2024-09-16T10:34:18.201878+0000","last_clean":"2024-09-16T10:34:18.201878+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:18.201878+0000","last_undegraded":"2024-09-16T10:34:18.201878+0000","last_fullsized":"2024-09-16T10:34:18.201878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20295,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200596+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:17.200596+0000","last_peered":"2024-09-16T10:34:17.200596+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:17.200596+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201982+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:18.201982+0000","last_peered":"2024-09-16T10:34:18.201982+0000","last_clean":"2024-09-16T10:34:18.201982+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:18.201982+0000","last_undegraded":"2024-09-16T10:34:18.201982+0000","last_fullsized":"2024-09-16T10:34:18.201982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21200,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200322+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:17.200322+0000","last_peered":"2024-09-16T10:34:17.200322+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:17.200322+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201967+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:18.201967+0000","last_peered":"2024-09-16T10:34:18.201967+0000","last_clean":"2024-09-16T10:34:18.201967+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:18.201967+0000","last_undegraded":"2024-09-16T10:34:18.201967+0000","last_fullsized":"2024-09-16T10:34:18.201967+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21148,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200145+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:17.200145+0000","last_peered":"2024-09-16T10:34:17.200145+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:17.200145+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28139,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019024,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.26200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.22500000000000001},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.248}]}]},{"osd":1,"up_from":40,"seq":171798691968,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461172,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910668,"statfs":{"total":96636764160,"available":96164524032,"internally_reserved":0,"allocated":503808,"data_stored":157706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.189},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.254}]}]},{"osd":0,"up_from":24,"seq":103079215250,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.25800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:33.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:33.480 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:33.481 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:33.494 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-09-16T10:34:32.771423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29711,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464692,"kb_used_data":206604,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650828,"statfs":{"total":289910292480,"available":288410447872,"internally_reserved":0,"allocated":211562496,"data_stored":210607290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3774,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002347"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28943,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.200232+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:17.200232+0000","last_peered":"2024-09-16T10:34:17.200232+0000","last_clean":"2024-09-16T10:34:17.200232+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:17.200232+0000","last_undegraded":"2024-09-16T10:34:17.200232+0000","last_fullsized":"2024-09-16T10:34:17.200232+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199693+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:17.199693+0000","last_peered":"2024-09-16T10:34:17.199693+0000","last_clean":"2024-09-16T10:34:17.199693+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:17.199693+0000","last_undegraded":"2024-09-16T10:34:17.199693+0000","last_fullsized":"2024-09-16T10:34:17.199693+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.199768+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:17.199768+0000","last_peered":"2024-09-16T10:34:17.199768+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:17.199768+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199363+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:17.199363+0000","last_peered":"2024-09-16T10:34:17.199363+0000","last_clean":"2024-09-16T10:34:17.199363+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:17.199363+0000","last_undegraded":"2024-09-16T10:34:17.199363+0000","last_fullsized":"2024-09-16T10:34:17.199363+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28978,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201909+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:18.201909+0000","last_peered":"2024-09-16T10:34:18.201909+0000","last_clean":"2024-09-16T10:34:18.201909+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:18.201909+0000","last_undegraded":"2024-09-16T10:34:18.201909+0000","last_fullsized":"2024-09-16T10:34:18.201909+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21326,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.206563+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:17.206563+0000","last_peered":"2024-09-16T10:34:17.206563+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:17.206563+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":101,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199411+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:17.199411+0000","last_peered":"2024-09-16T10:34:17.199411+0000","last_clean":"2024-09-16T10:34:17.199411+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:17.199411+0000","last_undegraded":"2024-09-16T10:34:17.199411+0000","last_fullsized":"2024-09-16T10:34:17.199411+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":28423,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:32.698191+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:32.698191+0000","last_peered":"2024-09-16T10:34:32.698191+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:32.698191+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-335}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201878+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:18.201878+0000","last_peered":"2024-09-16T10:34:18.201878+0000","last_clean":"2024-09-16T10:34:18.201878+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:18.201878+0000","last_undegraded":"2024-09-16T10:34:18.201878+0000","last_fullsized":"2024-09-16T10:34:18.201878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20295,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200596+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:17.200596+0000","last_peered":"2024-09-16T10:34:17.200596+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:17.200596+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201982+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:18.201982+0000","last_peered":"2024-09-16T10:34:18.201982+0000","last_clean":"2024-09-16T10:34:18.201982+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:18.201982+0000","last_undegraded":"2024-09-16T10:34:18.201982+0000","last_fullsized":"2024-09-16T10:34:18.201982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21200,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200322+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:17.200322+0000","last_peered":"2024-09-16T10:34:17.200322+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:17.200322+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201967+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:18.201967+0000","last_peered":"2024-09-16T10:34:18.201967+0000","last_clean":"2024-09-16T10:34:18.201967+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:18.201967+0000","last_undegraded":"2024-09-16T10:34:18.201967+0000","last_fullsized":"2024-09-16T10:34:18.201967+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21148,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200145+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:17.200145+0000","last_peered":"2024-09-16T10:34:17.200145+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:17.200145+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29710,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019025,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.188}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.249},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.21299999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691968,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461172,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910668,"statfs":{"total":96636764160,"available":96164524032,"internally_reserved":0,"allocated":503808,"data_stored":157706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.189},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.254}]}]},{"osd":0,"up_from":24,"seq":103079215251,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.22600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.247},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.308}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:33.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:33.838 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:33.855 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-09-16T10:34:32.771423+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29711,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1464692,"kb_used_data":206604,"kb_used_omap":12,"kb_used_meta":1258035,"kb_avail":281650828,"statfs":{"total":289910292480,"available":288410447872,"internally_reserved":0,"allocated":211562496,"data_stored":210607290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1288227842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3774,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002347"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28943,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.200232+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:17.200232+0000","last_peered":"2024-09-16T10:34:17.200232+0000","last_clean":"2024-09-16T10:34:17.200232+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:17.200232+0000","last_undegraded":"2024-09-16T10:34:17.200232+0000","last_fullsized":"2024-09-16T10:34:17.200232+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199693+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:17.199693+0000","last_peered":"2024-09-16T10:34:17.199693+0000","last_clean":"2024-09-16T10:34:17.199693+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:17.199693+0000","last_undegraded":"2024-09-16T10:34:17.199693+0000","last_fullsized":"2024-09-16T10:34:17.199693+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.199768+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:17.199768+0000","last_peered":"2024-09-16T10:34:17.199768+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:17.199768+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199363+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:17.199363+0000","last_peered":"2024-09-16T10:34:17.199363+0000","last_clean":"2024-09-16T10:34:17.199363+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:17.199363+0000","last_undegraded":"2024-09-16T10:34:17.199363+0000","last_fullsized":"2024-09-16T10:34:17.199363+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28978,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201909+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:18.201909+0000","last_peered":"2024-09-16T10:34:18.201909+0000","last_clean":"2024-09-16T10:34:18.201909+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:18.201909+0000","last_undegraded":"2024-09-16T10:34:18.201909+0000","last_fullsized":"2024-09-16T10:34:18.201909+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21326,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.206563+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:17.206563+0000","last_peered":"2024-09-16T10:34:17.206563+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:17.206563+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":101,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199411+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:17.199411+0000","last_peered":"2024-09-16T10:34:17.199411+0000","last_clean":"2024-09-16T10:34:17.199411+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:17.199411+0000","last_undegraded":"2024-09-16T10:34:17.199411+0000","last_fullsized":"2024-09-16T10:34:17.199411+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":28423,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:32.698191+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:32.698191+0000","last_peered":"2024-09-16T10:34:32.698191+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:32.698191+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-335}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201878+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:18.201878+0000","last_peered":"2024-09-16T10:34:18.201878+0000","last_clean":"2024-09-16T10:34:18.201878+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:18.201878+0000","last_undegraded":"2024-09-16T10:34:18.201878+0000","last_fullsized":"2024-09-16T10:34:18.201878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20295,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200596+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:17.200596+0000","last_peered":"2024-09-16T10:34:17.200596+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:17.200596+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201982+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:18.201982+0000","last_peered":"2024-09-16T10:34:18.201982+0000","last_clean":"2024-09-16T10:34:18.201982+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:18.201982+0000","last_undegraded":"2024-09-16T10:34:18.201982+0000","last_fullsized":"2024-09-16T10:34:18.201982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21200,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200322+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:17.200322+0000","last_peered":"2024-09-16T10:34:17.200322+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:17.200322+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201967+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:18.201967+0000","last_peered":"2024-09-16T10:34:18.201967+0000","last_clean":"2024-09-16T10:34:18.201967+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:18.201967+0000","last_undegraded":"2024-09-16T10:34:18.201967+0000","last_fullsized":"2024-09-16T10:34:18.201967+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21148,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200145+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:17.200145+0000","last_peered":"2024-09-16T10:34:17.200145+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:17.200145+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29710,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019025,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.188}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.249},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.21299999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691968,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461172,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93910668,"statfs":{"total":96636764160,"available":96164524032,"internally_reserved":0,"allocated":503808,"data_stored":157706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":471723690},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.189},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.254}]}]},{"osd":0,"up_from":24,"seq":103079215251,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.22600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.247},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.308}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:33.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:33.856 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21339, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.199768+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:17.199768+0000', 'last_peered': '2024-09-16T10:34:17.199768+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:17.199768+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:33.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:33.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21326, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.206563+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:17.206563+0000', 'last_peered': '2024-09-16T10:34:17.206563+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:17.206563+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:33.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:33.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 28423, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:32.698191+0000', 'last_change': '2024-09-16T10:34:17.012403+0000', 'last_active': '2024-09-16T10:34:32.698191+0000', 'last_peered': '2024-09-16T10:34:32.698191+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:34:32.698191+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3944, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -335}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:33.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:33.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20295, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.200596+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:17.200596+0000', 'last_peered': '2024-09-16T10:34:17.200596+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:17.200596+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:33.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:33.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21200, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.200322+0000', 'last_change': '2024-09-16T10:23:56.932500+0000', 'last_active': '2024-09-16T10:34:17.200322+0000', 'last_peered': '2024-09-16T10:34:17.200322+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:17.200322+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 73, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4264}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:33.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:33.857 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21148, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:17.200145+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:17.200145+0000', 'last_peered': '2024-09-16T10:34:17.200145+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:17.200145+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:33.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:34.303 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:34.303 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21636/40 objects degraded (54090.000%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":763,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":55,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1499844608,"bytes_avail":288410447872,"bytes_total":289910292480,"degraded_objects":21636,"degraded_total":40,"degraded_ratio":540.89999999999998,"recovering_objects_per_sec":314,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":3774,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:34.317 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:34.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:34.649 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:34.649 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:34.662 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-16T10:34:34.226802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29711,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481076,"kb_used_data":206604,"kb_used_omap":12,"kb_used_meta":1274419,"kb_avail":281634444,"statfs":{"total":289910292480,"available":288393670656,"internally_reserved":0,"allocated":211562496,"data_stored":210607290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1305005058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2793,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.457316"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28943,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.200232+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:17.200232+0000","last_peered":"2024-09-16T10:34:17.200232+0000","last_clean":"2024-09-16T10:34:17.200232+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:17.200232+0000","last_undegraded":"2024-09-16T10:34:17.200232+0000","last_fullsized":"2024-09-16T10:34:17.200232+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20675,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199693+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:17.199693+0000","last_peered":"2024-09-16T10:34:17.199693+0000","last_clean":"2024-09-16T10:34:17.199693+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:17.199693+0000","last_undegraded":"2024-09-16T10:34:17.199693+0000","last_fullsized":"2024-09-16T10:34:17.199693+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.199768+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:17.199768+0000","last_peered":"2024-09-16T10:34:17.199768+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:17.199768+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20578,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199363+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:17.199363+0000","last_peered":"2024-09-16T10:34:17.199363+0000","last_clean":"2024-09-16T10:34:17.199363+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:17.199363+0000","last_undegraded":"2024-09-16T10:34:17.199363+0000","last_fullsized":"2024-09-16T10:34:17.199363+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28978,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201909+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:18.201909+0000","last_peered":"2024-09-16T10:34:18.201909+0000","last_clean":"2024-09-16T10:34:18.201909+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:18.201909+0000","last_undegraded":"2024-09-16T10:34:18.201909+0000","last_fullsized":"2024-09-16T10:34:18.201909+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21326,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.206563+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:17.206563+0000","last_peered":"2024-09-16T10:34:17.206563+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:17.206563+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":101,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.199411+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:17.199411+0000","last_peered":"2024-09-16T10:34:17.199411+0000","last_clean":"2024-09-16T10:34:17.199411+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:17.199411+0000","last_undegraded":"2024-09-16T10:34:17.199411+0000","last_fullsized":"2024-09-16T10:34:17.199411+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":28423,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:32.698191+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:32.698191+0000","last_peered":"2024-09-16T10:34:32.698191+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:32.698191+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-335}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201878+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:18.201878+0000","last_peered":"2024-09-16T10:34:18.201878+0000","last_clean":"2024-09-16T10:34:18.201878+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:18.201878+0000","last_undegraded":"2024-09-16T10:34:18.201878+0000","last_fullsized":"2024-09-16T10:34:18.201878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20295,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200596+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:17.200596+0000","last_peered":"2024-09-16T10:34:17.200596+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:17.200596+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201982+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:18.201982+0000","last_peered":"2024-09-16T10:34:18.201982+0000","last_clean":"2024-09-16T10:34:18.201982+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:18.201982+0000","last_undegraded":"2024-09-16T10:34:18.201982+0000","last_fullsized":"2024-09-16T10:34:18.201982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21200,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200322+0000","last_change":"2024-09-16T10:23:56.932500+0000","last_active":"2024-09-16T10:34:17.200322+0000","last_peered":"2024-09-16T10:34:17.200322+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:17.200322+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4264}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201967+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:18.201967+0000","last_peered":"2024-09-16T10:34:18.201967+0000","last_clean":"2024-09-16T10:34:18.201967+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:18.201967+0000","last_undegraded":"2024-09-16T10:34:18.201967+0000","last_fullsized":"2024-09-16T10:34:18.201967+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21148,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:17.200145+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:17.200145+0000","last_peered":"2024-09-16T10:34:17.200145+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:17.200145+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21636,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29710,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019025,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.188}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.249},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.21299999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691969,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477556,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":477051,"kb_avail":93894284,"statfs":{"total":96636764160,"available":96147746816,"internally_reserved":0,"allocated":503808,"data_stored":157706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":488500906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.224}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.25800000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215251,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.22600000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.247},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.308}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:37.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:38.069 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:38.069 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:38.082 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-16T10:34:36.772191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29765,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481076,"kb_used_data":206604,"kb_used_omap":12,"kb_used_meta":1274419,"kb_avail":281634444,"statfs":{"total":289910292480,"available":288393670656,"internally_reserved":0,"allocated":211562496,"data_stored":210607290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1305005058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001586"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28944,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.204643+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:18.204643+0000","last_peered":"2024-09-16T10:34:18.204643+0000","last_clean":"2024-09-16T10:34:18.204643+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:18.204643+0000","last_undegraded":"2024-09-16T10:34:18.204643+0000","last_fullsized":"2024-09-16T10:34:18.204643+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20676,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.203523+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:18.203523+0000","last_peered":"2024-09-16T10:34:18.203523+0000","last_clean":"2024-09-16T10:34:18.203523+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:18.203523+0000","last_undegraded":"2024-09-16T10:34:18.203523+0000","last_fullsized":"2024-09-16T10:34:18.203523+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21340,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.203830+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:18.203830+0000","last_peered":"2024-09-16T10:34:18.203830+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:18.203830+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20579,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.203443+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:18.203443+0000","last_peered":"2024-09-16T10:34:18.203443+0000","last_clean":"2024-09-16T10:34:18.203443+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:18.203443+0000","last_undegraded":"2024-09-16T10:34:18.203443+0000","last_fullsized":"2024-09-16T10:34:18.203443+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28978,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201909+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:18.201909+0000","last_peered":"2024-09-16T10:34:18.201909+0000","last_clean":"2024-09-16T10:34:18.201909+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:18.201909+0000","last_undegraded":"2024-09-16T10:34:18.201909+0000","last_fullsized":"2024-09-16T10:34:18.201909+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21327,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.204043+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:18.204043+0000","last_peered":"2024-09-16T10:34:18.204043+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:18.204043+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":102,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.203551+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:18.203551+0000","last_peered":"2024-09-16T10:34:18.203551+0000","last_clean":"2024-09-16T10:34:18.203551+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:18.203551+0000","last_undegraded":"2024-09-16T10:34:18.203551+0000","last_fullsized":"2024-09-16T10:34:18.203551+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":28423,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:32.698191+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:32.698191+0000","last_peered":"2024-09-16T10:34:32.698191+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:32.698191+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-335}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201878+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:18.201878+0000","last_peered":"2024-09-16T10:34:18.201878+0000","last_clean":"2024-09-16T10:34:18.201878+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:18.201878+0000","last_undegraded":"2024-09-16T10:34:18.201878+0000","last_fullsized":"2024-09-16T10:34:18.201878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20296,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.204606+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:18.204606+0000","last_peered":"2024-09-16T10:34:18.204606+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:18.204606+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201982+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:18.201982+0000","last_peered":"2024-09-16T10:34:18.201982+0000","last_clean":"2024-09-16T10:34:18.201982+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:18.201982+0000","last_undegraded":"2024-09-16T10:34:18.201982+0000","last_fullsized":"2024-09-16T10:34:18.201982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21310,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.295400+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:34.295400+0000","last_peered":"2024-09-16T10:34:34.295400+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:34.295400+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201967+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:18.201967+0000","last_peered":"2024-09-16T10:34:18.201967+0000","last_clean":"2024-09-16T10:34:18.201967+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:18.201967+0000","last_undegraded":"2024-09-16T10:34:18.201967+0000","last_fullsized":"2024-09-16T10:34:18.201967+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21149,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.203928+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:18.203928+0000","last_peered":"2024-09-16T10:34:18.203928+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:18.203928+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29764,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019025,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.188}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.249},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.21299999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691969,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477556,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":477051,"kb_avail":93894284,"statfs":{"total":96636764160,"available":96147746816,"internally_reserved":0,"allocated":503808,"data_stored":157706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":488500906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.224}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.25800000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215252,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:38.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:38.430 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:38.430 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:38.444 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2024-09-16T10:34:36.772191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29765,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":42,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1481076,"kb_used_data":206604,"kb_used_omap":12,"kb_used_meta":1274419,"kb_avail":281634444,"statfs":{"total":289910292480,"available":288393670656,"internally_reserved":0,"allocated":211562496,"data_stored":210607290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1305005058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001586"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28944,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.204643+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:18.204643+0000","last_peered":"2024-09-16T10:34:18.204643+0000","last_clean":"2024-09-16T10:34:18.204643+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:18.204643+0000","last_undegraded":"2024-09-16T10:34:18.204643+0000","last_fullsized":"2024-09-16T10:34:18.204643+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20632,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197447+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:17.197447+0000","last_peered":"2024-09-16T10:34:17.197447+0000","last_clean":"2024-09-16T10:34:17.197447+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:17.197447+0000","last_undegraded":"2024-09-16T10:34:17.197447+0000","last_fullsized":"2024-09-16T10:34:17.197447+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20935,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196903+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:17.196903+0000","last_peered":"2024-09-16T10:34:17.196903+0000","last_clean":"2024-09-16T10:34:17.196903+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:17.196903+0000","last_undegraded":"2024-09-16T10:34:17.196903+0000","last_fullsized":"2024-09-16T10:34:17.196903+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20532,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197398+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:17.197398+0000","last_peered":"2024-09-16T10:34:17.197398+0000","last_clean":"2024-09-16T10:34:17.197398+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:17.197398+0000","last_undegraded":"2024-09-16T10:34:17.197398+0000","last_fullsized":"2024-09-16T10:34:17.197398+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20676,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.203523+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:18.203523+0000","last_peered":"2024-09-16T10:34:18.203523+0000","last_clean":"2024-09-16T10:34:18.203523+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:18.203523+0000","last_undegraded":"2024-09-16T10:34:18.203523+0000","last_fullsized":"2024-09-16T10:34:18.203523+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20714,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.285146+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:18.285146+0000","last_peered":"2024-09-16T10:34:18.285146+0000","last_clean":"2024-09-16T10:34:18.285146+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:18.285146+0000","last_undegraded":"2024-09-16T10:34:18.285146+0000","last_fullsized":"2024-09-16T10:34:18.285146+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21340,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.203830+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:18.203830+0000","last_peered":"2024-09-16T10:34:18.203830+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:18.203830+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20579,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.203443+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:18.203443+0000","last_peered":"2024-09-16T10:34:18.203443+0000","last_clean":"2024-09-16T10:34:18.203443+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:18.203443+0000","last_undegraded":"2024-09-16T10:34:18.203443+0000","last_fullsized":"2024-09-16T10:34:18.203443+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":68,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196894+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:17.196894+0000","last_peered":"2024-09-16T10:34:17.196894+0000","last_clean":"2024-09-16T10:34:17.196894+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:17.196894+0000","last_undegraded":"2024-09-16T10:34:17.196894+0000","last_fullsized":"2024-09-16T10:34:17.196894+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28978,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201909+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:18.201909+0000","last_peered":"2024-09-16T10:34:18.201909+0000","last_clean":"2024-09-16T10:34:18.201909+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:18.201909+0000","last_undegraded":"2024-09-16T10:34:18.201909+0000","last_fullsized":"2024-09-16T10:34:18.201909+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197945+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:17.197945+0000","last_peered":"2024-09-16T10:34:17.197945+0000","last_clean":"2024-09-16T10:34:17.197945+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:17.197945+0000","last_undegraded":"2024-09-16T10:34:17.197945+0000","last_fullsized":"2024-09-16T10:34:17.197945+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21327,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.204043+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:18.204043+0000","last_peered":"2024-09-16T10:34:18.204043+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:18.204043+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":102,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.203551+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:18.203551+0000","last_peered":"2024-09-16T10:34:18.203551+0000","last_clean":"2024-09-16T10:34:18.203551+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:18.203551+0000","last_undegraded":"2024-09-16T10:34:18.203551+0000","last_fullsized":"2024-09-16T10:34:18.203551+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":28423,"reported_epoch":54,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:32.698191+0000","last_change":"2024-09-16T10:34:17.012403+0000","last_active":"2024-09-16T10:34:32.698191+0000","last_peered":"2024-09-16T10:34:32.698191+0000","last_clean":"2024-09-16T10:23:54.351639+0000","last_became_active":"2024-09-16T10:23:56.900070+0000","last_became_peered":"2024-09-16T10:23:56.900070+0000","last_unstale":"2024-09-16T10:34:32.698191+0000","last_undegraded":"2024-09-16T10:23:56.407561+0000","last_fullsized":"2024-09-16T10:23:56.364487+0000","mapping_epoch":41,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3944,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","0"],"object_location_counts":[{"shards":"2","objects":-335}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":82,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201878+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:18.201878+0000","last_peered":"2024-09-16T10:34:18.201878+0000","last_clean":"2024-09-16T10:34:18.201878+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:18.201878+0000","last_undegraded":"2024-09-16T10:34:18.201878+0000","last_fullsized":"2024-09-16T10:34:18.201878+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29071,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196896+0000","last_change":"2024-09-16T10:31:29.359164+0000","last_active":"2024-09-16T10:34:17.196896+0000","last_peered":"2024-09-16T10:34:17.196896+0000","last_clean":"2024-09-16T10:34:17.196896+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:17.196896+0000","last_undegraded":"2024-09-16T10:34:17.196896+0000","last_fullsized":"2024-09-16T10:34:17.196896+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20296,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.204606+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:18.204606+0000","last_peered":"2024-09-16T10:34:18.204606+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:18.204606+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201982+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:18.201982+0000","last_peered":"2024-09-16T10:34:18.201982+0000","last_clean":"2024-09-16T10:34:18.201982+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:18.201982+0000","last_undegraded":"2024-09-16T10:34:18.201982+0000","last_fullsized":"2024-09-16T10:34:18.201982+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21310,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.295400+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:34.295400+0000","last_peered":"2024-09-16T10:34:34.295400+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:34.295400+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":70,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197592+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:17.197592+0000","last_peered":"2024-09-16T10:34:17.197592+0000","last_clean":"2024-09-16T10:34:17.197592+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:17.197592+0000","last_undegraded":"2024-09-16T10:34:17.197592+0000","last_fullsized":"2024-09-16T10:34:17.197592+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24509,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.196988+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:17.196988+0000","last_peered":"2024-09-16T10:34:17.196988+0000","last_clean":"2024-09-16T10:34:17.196988+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:17.196988+0000","last_undegraded":"2024-09-16T10:34:17.196988+0000","last_fullsized":"2024-09-16T10:34:17.196988+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":68,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.201967+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:18.201967+0000","last_peered":"2024-09-16T10:34:18.201967+0000","last_clean":"2024-09-16T10:34:18.201967+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:18.201967+0000","last_undegraded":"2024-09-16T10:34:18.201967+0000","last_fullsized":"2024-09-16T10:34:18.201967+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":76,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-16T10:34:17.197202+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:17.197202+0000","last_peered":"2024-09-16T10:34:17.197202+0000","last_clean":"2024-09-16T10:34:17.197202+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:17.197202+0000","last_undegraded":"2024-09-16T10:34:17.197202+0000","last_fullsized":"2024-09-16T10:34:17.197202+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21149,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.203928+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:18.203928+0000","last_peered":"2024-09-16T10:34:18.203928+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:18.203928+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29764,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210152042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":26,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019025,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480208,"kb_used_data":137664,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93891632,"statfs":{"total":96636764160,"available":96145031168,"internally_reserved":0,"allocated":140967936,"data_stored":140661577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.188}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.249},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.21299999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691969,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477556,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":477051,"kb_avail":93894284,"statfs":{"total":96636764160,"available":96147746816,"internally_reserved":0,"allocated":503808,"data_stored":157706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":488500906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.19500000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.224}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.25800000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215252,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":5796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140513280,"data_stored":140509667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:38.447 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:38.447 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21340, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:18.203830+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:18.203830+0000', 'last_peered': '2024-09-16T10:34:18.203830+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:18.203830+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:38.447 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:38.447 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21327, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:18.204043+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:18.204043+0000', 'last_peered': '2024-09-16T10:34:18.204043+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:18.204043+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:38.448 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:38.448 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 28423, 'reported_epoch': 54, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:32.698191+0000', 'last_change': '2024-09-16T10:34:17.012403+0000', 'last_active': '2024-09-16T10:34:32.698191+0000', 'last_peered': '2024-09-16T10:34:32.698191+0000', 'last_clean': '2024-09-16T10:23:54.351639+0000', 'last_became_active': '2024-09-16T10:23:56.900070+0000', 'last_became_peered': '2024-09-16T10:23:56.900070+0000', 'last_unstale': '2024-09-16T10:34:32.698191+0000', 'last_undegraded': '2024-09-16T10:23:56.407561+0000', 'last_fullsized': '2024-09-16T10:23:56.364487+0000', 'mapping_epoch': 41, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3944, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2], 'avail_no_missing': ['2', '0'], 'object_location_counts': [{'shards': '2', 'objects': -335}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:38.448 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:38.448 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20296, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:18.204606+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:18.204606+0000', 'last_peered': '2024-09-16T10:34:18.204606+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:18.204606+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:38.448 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:38.448 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 21310, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.295400+0000', 'last_change': '2024-09-16T10:34:33.677585+0000', 'last_active': '2024-09-16T10:34:34.295400+0000', 'last_peered': '2024-09-16T10:34:34.295400+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:34.295400+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4216, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 127, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4213}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:38.449 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:38.449 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21149, 'reported_epoch': 54, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:18.203928+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:18.203928+0000', 'last_peered': '2024-09-16T10:34:18.203928+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:18.203928+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:38.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:38.883 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:38.883 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21582/40 objects degraded (53955.000%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":767,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":56,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1516621824,"bytes_avail":288393670656,"bytes_total":289910292480,"degraded_objects":21582,"degraded_total":40,"degraded_ratio":539.54999999999995,"recovering_objects_per_sec":203,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1625,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:38.899 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:38.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:39.257 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:39.258 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:39.270 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-16T10:34:38.772549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30102,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1422596,"kb_used_data":156316,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281692924,"statfs":{"total":289910292480,"available":288453554176,"internally_reserved":0,"allocated":160067584,"data_stored":159105904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1962,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001501"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28944,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.204643+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:18.204643+0000","last_peered":"2024-09-16T10:34:18.204643+0000","last_clean":"2024-09-16T10:34:18.204643+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:18.204643+0000","last_undegraded":"2024-09-16T10:34:18.204643+0000","last_fullsized":"2024-09-16T10:34:18.204643+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20634,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231223+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:34.231223+0000","last_peered":"2024-09-16T10:34:34.231223+0000","last_clean":"2024-09-16T10:34:34.231223+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:34.231223+0000","last_undegraded":"2024-09-16T10:34:34.231223+0000","last_fullsized":"2024-09-16T10:34:34.231223+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20937,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231979+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:34.231979+0000","last_peered":"2024-09-16T10:34:34.231979+0000","last_clean":"2024-09-16T10:34:34.231979+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:34.231979+0000","last_undegraded":"2024-09-16T10:34:34.231979+0000","last_fullsized":"2024-09-16T10:34:34.231979+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20534,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231067+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:34.231067+0000","last_peered":"2024-09-16T10:34:34.231067+0000","last_clean":"2024-09-16T10:34:34.231067+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:34.231067+0000","last_undegraded":"2024-09-16T10:34:34.231067+0000","last_fullsized":"2024-09-16T10:34:34.231067+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20676,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.203523+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:18.203523+0000","last_peered":"2024-09-16T10:34:18.203523+0000","last_clean":"2024-09-16T10:34:18.203523+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:18.203523+0000","last_undegraded":"2024-09-16T10:34:18.203523+0000","last_fullsized":"2024-09-16T10:34:18.203523+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20716,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231781+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:34.231781+0000","last_peered":"2024-09-16T10:34:34.231781+0000","last_clean":"2024-09-16T10:34:34.231781+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:34.231781+0000","last_undegraded":"2024-09-16T10:34:34.231781+0000","last_fullsized":"2024-09-16T10:34:34.231781+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21340,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.203830+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:18.203830+0000","last_peered":"2024-09-16T10:34:18.203830+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:18.203830+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20579,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.203443+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:18.203443+0000","last_peered":"2024-09-16T10:34:18.203443+0000","last_clean":"2024-09-16T10:34:18.203443+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:18.203443+0000","last_undegraded":"2024-09-16T10:34:18.203443+0000","last_fullsized":"2024-09-16T10:34:18.203443+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231624+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:34.231624+0000","last_peered":"2024-09-16T10:34:34.231624+0000","last_clean":"2024-09-16T10:34:34.231624+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:34.231624+0000","last_undegraded":"2024-09-16T10:34:34.231624+0000","last_fullsized":"2024-09-16T10:34:34.231624+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28980,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231726+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:34.231726+0000","last_peered":"2024-09-16T10:34:34.231726+0000","last_clean":"2024-09-16T10:34:34.231726+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:34.231726+0000","last_undegraded":"2024-09-16T10:34:34.231726+0000","last_fullsized":"2024-09-16T10:34:34.231726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21327,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.204043+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:18.204043+0000","last_peered":"2024-09-16T10:34:18.204043+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:18.204043+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":102,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-16T10:34:18.203551+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:18.203551+0000","last_peered":"2024-09-16T10:34:18.203551+0000","last_clean":"2024-09-16T10:34:18.203551+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:18.203551+0000","last_undegraded":"2024-09-16T10:34:18.203551+0000","last_fullsized":"2024-09-16T10:34:18.203551+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29108,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.294036+0000","last_change":"2024-09-16T10:34:35.294036+0000","last_active":"2024-09-16T10:34:35.294036+0000","last_peered":"2024-09-16T10:34:35.294036+0000","last_clean":"2024-09-16T10:34:35.294036+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:35.294036+0000","last_undegraded":"2024-09-16T10:34:35.294036+0000","last_fullsized":"2024-09-16T10:34:35.294036+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231881+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:34.231881+0000","last_peered":"2024-09-16T10:34:34.231881+0000","last_clean":"2024-09-16T10:34:34.231881+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:34.231881+0000","last_undegraded":"2024-09-16T10:34:34.231881+0000","last_fullsized":"2024-09-16T10:34:34.231881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29074,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.431930+0000","last_change":"2024-09-16T10:34:35.233140+0000","last_active":"2024-09-16T10:34:34.431930+0000","last_peered":"2024-09-16T10:34:34.431930+0000","last_clean":"2024-09-16T10:34:34.431930+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:34.431930+0000","last_undegraded":"2024-09-16T10:34:34.431930+0000","last_fullsized":"2024-09-16T10:34:34.431930+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20296,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.204606+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:18.204606+0000","last_peered":"2024-09-16T10:34:18.204606+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:18.204606+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231927+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:34.231927+0000","last_peered":"2024-09-16T10:34:34.231927+0000","last_clean":"2024-09-16T10:34:34.231927+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:34.231927+0000","last_undegraded":"2024-09-16T10:34:34.231927+0000","last_fullsized":"2024-09-16T10:34:34.231927+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21310,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.295400+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:34.295400+0000","last_peered":"2024-09-16T10:34:34.295400+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:34.295400+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4216,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4213}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231221+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:34.231221+0000","last_peered":"2024-09-16T10:34:34.231221+0000","last_clean":"2024-09-16T10:34:34.231221+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:34.231221+0000","last_undegraded":"2024-09-16T10:34:34.231221+0000","last_fullsized":"2024-09-16T10:34:34.231221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24511,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232000+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:34.232000+0000","last_peered":"2024-09-16T10:34:34.232000+0000","last_clean":"2024-09-16T10:34:34.232000+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:34.232000+0000","last_undegraded":"2024-09-16T10:34:34.232000+0000","last_fullsized":"2024-09-16T10:34:34.232000+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232069+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:34.232069+0000","last_peered":"2024-09-16T10:34:34.232069+0000","last_clean":"2024-09-16T10:34:34.232069+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:34.232069+0000","last_undegraded":"2024-09-16T10:34:34.232069+0000","last_fullsized":"2024-09-16T10:34:34.232069+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21149,"reported_epoch":54,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:18.203928+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:18.203928+0000","last_peered":"2024-09-16T10:34:18.203928+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:18.203928+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30101,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158639297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019026,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429916,"kb_used_data":87372,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93941924,"statfs":{"total":96636764160,"available":96196530176,"internally_reserved":0,"allocated":89468928,"data_stored":89155951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.41099999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.27300000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691970,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469368,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902472,"statfs":{"total":96636764160,"available":96156131328,"internally_reserved":0,"allocated":507904,"data_stored":161946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.215},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215252,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523312,"kb_used_data":68448,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848528,"statfs":{"total":96636764160,"available":96100892672,"internally_reserved":0,"allocated":70090752,"data_stored":69788007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.18099999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.26500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89001984,"data_stored":88998371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:42.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:42.638 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:42.638 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:42.653 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":434,"stamp":"2024-09-16T10:34:40.772896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31026,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1422612,"kb_used_data":156332,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281692908,"statfs":{"total":289910292480,"available":288453537792,"internally_reserved":0,"allocated":160083968,"data_stored":159111593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1315,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001473"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28945,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.230332+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:34.230332+0000","last_peered":"2024-09-16T10:34:34.230332+0000","last_clean":"2024-09-16T10:34:34.230332+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:34.230332+0000","last_undegraded":"2024-09-16T10:34:34.230332+0000","last_fullsized":"2024-09-16T10:34:34.230332+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20634,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231223+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:34.231223+0000","last_peered":"2024-09-16T10:34:34.231223+0000","last_clean":"2024-09-16T10:34:34.231223+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:34.231223+0000","last_undegraded":"2024-09-16T10:34:34.231223+0000","last_fullsized":"2024-09-16T10:34:34.231223+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20937,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231979+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:34.231979+0000","last_peered":"2024-09-16T10:34:34.231979+0000","last_clean":"2024-09-16T10:34:34.231979+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:34.231979+0000","last_undegraded":"2024-09-16T10:34:34.231979+0000","last_fullsized":"2024-09-16T10:34:34.231979+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20534,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231067+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:34.231067+0000","last_peered":"2024-09-16T10:34:34.231067+0000","last_clean":"2024-09-16T10:34:34.231067+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:34.231067+0000","last_undegraded":"2024-09-16T10:34:34.231067+0000","last_fullsized":"2024-09-16T10:34:34.231067+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20677,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.229086+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:34.229086+0000","last_peered":"2024-09-16T10:34:34.229086+0000","last_clean":"2024-09-16T10:34:34.229086+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:34.229086+0000","last_undegraded":"2024-09-16T10:34:34.229086+0000","last_fullsized":"2024-09-16T10:34:34.229086+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20716,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231781+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:34.231781+0000","last_peered":"2024-09-16T10:34:34.231781+0000","last_clean":"2024-09-16T10:34:34.231781+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:34.231781+0000","last_undegraded":"2024-09-16T10:34:34.231781+0000","last_fullsized":"2024-09-16T10:34:34.231781+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21341,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229200+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:34.229200+0000","last_peered":"2024-09-16T10:34:34.229200+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:34.229200+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20580,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228121+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:34.228121+0000","last_peered":"2024-09-16T10:34:34.228121+0000","last_clean":"2024-09-16T10:34:34.228121+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:34.228121+0000","last_undegraded":"2024-09-16T10:34:34.228121+0000","last_fullsized":"2024-09-16T10:34:34.228121+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231624+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:34.231624+0000","last_peered":"2024-09-16T10:34:34.231624+0000","last_clean":"2024-09-16T10:34:34.231624+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:34.231624+0000","last_undegraded":"2024-09-16T10:34:34.231624+0000","last_fullsized":"2024-09-16T10:34:34.231624+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28980,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231726+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:34.231726+0000","last_peered":"2024-09-16T10:34:34.231726+0000","last_clean":"2024-09-16T10:34:34.231726+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:34.231726+0000","last_undegraded":"2024-09-16T10:34:34.231726+0000","last_fullsized":"2024-09-16T10:34:34.231726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21328,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229289+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:34.229289+0000","last_peered":"2024-09-16T10:34:34.229289+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:34.229289+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":103,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228206+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:34.228206+0000","last_peered":"2024-09-16T10:34:34.228206+0000","last_clean":"2024-09-16T10:34:34.228206+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:34.228206+0000","last_undegraded":"2024-09-16T10:34:34.228206+0000","last_fullsized":"2024-09-16T10:34:34.228206+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29108,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.294036+0000","last_change":"2024-09-16T10:34:35.294036+0000","last_active":"2024-09-16T10:34:35.294036+0000","last_peered":"2024-09-16T10:34:35.294036+0000","last_clean":"2024-09-16T10:34:35.294036+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:35.294036+0000","last_undegraded":"2024-09-16T10:34:35.294036+0000","last_fullsized":"2024-09-16T10:34:35.294036+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231881+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:34.231881+0000","last_peered":"2024-09-16T10:34:34.231881+0000","last_clean":"2024-09-16T10:34:34.231881+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:34.231881+0000","last_undegraded":"2024-09-16T10:34:34.231881+0000","last_fullsized":"2024-09-16T10:34:34.231881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29074,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.431930+0000","last_change":"2024-09-16T10:34:35.233140+0000","last_active":"2024-09-16T10:34:34.431930+0000","last_peered":"2024-09-16T10:34:34.431930+0000","last_clean":"2024-09-16T10:34:34.431930+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:34.431930+0000","last_undegraded":"2024-09-16T10:34:34.431930+0000","last_fullsized":"2024-09-16T10:34:34.431930+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20297,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229898+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:34.229898+0000","last_peered":"2024-09-16T10:34:34.229898+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:34.229898+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231927+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:34.231927+0000","last_peered":"2024-09-16T10:34:34.231927+0000","last_clean":"2024-09-16T10:34:34.231927+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:34.231927+0000","last_undegraded":"2024-09-16T10:34:34.231927+0000","last_fullsized":"2024-09-16T10:34:34.231927+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":23160,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:39.294139+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:39.294139+0000","last_peered":"2024-09-16T10:34:39.294139+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:39.294139+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231221+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:34.231221+0000","last_peered":"2024-09-16T10:34:34.231221+0000","last_clean":"2024-09-16T10:34:34.231221+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:34.231221+0000","last_undegraded":"2024-09-16T10:34:34.231221+0000","last_fullsized":"2024-09-16T10:34:34.231221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24511,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232000+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:34.232000+0000","last_peered":"2024-09-16T10:34:34.232000+0000","last_clean":"2024-09-16T10:34:34.232000+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:34.232000+0000","last_undegraded":"2024-09-16T10:34:34.232000+0000","last_fullsized":"2024-09-16T10:34:34.232000+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232069+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:34.232069+0000","last_peered":"2024-09-16T10:34:34.232069+0000","last_clean":"2024-09-16T10:34:34.232069+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:34.232069+0000","last_undegraded":"2024-09-16T10:34:34.232069+0000","last_fullsized":"2024-09-16T10:34:34.232069+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21150,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229518+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:34.229518+0000","last_peered":"2024-09-16T10:34:34.229518+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:34.229518+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31025,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158639297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019026,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429916,"kb_used_data":87372,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93941924,"statfs":{"total":96636764160,"available":96196530176,"internally_reserved":0,"allocated":89468928,"data_stored":89155951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.41099999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.27300000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691970,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469368,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902472,"statfs":{"total":96636764160,"available":96156131328,"internally_reserved":0,"allocated":507904,"data_stored":161946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.215},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215253,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523328,"kb_used_data":68464,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848512,"statfs":{"total":96636764160,"available":96100876288,"internally_reserved":0,"allocated":70107136,"data_stored":69793696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.25},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.36599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.21299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89001984,"data_stored":88998371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:42.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:42.981 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:42.981 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:42.995 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-16T10:34:42.773214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31026,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1422612,"kb_used_data":156332,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281692908,"statfs":{"total":289910292480,"available":288453537792,"internally_reserved":0,"allocated":160083968,"data_stored":159111593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1315,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.546412"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28945,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.230332+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:34.230332+0000","last_peered":"2024-09-16T10:34:34.230332+0000","last_clean":"2024-09-16T10:34:34.230332+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:34.230332+0000","last_undegraded":"2024-09-16T10:34:34.230332+0000","last_fullsized":"2024-09-16T10:34:34.230332+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20634,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231223+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:34.231223+0000","last_peered":"2024-09-16T10:34:34.231223+0000","last_clean":"2024-09-16T10:34:34.231223+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:34.231223+0000","last_undegraded":"2024-09-16T10:34:34.231223+0000","last_fullsized":"2024-09-16T10:34:34.231223+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20937,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231979+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:34.231979+0000","last_peered":"2024-09-16T10:34:34.231979+0000","last_clean":"2024-09-16T10:34:34.231979+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:34.231979+0000","last_undegraded":"2024-09-16T10:34:34.231979+0000","last_fullsized":"2024-09-16T10:34:34.231979+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20534,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231067+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:34.231067+0000","last_peered":"2024-09-16T10:34:34.231067+0000","last_clean":"2024-09-16T10:34:34.231067+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:34.231067+0000","last_undegraded":"2024-09-16T10:34:34.231067+0000","last_fullsized":"2024-09-16T10:34:34.231067+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20677,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.229086+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:34.229086+0000","last_peered":"2024-09-16T10:34:34.229086+0000","last_clean":"2024-09-16T10:34:34.229086+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:34.229086+0000","last_undegraded":"2024-09-16T10:34:34.229086+0000","last_fullsized":"2024-09-16T10:34:34.229086+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20716,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231781+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:34.231781+0000","last_peered":"2024-09-16T10:34:34.231781+0000","last_clean":"2024-09-16T10:34:34.231781+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:34.231781+0000","last_undegraded":"2024-09-16T10:34:34.231781+0000","last_fullsized":"2024-09-16T10:34:34.231781+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21341,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229200+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:34.229200+0000","last_peered":"2024-09-16T10:34:34.229200+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:34.229200+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20580,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228121+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:34.228121+0000","last_peered":"2024-09-16T10:34:34.228121+0000","last_clean":"2024-09-16T10:34:34.228121+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:34.228121+0000","last_undegraded":"2024-09-16T10:34:34.228121+0000","last_fullsized":"2024-09-16T10:34:34.228121+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231624+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:34.231624+0000","last_peered":"2024-09-16T10:34:34.231624+0000","last_clean":"2024-09-16T10:34:34.231624+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:34.231624+0000","last_undegraded":"2024-09-16T10:34:34.231624+0000","last_fullsized":"2024-09-16T10:34:34.231624+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28980,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231726+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:34.231726+0000","last_peered":"2024-09-16T10:34:34.231726+0000","last_clean":"2024-09-16T10:34:34.231726+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:34.231726+0000","last_undegraded":"2024-09-16T10:34:34.231726+0000","last_fullsized":"2024-09-16T10:34:34.231726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21328,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229289+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:34.229289+0000","last_peered":"2024-09-16T10:34:34.229289+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:34.229289+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":103,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228206+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:34.228206+0000","last_peered":"2024-09-16T10:34:34.228206+0000","last_clean":"2024-09-16T10:34:34.228206+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:34.228206+0000","last_undegraded":"2024-09-16T10:34:34.228206+0000","last_fullsized":"2024-09-16T10:34:34.228206+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29108,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.294036+0000","last_change":"2024-09-16T10:34:35.294036+0000","last_active":"2024-09-16T10:34:35.294036+0000","last_peered":"2024-09-16T10:34:35.294036+0000","last_clean":"2024-09-16T10:34:35.294036+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:35.294036+0000","last_undegraded":"2024-09-16T10:34:35.294036+0000","last_fullsized":"2024-09-16T10:34:35.294036+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231881+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:34.231881+0000","last_peered":"2024-09-16T10:34:34.231881+0000","last_clean":"2024-09-16T10:34:34.231881+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:34.231881+0000","last_undegraded":"2024-09-16T10:34:34.231881+0000","last_fullsized":"2024-09-16T10:34:34.231881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29074,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.431930+0000","last_change":"2024-09-16T10:34:35.233140+0000","last_active":"2024-09-16T10:34:34.431930+0000","last_peered":"2024-09-16T10:34:34.431930+0000","last_clean":"2024-09-16T10:34:34.431930+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:34.431930+0000","last_undegraded":"2024-09-16T10:34:34.431930+0000","last_fullsized":"2024-09-16T10:34:34.431930+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20297,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229898+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:34.229898+0000","last_peered":"2024-09-16T10:34:34.229898+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:34.229898+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231927+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:34.231927+0000","last_peered":"2024-09-16T10:34:34.231927+0000","last_clean":"2024-09-16T10:34:34.231927+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:34.231927+0000","last_undegraded":"2024-09-16T10:34:34.231927+0000","last_fullsized":"2024-09-16T10:34:34.231927+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":23160,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:39.294139+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:39.294139+0000","last_peered":"2024-09-16T10:34:39.294139+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:39.294139+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231221+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:34.231221+0000","last_peered":"2024-09-16T10:34:34.231221+0000","last_clean":"2024-09-16T10:34:34.231221+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:34.231221+0000","last_undegraded":"2024-09-16T10:34:34.231221+0000","last_fullsized":"2024-09-16T10:34:34.231221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24511,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232000+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:34.232000+0000","last_peered":"2024-09-16T10:34:34.232000+0000","last_clean":"2024-09-16T10:34:34.232000+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:34.232000+0000","last_undegraded":"2024-09-16T10:34:34.232000+0000","last_fullsized":"2024-09-16T10:34:34.232000+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232069+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:34.232069+0000","last_peered":"2024-09-16T10:34:34.232069+0000","last_clean":"2024-09-16T10:34:34.232069+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:34.232069+0000","last_undegraded":"2024-09-16T10:34:34.232069+0000","last_fullsized":"2024-09-16T10:34:34.232069+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21150,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229518+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:34.229518+0000","last_peered":"2024-09-16T10:34:34.229518+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:34.229518+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31025,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158639297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019027,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429916,"kb_used_data":87372,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93941924,"statfs":{"total":96636764160,"available":96196530176,"internally_reserved":0,"allocated":89468928,"data_stored":89155951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691970,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469368,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902472,"statfs":{"total":96636764160,"available":96156131328,"internally_reserved":0,"allocated":507904,"data_stored":161946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.215},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215253,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523328,"kb_used_data":68464,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848512,"statfs":{"total":96636764160,"available":96100876288,"internally_reserved":0,"allocated":70107136,"data_stored":69793696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.25},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.36599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.21299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89001984,"data_stored":88998371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:42.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:42.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21341, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229200+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:34.229200+0000', 'last_peered': '2024-09-16T10:34:34.229200+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:34.229200+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:42.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:42.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21328, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229289+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:34.229289+0000', 'last_peered': '2024-09-16T10:34:34.229289+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:34.229289+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:42.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:42.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20297, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229898+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:34.229898+0000', 'last_peered': '2024-09-16T10:34:34.229898+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:34.229898+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:42.997 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:42.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 23160, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:39.294139+0000', 'last_change': '2024-09-16T10:34:33.677585+0000', 'last_active': '2024-09-16T10:34:39.294139+0000', 'last_peered': '2024-09-16T10:34:39.294139+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:39.294139+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3292, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1051, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:42.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:42.998 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21150, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229518+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:34.229518+0000', 'last_peered': '2024-09-16T10:34:34.229518+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:34.229518+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:42.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:43.417 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 20321/40 objects degraded (50802.500%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":772,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":56,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1456754688,"bytes_avail":288453537792,"bytes_total":289910292480,"degraded_objects":20321,"degraded_total":40,"degraded_ratio":508.02499999999998,"recovering_objects_per_sec":153,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1315,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:43.432 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:43.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:43.759 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:43.760 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:43.773 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-16T10:34:42.773214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31026,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1422612,"kb_used_data":156332,"kb_used_omap":12,"kb_used_meta":1266227,"kb_avail":281692908,"statfs":{"total":289910292480,"available":288453537792,"internally_reserved":0,"allocated":160083968,"data_stored":159111593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1296616450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1315,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.546412"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28945,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.230332+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:34.230332+0000","last_peered":"2024-09-16T10:34:34.230332+0000","last_clean":"2024-09-16T10:34:34.230332+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:34.230332+0000","last_undegraded":"2024-09-16T10:34:34.230332+0000","last_fullsized":"2024-09-16T10:34:34.230332+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20634,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231223+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:34.231223+0000","last_peered":"2024-09-16T10:34:34.231223+0000","last_clean":"2024-09-16T10:34:34.231223+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:34.231223+0000","last_undegraded":"2024-09-16T10:34:34.231223+0000","last_fullsized":"2024-09-16T10:34:34.231223+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20937,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231979+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:34.231979+0000","last_peered":"2024-09-16T10:34:34.231979+0000","last_clean":"2024-09-16T10:34:34.231979+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:34.231979+0000","last_undegraded":"2024-09-16T10:34:34.231979+0000","last_fullsized":"2024-09-16T10:34:34.231979+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20534,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231067+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:34.231067+0000","last_peered":"2024-09-16T10:34:34.231067+0000","last_clean":"2024-09-16T10:34:34.231067+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:34.231067+0000","last_undegraded":"2024-09-16T10:34:34.231067+0000","last_fullsized":"2024-09-16T10:34:34.231067+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20677,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.229086+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:34.229086+0000","last_peered":"2024-09-16T10:34:34.229086+0000","last_clean":"2024-09-16T10:34:34.229086+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:34.229086+0000","last_undegraded":"2024-09-16T10:34:34.229086+0000","last_fullsized":"2024-09-16T10:34:34.229086+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20716,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231781+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:34.231781+0000","last_peered":"2024-09-16T10:34:34.231781+0000","last_clean":"2024-09-16T10:34:34.231781+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:34.231781+0000","last_undegraded":"2024-09-16T10:34:34.231781+0000","last_fullsized":"2024-09-16T10:34:34.231781+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21341,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229200+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:34.229200+0000","last_peered":"2024-09-16T10:34:34.229200+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:34.229200+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20580,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228121+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:34.228121+0000","last_peered":"2024-09-16T10:34:34.228121+0000","last_clean":"2024-09-16T10:34:34.228121+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:34.228121+0000","last_undegraded":"2024-09-16T10:34:34.228121+0000","last_fullsized":"2024-09-16T10:34:34.228121+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231624+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:34.231624+0000","last_peered":"2024-09-16T10:34:34.231624+0000","last_clean":"2024-09-16T10:34:34.231624+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:34.231624+0000","last_undegraded":"2024-09-16T10:34:34.231624+0000","last_fullsized":"2024-09-16T10:34:34.231624+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28980,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231726+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:34.231726+0000","last_peered":"2024-09-16T10:34:34.231726+0000","last_clean":"2024-09-16T10:34:34.231726+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:34.231726+0000","last_undegraded":"2024-09-16T10:34:34.231726+0000","last_fullsized":"2024-09-16T10:34:34.231726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21328,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229289+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:34.229289+0000","last_peered":"2024-09-16T10:34:34.229289+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:34.229289+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":103,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228206+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:34.228206+0000","last_peered":"2024-09-16T10:34:34.228206+0000","last_clean":"2024-09-16T10:34:34.228206+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:34.228206+0000","last_undegraded":"2024-09-16T10:34:34.228206+0000","last_fullsized":"2024-09-16T10:34:34.228206+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29108,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.294036+0000","last_change":"2024-09-16T10:34:35.294036+0000","last_active":"2024-09-16T10:34:35.294036+0000","last_peered":"2024-09-16T10:34:35.294036+0000","last_clean":"2024-09-16T10:34:35.294036+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:35.294036+0000","last_undegraded":"2024-09-16T10:34:35.294036+0000","last_fullsized":"2024-09-16T10:34:35.294036+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231881+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:34.231881+0000","last_peered":"2024-09-16T10:34:34.231881+0000","last_clean":"2024-09-16T10:34:34.231881+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:34.231881+0000","last_undegraded":"2024-09-16T10:34:34.231881+0000","last_fullsized":"2024-09-16T10:34:34.231881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29074,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.431930+0000","last_change":"2024-09-16T10:34:35.233140+0000","last_active":"2024-09-16T10:34:34.431930+0000","last_peered":"2024-09-16T10:34:34.431930+0000","last_clean":"2024-09-16T10:34:34.431930+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:34.431930+0000","last_undegraded":"2024-09-16T10:34:34.431930+0000","last_fullsized":"2024-09-16T10:34:34.431930+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20297,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229898+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:34.229898+0000","last_peered":"2024-09-16T10:34:34.229898+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:34.229898+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231927+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:34.231927+0000","last_peered":"2024-09-16T10:34:34.231927+0000","last_clean":"2024-09-16T10:34:34.231927+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:34.231927+0000","last_undegraded":"2024-09-16T10:34:34.231927+0000","last_fullsized":"2024-09-16T10:34:34.231927+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":23160,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:39.294139+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:39.294139+0000","last_peered":"2024-09-16T10:34:39.294139+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:39.294139+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1051,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231221+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:34.231221+0000","last_peered":"2024-09-16T10:34:34.231221+0000","last_clean":"2024-09-16T10:34:34.231221+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:34.231221+0000","last_undegraded":"2024-09-16T10:34:34.231221+0000","last_fullsized":"2024-09-16T10:34:34.231221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24511,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232000+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:34.232000+0000","last_peered":"2024-09-16T10:34:34.232000+0000","last_clean":"2024-09-16T10:34:34.232000+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:34.232000+0000","last_undegraded":"2024-09-16T10:34:34.232000+0000","last_fullsized":"2024-09-16T10:34:34.232000+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232069+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:34.232069+0000","last_peered":"2024-09-16T10:34:34.232069+0000","last_clean":"2024-09-16T10:34:34.232069+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:34.232069+0000","last_undegraded":"2024-09-16T10:34:34.232069+0000","last_fullsized":"2024-09-16T10:34:34.232069+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21150,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229518+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:34.229518+0000","last_peered":"2024-09-16T10:34:34.229518+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:34.229518+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31025,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158639297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019027,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429916,"kb_used_data":87372,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93941924,"statfs":{"total":96636764160,"available":96196530176,"internally_reserved":0,"allocated":89468928,"data_stored":89155951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691970,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":469368,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":468859,"kb_avail":93902472,"statfs":{"total":96636764160,"available":96156131328,"internally_reserved":0,"allocated":507904,"data_stored":161946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":480112298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.215},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215253,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523328,"kb_used_data":68464,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848512,"statfs":{"total":96636764160,"available":96100876288,"internally_reserved":0,"allocated":70107136,"data_stored":69793696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.25},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.36599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.21299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89001984,"data_stored":88998371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:46.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:47.109 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:47.121 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-16T10:34:46.773907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32186,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1430804,"kb_used_data":156332,"kb_used_omap":12,"kb_used_meta":1274419,"kb_avail":281684716,"statfs":{"total":289910292480,"available":288445149184,"internally_reserved":0,"allocated":160083968,"data_stored":159111593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1305005058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.543111"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28945,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.230332+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:34.230332+0000","last_peered":"2024-09-16T10:34:34.230332+0000","last_clean":"2024-09-16T10:34:34.230332+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:34.230332+0000","last_undegraded":"2024-09-16T10:34:34.230332+0000","last_fullsized":"2024-09-16T10:34:34.230332+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20634,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231223+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:34.231223+0000","last_peered":"2024-09-16T10:34:34.231223+0000","last_clean":"2024-09-16T10:34:34.231223+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:34.231223+0000","last_undegraded":"2024-09-16T10:34:34.231223+0000","last_fullsized":"2024-09-16T10:34:34.231223+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20937,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231979+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:34.231979+0000","last_peered":"2024-09-16T10:34:34.231979+0000","last_clean":"2024-09-16T10:34:34.231979+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:34.231979+0000","last_undegraded":"2024-09-16T10:34:34.231979+0000","last_fullsized":"2024-09-16T10:34:34.231979+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20534,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231067+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:34.231067+0000","last_peered":"2024-09-16T10:34:34.231067+0000","last_clean":"2024-09-16T10:34:34.231067+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:34.231067+0000","last_undegraded":"2024-09-16T10:34:34.231067+0000","last_fullsized":"2024-09-16T10:34:34.231067+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20677,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.229086+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:34.229086+0000","last_peered":"2024-09-16T10:34:34.229086+0000","last_clean":"2024-09-16T10:34:34.229086+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:34.229086+0000","last_undegraded":"2024-09-16T10:34:34.229086+0000","last_fullsized":"2024-09-16T10:34:34.229086+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20716,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231781+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:34.231781+0000","last_peered":"2024-09-16T10:34:34.231781+0000","last_clean":"2024-09-16T10:34:34.231781+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:34.231781+0000","last_undegraded":"2024-09-16T10:34:34.231781+0000","last_fullsized":"2024-09-16T10:34:34.231781+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21341,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229200+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:34.229200+0000","last_peered":"2024-09-16T10:34:34.229200+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:34.229200+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20580,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228121+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:34.228121+0000","last_peered":"2024-09-16T10:34:34.228121+0000","last_clean":"2024-09-16T10:34:34.228121+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:34.228121+0000","last_undegraded":"2024-09-16T10:34:34.228121+0000","last_fullsized":"2024-09-16T10:34:34.228121+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231624+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:34.231624+0000","last_peered":"2024-09-16T10:34:34.231624+0000","last_clean":"2024-09-16T10:34:34.231624+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:34.231624+0000","last_undegraded":"2024-09-16T10:34:34.231624+0000","last_fullsized":"2024-09-16T10:34:34.231624+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28980,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231726+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:34.231726+0000","last_peered":"2024-09-16T10:34:34.231726+0000","last_clean":"2024-09-16T10:34:34.231726+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:34.231726+0000","last_undegraded":"2024-09-16T10:34:34.231726+0000","last_fullsized":"2024-09-16T10:34:34.231726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21328,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229289+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:34.229289+0000","last_peered":"2024-09-16T10:34:34.229289+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:34.229289+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":103,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228206+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:34.228206+0000","last_peered":"2024-09-16T10:34:34.228206+0000","last_clean":"2024-09-16T10:34:34.228206+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:34.228206+0000","last_undegraded":"2024-09-16T10:34:34.228206+0000","last_fullsized":"2024-09-16T10:34:34.228206+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29108,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.294036+0000","last_change":"2024-09-16T10:34:35.294036+0000","last_active":"2024-09-16T10:34:35.294036+0000","last_peered":"2024-09-16T10:34:35.294036+0000","last_clean":"2024-09-16T10:34:35.294036+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:35.294036+0000","last_undegraded":"2024-09-16T10:34:35.294036+0000","last_fullsized":"2024-09-16T10:34:35.294036+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231881+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:34.231881+0000","last_peered":"2024-09-16T10:34:34.231881+0000","last_clean":"2024-09-16T10:34:34.231881+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:34.231881+0000","last_undegraded":"2024-09-16T10:34:34.231881+0000","last_fullsized":"2024-09-16T10:34:34.231881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29074,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.431930+0000","last_change":"2024-09-16T10:34:35.233140+0000","last_active":"2024-09-16T10:34:34.431930+0000","last_peered":"2024-09-16T10:34:34.431930+0000","last_clean":"2024-09-16T10:34:34.431930+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:34.431930+0000","last_undegraded":"2024-09-16T10:34:34.431930+0000","last_fullsized":"2024-09-16T10:34:34.431930+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20297,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229898+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:34.229898+0000","last_peered":"2024-09-16T10:34:34.229898+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:34.229898+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231927+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:34.231927+0000","last_peered":"2024-09-16T10:34:34.231927+0000","last_clean":"2024-09-16T10:34:34.231927+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:34.231927+0000","last_undegraded":"2024-09-16T10:34:34.231927+0000","last_fullsized":"2024-09-16T10:34:34.231927+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":25480,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:44.293956+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:44.293956+0000","last_peered":"2024-09-16T10:34:44.293956+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:44.293956+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2211,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2129}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231221+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:34.231221+0000","last_peered":"2024-09-16T10:34:34.231221+0000","last_clean":"2024-09-16T10:34:34.231221+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:34.231221+0000","last_undegraded":"2024-09-16T10:34:34.231221+0000","last_fullsized":"2024-09-16T10:34:34.231221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24511,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232000+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:34.232000+0000","last_peered":"2024-09-16T10:34:34.232000+0000","last_clean":"2024-09-16T10:34:34.232000+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:34.232000+0000","last_undegraded":"2024-09-16T10:34:34.232000+0000","last_fullsized":"2024-09-16T10:34:34.232000+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232069+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:34.232069+0000","last_peered":"2024-09-16T10:34:34.232069+0000","last_clean":"2024-09-16T10:34:34.232069+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:34.232069+0000","last_undegraded":"2024-09-16T10:34:34.232069+0000","last_fullsized":"2024-09-16T10:34:34.232069+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21150,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229518+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:34.229518+0000","last_peered":"2024-09-16T10:34:34.229518+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:34.229518+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32185,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158639297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019027,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429916,"kb_used_data":87372,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93941924,"statfs":{"total":96636764160,"available":96196530176,"internally_reserved":0,"allocated":89468928,"data_stored":89155951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691971,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477560,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":477051,"kb_avail":93894280,"statfs":{"total":96636764160,"available":96147742720,"internally_reserved":0,"allocated":507904,"data_stored":161946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":488500906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.188},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.26900000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523328,"kb_used_data":68464,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848512,"statfs":{"total":96636764160,"available":96100876288,"internally_reserved":0,"allocated":70107136,"data_stored":69793696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.182}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.221},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.19800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89001984,"data_stored":88998371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:47.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:47.452 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:47.452 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:47.465 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-16T10:34:46.773907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32186,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1430804,"kb_used_data":156332,"kb_used_omap":12,"kb_used_meta":1274419,"kb_avail":281684716,"statfs":{"total":289910292480,"available":288445149184,"internally_reserved":0,"allocated":160083968,"data_stored":159111593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1305005058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.543111"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28945,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.230332+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:34.230332+0000","last_peered":"2024-09-16T10:34:34.230332+0000","last_clean":"2024-09-16T10:34:34.230332+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:34.230332+0000","last_undegraded":"2024-09-16T10:34:34.230332+0000","last_fullsized":"2024-09-16T10:34:34.230332+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20634,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231223+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:34.231223+0000","last_peered":"2024-09-16T10:34:34.231223+0000","last_clean":"2024-09-16T10:34:34.231223+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:34.231223+0000","last_undegraded":"2024-09-16T10:34:34.231223+0000","last_fullsized":"2024-09-16T10:34:34.231223+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20937,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231979+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:34.231979+0000","last_peered":"2024-09-16T10:34:34.231979+0000","last_clean":"2024-09-16T10:34:34.231979+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:34.231979+0000","last_undegraded":"2024-09-16T10:34:34.231979+0000","last_fullsized":"2024-09-16T10:34:34.231979+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20534,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231067+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:34.231067+0000","last_peered":"2024-09-16T10:34:34.231067+0000","last_clean":"2024-09-16T10:34:34.231067+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:34.231067+0000","last_undegraded":"2024-09-16T10:34:34.231067+0000","last_fullsized":"2024-09-16T10:34:34.231067+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20677,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.229086+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:34.229086+0000","last_peered":"2024-09-16T10:34:34.229086+0000","last_clean":"2024-09-16T10:34:34.229086+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:34.229086+0000","last_undegraded":"2024-09-16T10:34:34.229086+0000","last_fullsized":"2024-09-16T10:34:34.229086+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20716,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231781+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:34.231781+0000","last_peered":"2024-09-16T10:34:34.231781+0000","last_clean":"2024-09-16T10:34:34.231781+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:34.231781+0000","last_undegraded":"2024-09-16T10:34:34.231781+0000","last_fullsized":"2024-09-16T10:34:34.231781+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21341,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229200+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:34.229200+0000","last_peered":"2024-09-16T10:34:34.229200+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:34.229200+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20580,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228121+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:34.228121+0000","last_peered":"2024-09-16T10:34:34.228121+0000","last_clean":"2024-09-16T10:34:34.228121+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:34.228121+0000","last_undegraded":"2024-09-16T10:34:34.228121+0000","last_fullsized":"2024-09-16T10:34:34.228121+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231624+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:34.231624+0000","last_peered":"2024-09-16T10:34:34.231624+0000","last_clean":"2024-09-16T10:34:34.231624+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:34.231624+0000","last_undegraded":"2024-09-16T10:34:34.231624+0000","last_fullsized":"2024-09-16T10:34:34.231624+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28980,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231726+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:34.231726+0000","last_peered":"2024-09-16T10:34:34.231726+0000","last_clean":"2024-09-16T10:34:34.231726+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:34.231726+0000","last_undegraded":"2024-09-16T10:34:34.231726+0000","last_fullsized":"2024-09-16T10:34:34.231726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21328,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229289+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:34.229289+0000","last_peered":"2024-09-16T10:34:34.229289+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:34.229289+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":103,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228206+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:34.228206+0000","last_peered":"2024-09-16T10:34:34.228206+0000","last_clean":"2024-09-16T10:34:34.228206+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:34.228206+0000","last_undegraded":"2024-09-16T10:34:34.228206+0000","last_fullsized":"2024-09-16T10:34:34.228206+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29108,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.294036+0000","last_change":"2024-09-16T10:34:35.294036+0000","last_active":"2024-09-16T10:34:35.294036+0000","last_peered":"2024-09-16T10:34:35.294036+0000","last_clean":"2024-09-16T10:34:35.294036+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:35.294036+0000","last_undegraded":"2024-09-16T10:34:35.294036+0000","last_fullsized":"2024-09-16T10:34:35.294036+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231881+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:34.231881+0000","last_peered":"2024-09-16T10:34:34.231881+0000","last_clean":"2024-09-16T10:34:34.231881+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:34.231881+0000","last_undegraded":"2024-09-16T10:34:34.231881+0000","last_fullsized":"2024-09-16T10:34:34.231881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29074,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.431930+0000","last_change":"2024-09-16T10:34:35.233140+0000","last_active":"2024-09-16T10:34:34.431930+0000","last_peered":"2024-09-16T10:34:34.431930+0000","last_clean":"2024-09-16T10:34:34.431930+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:34.431930+0000","last_undegraded":"2024-09-16T10:34:34.431930+0000","last_fullsized":"2024-09-16T10:34:34.431930+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20297,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229898+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:34.229898+0000","last_peered":"2024-09-16T10:34:34.229898+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:34.229898+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231927+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:34.231927+0000","last_peered":"2024-09-16T10:34:34.231927+0000","last_clean":"2024-09-16T10:34:34.231927+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:34.231927+0000","last_undegraded":"2024-09-16T10:34:34.231927+0000","last_fullsized":"2024-09-16T10:34:34.231927+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":25480,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:44.293956+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:44.293956+0000","last_peered":"2024-09-16T10:34:44.293956+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:44.293956+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2211,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2129}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231221+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:34.231221+0000","last_peered":"2024-09-16T10:34:34.231221+0000","last_clean":"2024-09-16T10:34:34.231221+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:34.231221+0000","last_undegraded":"2024-09-16T10:34:34.231221+0000","last_fullsized":"2024-09-16T10:34:34.231221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24511,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232000+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:34.232000+0000","last_peered":"2024-09-16T10:34:34.232000+0000","last_clean":"2024-09-16T10:34:34.232000+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:34.232000+0000","last_undegraded":"2024-09-16T10:34:34.232000+0000","last_fullsized":"2024-09-16T10:34:34.232000+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232069+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:34.232069+0000","last_peered":"2024-09-16T10:34:34.232069+0000","last_clean":"2024-09-16T10:34:34.232069+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:34.232069+0000","last_undegraded":"2024-09-16T10:34:34.232069+0000","last_fullsized":"2024-09-16T10:34:34.232069+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21150,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229518+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:34.229518+0000","last_peered":"2024-09-16T10:34:34.229518+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:34.229518+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32185,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158639297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019027,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429916,"kb_used_data":87372,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93941924,"statfs":{"total":96636764160,"available":96196530176,"internally_reserved":0,"allocated":89468928,"data_stored":89155951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691971,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477560,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":477051,"kb_avail":93894280,"statfs":{"total":96636764160,"available":96147742720,"internally_reserved":0,"allocated":507904,"data_stored":161946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":488500906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.188},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.26900000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523328,"kb_used_data":68464,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848512,"statfs":{"total":96636764160,"available":96100876288,"internally_reserved":0,"allocated":70107136,"data_stored":69793696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.182}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.221},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.19800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89001984,"data_stored":88998371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:47.466 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:47.466 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21341, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229200+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:34.229200+0000', 'last_peered': '2024-09-16T10:34:34.229200+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:34.229200+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:47.466 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:47.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21328, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229289+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:34.229289+0000', 'last_peered': '2024-09-16T10:34:34.229289+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:34.229289+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:47.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:47.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20297, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229898+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:34.229898+0000', 'last_peered': '2024-09-16T10:34:34.229898+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:34.229898+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:47.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:47.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 25480, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:44.293956+0000', 'last_change': '2024-09-16T10:34:33.677585+0000', 'last_active': '2024-09-16T10:34:44.293956+0000', 'last_peered': '2024-09-16T10:34:44.293956+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:44.293956+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2211, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2129}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:47.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:47.467 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21150, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229518+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:34.229518+0000', 'last_peered': '2024-09-16T10:34:34.229518+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:34.229518+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:47.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:47.884 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:47.884 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19161/40 objects degraded (47902.500%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":776,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":56,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1465143296,"bytes_avail":288445149184,"bytes_total":289910292480,"degraded_objects":19161,"degraded_total":40,"degraded_ratio":479.02499999999998,"recovering_objects_per_sec":209,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2421,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:47.898 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:47.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:48.237 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:48.238 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:48.250 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-16T10:34:46.773907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32186,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1430804,"kb_used_data":156332,"kb_used_omap":12,"kb_used_meta":1274419,"kb_avail":281684716,"statfs":{"total":289910292480,"available":288445149184,"internally_reserved":0,"allocated":160083968,"data_stored":159111593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1305005058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2421,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.543111"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28945,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.230332+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:34.230332+0000","last_peered":"2024-09-16T10:34:34.230332+0000","last_clean":"2024-09-16T10:34:34.230332+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:34.230332+0000","last_undegraded":"2024-09-16T10:34:34.230332+0000","last_fullsized":"2024-09-16T10:34:34.230332+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20634,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231223+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:34.231223+0000","last_peered":"2024-09-16T10:34:34.231223+0000","last_clean":"2024-09-16T10:34:34.231223+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:34.231223+0000","last_undegraded":"2024-09-16T10:34:34.231223+0000","last_fullsized":"2024-09-16T10:34:34.231223+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20937,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231979+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:34.231979+0000","last_peered":"2024-09-16T10:34:34.231979+0000","last_clean":"2024-09-16T10:34:34.231979+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:34.231979+0000","last_undegraded":"2024-09-16T10:34:34.231979+0000","last_fullsized":"2024-09-16T10:34:34.231979+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20534,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231067+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:34.231067+0000","last_peered":"2024-09-16T10:34:34.231067+0000","last_clean":"2024-09-16T10:34:34.231067+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:34.231067+0000","last_undegraded":"2024-09-16T10:34:34.231067+0000","last_fullsized":"2024-09-16T10:34:34.231067+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20677,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.229086+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:34.229086+0000","last_peered":"2024-09-16T10:34:34.229086+0000","last_clean":"2024-09-16T10:34:34.229086+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:34.229086+0000","last_undegraded":"2024-09-16T10:34:34.229086+0000","last_fullsized":"2024-09-16T10:34:34.229086+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20716,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231781+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:34.231781+0000","last_peered":"2024-09-16T10:34:34.231781+0000","last_clean":"2024-09-16T10:34:34.231781+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:34.231781+0000","last_undegraded":"2024-09-16T10:34:34.231781+0000","last_fullsized":"2024-09-16T10:34:34.231781+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21341,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229200+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:34.229200+0000","last_peered":"2024-09-16T10:34:34.229200+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:34.229200+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20580,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228121+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:34.228121+0000","last_peered":"2024-09-16T10:34:34.228121+0000","last_clean":"2024-09-16T10:34:34.228121+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:34.228121+0000","last_undegraded":"2024-09-16T10:34:34.228121+0000","last_fullsized":"2024-09-16T10:34:34.228121+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231624+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:34.231624+0000","last_peered":"2024-09-16T10:34:34.231624+0000","last_clean":"2024-09-16T10:34:34.231624+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:34.231624+0000","last_undegraded":"2024-09-16T10:34:34.231624+0000","last_fullsized":"2024-09-16T10:34:34.231624+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28980,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231726+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:34.231726+0000","last_peered":"2024-09-16T10:34:34.231726+0000","last_clean":"2024-09-16T10:34:34.231726+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:34.231726+0000","last_undegraded":"2024-09-16T10:34:34.231726+0000","last_fullsized":"2024-09-16T10:34:34.231726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21328,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229289+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:34.229289+0000","last_peered":"2024-09-16T10:34:34.229289+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:34.229289+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":103,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228206+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:34.228206+0000","last_peered":"2024-09-16T10:34:34.228206+0000","last_clean":"2024-09-16T10:34:34.228206+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:34.228206+0000","last_undegraded":"2024-09-16T10:34:34.228206+0000","last_fullsized":"2024-09-16T10:34:34.228206+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29108,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.294036+0000","last_change":"2024-09-16T10:34:35.294036+0000","last_active":"2024-09-16T10:34:35.294036+0000","last_peered":"2024-09-16T10:34:35.294036+0000","last_clean":"2024-09-16T10:34:35.294036+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:35.294036+0000","last_undegraded":"2024-09-16T10:34:35.294036+0000","last_fullsized":"2024-09-16T10:34:35.294036+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231881+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:34.231881+0000","last_peered":"2024-09-16T10:34:34.231881+0000","last_clean":"2024-09-16T10:34:34.231881+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:34.231881+0000","last_undegraded":"2024-09-16T10:34:34.231881+0000","last_fullsized":"2024-09-16T10:34:34.231881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29074,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.431930+0000","last_change":"2024-09-16T10:34:35.233140+0000","last_active":"2024-09-16T10:34:34.431930+0000","last_peered":"2024-09-16T10:34:34.431930+0000","last_clean":"2024-09-16T10:34:34.431930+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:34.431930+0000","last_undegraded":"2024-09-16T10:34:34.431930+0000","last_fullsized":"2024-09-16T10:34:34.431930+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20297,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229898+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:34.229898+0000","last_peered":"2024-09-16T10:34:34.229898+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:34.229898+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231927+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:34.231927+0000","last_peered":"2024-09-16T10:34:34.231927+0000","last_clean":"2024-09-16T10:34:34.231927+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:34.231927+0000","last_undegraded":"2024-09-16T10:34:34.231927+0000","last_fullsized":"2024-09-16T10:34:34.231927+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":25480,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:44.293956+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:44.293956+0000","last_peered":"2024-09-16T10:34:44.293956+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:44.293956+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2211,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2129}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231221+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:34.231221+0000","last_peered":"2024-09-16T10:34:34.231221+0000","last_clean":"2024-09-16T10:34:34.231221+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:34.231221+0000","last_undegraded":"2024-09-16T10:34:34.231221+0000","last_fullsized":"2024-09-16T10:34:34.231221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24511,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232000+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:34.232000+0000","last_peered":"2024-09-16T10:34:34.232000+0000","last_clean":"2024-09-16T10:34:34.232000+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:34.232000+0000","last_undegraded":"2024-09-16T10:34:34.232000+0000","last_fullsized":"2024-09-16T10:34:34.232000+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232069+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:34.232069+0000","last_peered":"2024-09-16T10:34:34.232069+0000","last_clean":"2024-09-16T10:34:34.232069+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:34.232069+0000","last_undegraded":"2024-09-16T10:34:34.232069+0000","last_fullsized":"2024-09-16T10:34:34.232069+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":78,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21150,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229518+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:34.229518+0000","last_peered":"2024-09-16T10:34:34.229518+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:34.229518+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32185,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158639297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019027,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429916,"kb_used_data":87372,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93941924,"statfs":{"total":96636764160,"available":96196530176,"internally_reserved":0,"allocated":89468928,"data_stored":89155951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.28299999999999997}]}]},{"osd":1,"up_from":40,"seq":171798691971,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477560,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":477051,"kb_avail":93894280,"statfs":{"total":96636764160,"available":96147742720,"internally_reserved":0,"allocated":507904,"data_stored":161946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":488500906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.188},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.26900000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215254,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523328,"kb_used_data":68464,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848512,"statfs":{"total":96636764160,"available":96100876288,"internally_reserved":0,"allocated":70107136,"data_stored":69793696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.182}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.221},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.19800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89001984,"data_stored":88998371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:51.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:51.581 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:51.581 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:51.596 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-16T10:34:51.109192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33687,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1440660,"kb_used_data":156332,"kb_used_omap":12,"kb_used_meta":1284275,"kb_avail":281674860,"statfs":{"total":289910292480,"available":288435056640,"internally_reserved":0,"allocated":160083968,"data_stored":159111593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1315097602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2661,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.336295"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28945,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.230332+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:34.230332+0000","last_peered":"2024-09-16T10:34:34.230332+0000","last_clean":"2024-09-16T10:34:34.230332+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:34.230332+0000","last_undegraded":"2024-09-16T10:34:34.230332+0000","last_fullsized":"2024-09-16T10:34:34.230332+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20634,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231223+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:34.231223+0000","last_peered":"2024-09-16T10:34:34.231223+0000","last_clean":"2024-09-16T10:34:34.231223+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:34.231223+0000","last_undegraded":"2024-09-16T10:34:34.231223+0000","last_fullsized":"2024-09-16T10:34:34.231223+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20937,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231979+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:34.231979+0000","last_peered":"2024-09-16T10:34:34.231979+0000","last_clean":"2024-09-16T10:34:34.231979+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:34.231979+0000","last_undegraded":"2024-09-16T10:34:34.231979+0000","last_fullsized":"2024-09-16T10:34:34.231979+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20534,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231067+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:34.231067+0000","last_peered":"2024-09-16T10:34:34.231067+0000","last_clean":"2024-09-16T10:34:34.231067+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:34.231067+0000","last_undegraded":"2024-09-16T10:34:34.231067+0000","last_fullsized":"2024-09-16T10:34:34.231067+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20677,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.229086+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:34.229086+0000","last_peered":"2024-09-16T10:34:34.229086+0000","last_clean":"2024-09-16T10:34:34.229086+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:34.229086+0000","last_undegraded":"2024-09-16T10:34:34.229086+0000","last_fullsized":"2024-09-16T10:34:34.229086+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20716,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231781+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:34.231781+0000","last_peered":"2024-09-16T10:34:34.231781+0000","last_clean":"2024-09-16T10:34:34.231781+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:34.231781+0000","last_undegraded":"2024-09-16T10:34:34.231781+0000","last_fullsized":"2024-09-16T10:34:34.231781+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21341,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229200+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:34.229200+0000","last_peered":"2024-09-16T10:34:34.229200+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:34.229200+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20581,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.237736+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:35.237736+0000","last_peered":"2024-09-16T10:34:35.237736+0000","last_clean":"2024-09-16T10:34:35.237736+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:35.237736+0000","last_undegraded":"2024-09-16T10:34:35.237736+0000","last_fullsized":"2024-09-16T10:34:35.237736+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231624+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:34.231624+0000","last_peered":"2024-09-16T10:34:34.231624+0000","last_clean":"2024-09-16T10:34:34.231624+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:34.231624+0000","last_undegraded":"2024-09-16T10:34:34.231624+0000","last_fullsized":"2024-09-16T10:34:34.231624+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28980,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231726+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:34.231726+0000","last_peered":"2024-09-16T10:34:34.231726+0000","last_clean":"2024-09-16T10:34:34.231726+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:34.231726+0000","last_undegraded":"2024-09-16T10:34:34.231726+0000","last_fullsized":"2024-09-16T10:34:34.231726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21328,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229289+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:34.229289+0000","last_peered":"2024-09-16T10:34:34.229289+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:34.229289+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":103,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228206+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:34.228206+0000","last_peered":"2024-09-16T10:34:34.228206+0000","last_clean":"2024-09-16T10:34:34.228206+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:34.228206+0000","last_undegraded":"2024-09-16T10:34:34.228206+0000","last_fullsized":"2024-09-16T10:34:34.228206+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29109,"reported_epoch":56,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:34:35.294444+0000","last_change":"2024-09-16T10:34:46.392110+0000","last_active":"2024-09-16T10:34:35.294444+0000","last_peered":"2024-09-16T10:34:35.294444+0000","last_clean":"2024-09-16T10:34:35.294444+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:35.294444+0000","last_undegraded":"2024-09-16T10:34:35.294444+0000","last_fullsized":"2024-09-16T10:34:35.294444+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231881+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:34.231881+0000","last_peered":"2024-09-16T10:34:34.231881+0000","last_clean":"2024-09-16T10:34:34.231881+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:34.231881+0000","last_undegraded":"2024-09-16T10:34:34.231881+0000","last_fullsized":"2024-09-16T10:34:34.231881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29074,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.431930+0000","last_change":"2024-09-16T10:34:35.233140+0000","last_active":"2024-09-16T10:34:34.431930+0000","last_peered":"2024-09-16T10:34:34.431930+0000","last_clean":"2024-09-16T10:34:34.431930+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:34.431930+0000","last_undegraded":"2024-09-16T10:34:34.431930+0000","last_fullsized":"2024-09-16T10:34:34.431930+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20297,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229898+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:34.229898+0000","last_peered":"2024-09-16T10:34:34.229898+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:34.229898+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231927+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:34.231927+0000","last_peered":"2024-09-16T10:34:34.231927+0000","last_clean":"2024-09-16T10:34:34.231927+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:34.231927+0000","last_undegraded":"2024-09-16T10:34:34.231927+0000","last_fullsized":"2024-09-16T10:34:34.231927+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":28482,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:49.297826+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:49.297826+0000","last_peered":"2024-09-16T10:34:49.297826+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:49.297826+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231221+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:34.231221+0000","last_peered":"2024-09-16T10:34:34.231221+0000","last_clean":"2024-09-16T10:34:34.231221+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:34.231221+0000","last_undegraded":"2024-09-16T10:34:34.231221+0000","last_fullsized":"2024-09-16T10:34:34.231221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24511,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232000+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:34.232000+0000","last_peered":"2024-09-16T10:34:34.232000+0000","last_clean":"2024-09-16T10:34:34.232000+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:34.232000+0000","last_undegraded":"2024-09-16T10:34:34.232000+0000","last_fullsized":"2024-09-16T10:34:34.232000+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232069+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:34.232069+0000","last_peered":"2024-09-16T10:34:34.232069+0000","last_clean":"2024-09-16T10:34:34.232069+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:34.232069+0000","last_undegraded":"2024-09-16T10:34:34.232069+0000","last_fullsized":"2024-09-16T10:34:34.232069+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.232276+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:35.232276+0000","last_peered":"2024-09-16T10:34:35.232276+0000","last_clean":"2024-09-16T10:34:35.232276+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:35.232276+0000","last_undegraded":"2024-09-16T10:34:35.232276+0000","last_fullsized":"2024-09-16T10:34:35.232276+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21150,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229518+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:34.229518+0000","last_peered":"2024-09-16T10:34:34.229518+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:34.229518+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33686,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158639297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019028,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429916,"kb_used_data":87372,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93941924,"statfs":{"total":96636764160,"available":96196530176,"internally_reserved":0,"allocated":89468928,"data_stored":89155951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.22800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.249},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.26200000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691972,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487416,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884424,"statfs":{"total":96636764160,"available":96137650176,"internally_reserved":0,"allocated":507904,"data_stored":161946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":498593450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.21099999999999999},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.249}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.33000000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215255,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523328,"kb_used_data":68464,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848512,"statfs":{"total":96636764160,"available":96100876288,"internally_reserved":0,"allocated":70107136,"data_stored":69793696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89001984,"data_stored":88998371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:51.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:51.921 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:51.921 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:51.934 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-16T10:34:51.109192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33687,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1440660,"kb_used_data":156332,"kb_used_omap":12,"kb_used_meta":1284275,"kb_avail":281674860,"statfs":{"total":289910292480,"available":288435056640,"internally_reserved":0,"allocated":160083968,"data_stored":159111593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1315097602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2661,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.336295"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28945,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.230332+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:34.230332+0000","last_peered":"2024-09-16T10:34:34.230332+0000","last_clean":"2024-09-16T10:34:34.230332+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:34.230332+0000","last_undegraded":"2024-09-16T10:34:34.230332+0000","last_fullsized":"2024-09-16T10:34:34.230332+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20634,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231223+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:34.231223+0000","last_peered":"2024-09-16T10:34:34.231223+0000","last_clean":"2024-09-16T10:34:34.231223+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:34.231223+0000","last_undegraded":"2024-09-16T10:34:34.231223+0000","last_fullsized":"2024-09-16T10:34:34.231223+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20937,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231979+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:34.231979+0000","last_peered":"2024-09-16T10:34:34.231979+0000","last_clean":"2024-09-16T10:34:34.231979+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:34.231979+0000","last_undegraded":"2024-09-16T10:34:34.231979+0000","last_fullsized":"2024-09-16T10:34:34.231979+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20534,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231067+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:34.231067+0000","last_peered":"2024-09-16T10:34:34.231067+0000","last_clean":"2024-09-16T10:34:34.231067+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:34.231067+0000","last_undegraded":"2024-09-16T10:34:34.231067+0000","last_fullsized":"2024-09-16T10:34:34.231067+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20677,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.229086+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:34.229086+0000","last_peered":"2024-09-16T10:34:34.229086+0000","last_clean":"2024-09-16T10:34:34.229086+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:34.229086+0000","last_undegraded":"2024-09-16T10:34:34.229086+0000","last_fullsized":"2024-09-16T10:34:34.229086+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20716,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231781+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:34.231781+0000","last_peered":"2024-09-16T10:34:34.231781+0000","last_clean":"2024-09-16T10:34:34.231781+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:34.231781+0000","last_undegraded":"2024-09-16T10:34:34.231781+0000","last_fullsized":"2024-09-16T10:34:34.231781+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21341,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229200+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:34.229200+0000","last_peered":"2024-09-16T10:34:34.229200+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:34.229200+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20581,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.237736+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:35.237736+0000","last_peered":"2024-09-16T10:34:35.237736+0000","last_clean":"2024-09-16T10:34:35.237736+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:35.237736+0000","last_undegraded":"2024-09-16T10:34:35.237736+0000","last_fullsized":"2024-09-16T10:34:35.237736+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231624+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:34.231624+0000","last_peered":"2024-09-16T10:34:34.231624+0000","last_clean":"2024-09-16T10:34:34.231624+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:34.231624+0000","last_undegraded":"2024-09-16T10:34:34.231624+0000","last_fullsized":"2024-09-16T10:34:34.231624+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28980,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231726+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:34.231726+0000","last_peered":"2024-09-16T10:34:34.231726+0000","last_clean":"2024-09-16T10:34:34.231726+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:34.231726+0000","last_undegraded":"2024-09-16T10:34:34.231726+0000","last_fullsized":"2024-09-16T10:34:34.231726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21328,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229289+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:34.229289+0000","last_peered":"2024-09-16T10:34:34.229289+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:34.229289+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":103,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228206+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:34.228206+0000","last_peered":"2024-09-16T10:34:34.228206+0000","last_clean":"2024-09-16T10:34:34.228206+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:34.228206+0000","last_undegraded":"2024-09-16T10:34:34.228206+0000","last_fullsized":"2024-09-16T10:34:34.228206+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29109,"reported_epoch":56,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:34:35.294444+0000","last_change":"2024-09-16T10:34:46.392110+0000","last_active":"2024-09-16T10:34:35.294444+0000","last_peered":"2024-09-16T10:34:35.294444+0000","last_clean":"2024-09-16T10:34:35.294444+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:35.294444+0000","last_undegraded":"2024-09-16T10:34:35.294444+0000","last_fullsized":"2024-09-16T10:34:35.294444+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231881+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:34.231881+0000","last_peered":"2024-09-16T10:34:34.231881+0000","last_clean":"2024-09-16T10:34:34.231881+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:34.231881+0000","last_undegraded":"2024-09-16T10:34:34.231881+0000","last_fullsized":"2024-09-16T10:34:34.231881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29074,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.431930+0000","last_change":"2024-09-16T10:34:35.233140+0000","last_active":"2024-09-16T10:34:34.431930+0000","last_peered":"2024-09-16T10:34:34.431930+0000","last_clean":"2024-09-16T10:34:34.431930+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:34.431930+0000","last_undegraded":"2024-09-16T10:34:34.431930+0000","last_fullsized":"2024-09-16T10:34:34.431930+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20297,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229898+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:34.229898+0000","last_peered":"2024-09-16T10:34:34.229898+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:34.229898+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231927+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:34.231927+0000","last_peered":"2024-09-16T10:34:34.231927+0000","last_clean":"2024-09-16T10:34:34.231927+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:34.231927+0000","last_undegraded":"2024-09-16T10:34:34.231927+0000","last_fullsized":"2024-09-16T10:34:34.231927+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":28482,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:49.297826+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:49.297826+0000","last_peered":"2024-09-16T10:34:49.297826+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:49.297826+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231221+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:34.231221+0000","last_peered":"2024-09-16T10:34:34.231221+0000","last_clean":"2024-09-16T10:34:34.231221+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:34.231221+0000","last_undegraded":"2024-09-16T10:34:34.231221+0000","last_fullsized":"2024-09-16T10:34:34.231221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24511,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232000+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:34.232000+0000","last_peered":"2024-09-16T10:34:34.232000+0000","last_clean":"2024-09-16T10:34:34.232000+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:34.232000+0000","last_undegraded":"2024-09-16T10:34:34.232000+0000","last_fullsized":"2024-09-16T10:34:34.232000+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232069+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:34.232069+0000","last_peered":"2024-09-16T10:34:34.232069+0000","last_clean":"2024-09-16T10:34:34.232069+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:34.232069+0000","last_undegraded":"2024-09-16T10:34:34.232069+0000","last_fullsized":"2024-09-16T10:34:34.232069+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.232276+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:35.232276+0000","last_peered":"2024-09-16T10:34:35.232276+0000","last_clean":"2024-09-16T10:34:35.232276+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:35.232276+0000","last_undegraded":"2024-09-16T10:34:35.232276+0000","last_fullsized":"2024-09-16T10:34:35.232276+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21150,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229518+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:34.229518+0000","last_peered":"2024-09-16T10:34:34.229518+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:34.229518+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33686,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158639297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019028,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429916,"kb_used_data":87372,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93941924,"statfs":{"total":96636764160,"available":96196530176,"internally_reserved":0,"allocated":89468928,"data_stored":89155951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.22800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.249},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.26200000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691972,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487416,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884424,"statfs":{"total":96636764160,"available":96137650176,"internally_reserved":0,"allocated":507904,"data_stored":161946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":498593450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.21099999999999999},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.249}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.33000000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215255,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523328,"kb_used_data":68464,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848512,"statfs":{"total":96636764160,"available":96100876288,"internally_reserved":0,"allocated":70107136,"data_stored":69793696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89001984,"data_stored":88998371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:51.936 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:51.936 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21341, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229200+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:34.229200+0000', 'last_peered': '2024-09-16T10:34:34.229200+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:34.229200+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:51.936 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:51.936 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21328, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229289+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:34.229289+0000', 'last_peered': '2024-09-16T10:34:34.229289+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:34.229289+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:51.936 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:51.936 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29109, 'reported_epoch': 56, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-09-16T10:34:35.294444+0000', 'last_change': '2024-09-16T10:34:46.392110+0000', 'last_active': '2024-09-16T10:34:35.294444+0000', 'last_peered': '2024-09-16T10:34:35.294444+0000', 'last_clean': '2024-09-16T10:34:35.294444+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:34:35.294444+0000', 'last_undegraded': '2024-09-16T10:34:35.294444+0000', 'last_fullsized': '2024-09-16T10:34:35.294444+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:51.936 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:51.937 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20297, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229898+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:34.229898+0000', 'last_peered': '2024-09-16T10:34:34.229898+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:34.229898+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:51.937 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.6 is not active+clean 2024-09-16T10:34:51.937 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.6', 'version': "48'20914", 'reported_seq': 28482, 'reported_epoch': 56, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:49.297826+0000', 'last_change': '2024-09-16T10:34:33.677585+0000', 'last_active': '2024-09-16T10:34:49.297826+0000', 'last_peered': '2024-09-16T10:34:49.297826+0000', 'last_clean': '2024-09-16T10:23:54.332560+0000', 'last_became_active': '2024-09-16T10:23:56.921710+0000', 'last_became_peered': '2024-09-16T10:23:56.921710+0000', 'last_unstale': '2024-09-16T10:34:49.297826+0000', 'last_undegraded': '2024-09-16T10:23:56.476308+0000', 'last_fullsized': '2024-09-16T10:23:56.367209+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 2199, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1449, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 631, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20911, 'num_write_kb': 41816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3712, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -628}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:51.937 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:51.937 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 21150, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:34.229518+0000', 'last_change': '2024-09-16T10:23:56.934794+0000', 'last_active': '2024-09-16T10:34:34.229518+0000', 'last_peered': '2024-09-16T10:34:34.229518+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:34.229518+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4325, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:51.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:52.367 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:52.367 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17660/40 objects degraded (44150.000%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":781,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":58,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean+scrubbing+deep","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1475235840,"bytes_avail":288435056640,"bytes_total":289910292480,"degraded_objects":17660,"degraded_total":40,"degraded_ratio":441.5,"recovering_objects_per_sec":298,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":3585,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:52.383 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:52.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:52.705 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:52.705 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:52.718 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-09-16T10:34:52.116128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33687,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":43,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1440660,"kb_used_data":156332,"kb_used_omap":12,"kb_used_meta":1284275,"kb_avail":281674860,"statfs":{"total":289910292480,"available":288435056640,"internally_reserved":0,"allocated":160083968,"data_stored":159111593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1315097602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2661,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.342913"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28945,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.230332+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:34.230332+0000","last_peered":"2024-09-16T10:34:34.230332+0000","last_clean":"2024-09-16T10:34:34.230332+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:34.230332+0000","last_undegraded":"2024-09-16T10:34:34.230332+0000","last_fullsized":"2024-09-16T10:34:34.230332+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20634,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231223+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:34.231223+0000","last_peered":"2024-09-16T10:34:34.231223+0000","last_clean":"2024-09-16T10:34:34.231223+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:34.231223+0000","last_undegraded":"2024-09-16T10:34:34.231223+0000","last_fullsized":"2024-09-16T10:34:34.231223+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20937,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231979+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:34.231979+0000","last_peered":"2024-09-16T10:34:34.231979+0000","last_clean":"2024-09-16T10:34:34.231979+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:34.231979+0000","last_undegraded":"2024-09-16T10:34:34.231979+0000","last_fullsized":"2024-09-16T10:34:34.231979+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20534,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231067+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:34.231067+0000","last_peered":"2024-09-16T10:34:34.231067+0000","last_clean":"2024-09-16T10:34:34.231067+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:34.231067+0000","last_undegraded":"2024-09-16T10:34:34.231067+0000","last_fullsized":"2024-09-16T10:34:34.231067+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20677,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.229086+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:34.229086+0000","last_peered":"2024-09-16T10:34:34.229086+0000","last_clean":"2024-09-16T10:34:34.229086+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:34.229086+0000","last_undegraded":"2024-09-16T10:34:34.229086+0000","last_fullsized":"2024-09-16T10:34:34.229086+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20716,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231781+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:34.231781+0000","last_peered":"2024-09-16T10:34:34.231781+0000","last_clean":"2024-09-16T10:34:34.231781+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:34.231781+0000","last_undegraded":"2024-09-16T10:34:34.231781+0000","last_fullsized":"2024-09-16T10:34:34.231781+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21341,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229200+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:34.229200+0000","last_peered":"2024-09-16T10:34:34.229200+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:34.229200+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20581,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.237736+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:35.237736+0000","last_peered":"2024-09-16T10:34:35.237736+0000","last_clean":"2024-09-16T10:34:35.237736+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:35.237736+0000","last_undegraded":"2024-09-16T10:34:35.237736+0000","last_fullsized":"2024-09-16T10:34:35.237736+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231624+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:34.231624+0000","last_peered":"2024-09-16T10:34:34.231624+0000","last_clean":"2024-09-16T10:34:34.231624+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:34.231624+0000","last_undegraded":"2024-09-16T10:34:34.231624+0000","last_fullsized":"2024-09-16T10:34:34.231624+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28980,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231923+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:34.231923+0000","last_peered":"2024-09-16T10:34:34.231923+0000","last_clean":"2024-09-16T10:34:34.231923+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:34.231923+0000","last_undegraded":"2024-09-16T10:34:34.231923+0000","last_fullsized":"2024-09-16T10:34:34.231923+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231726+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:34.231726+0000","last_peered":"2024-09-16T10:34:34.231726+0000","last_clean":"2024-09-16T10:34:34.231726+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:34.231726+0000","last_undegraded":"2024-09-16T10:34:34.231726+0000","last_fullsized":"2024-09-16T10:34:34.231726+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21328,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229289+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:34.229289+0000","last_peered":"2024-09-16T10:34:34.229289+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:34.229289+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":103,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.228206+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:34.228206+0000","last_peered":"2024-09-16T10:34:34.228206+0000","last_clean":"2024-09-16T10:34:34.228206+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:34.228206+0000","last_undegraded":"2024-09-16T10:34:34.228206+0000","last_fullsized":"2024-09-16T10:34:34.228206+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29109,"reported_epoch":56,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:34:35.294444+0000","last_change":"2024-09-16T10:34:46.392110+0000","last_active":"2024-09-16T10:34:35.294444+0000","last_peered":"2024-09-16T10:34:35.294444+0000","last_clean":"2024-09-16T10:34:35.294444+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:35.294444+0000","last_undegraded":"2024-09-16T10:34:35.294444+0000","last_fullsized":"2024-09-16T10:34:35.294444+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":84,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231881+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:34.231881+0000","last_peered":"2024-09-16T10:34:34.231881+0000","last_clean":"2024-09-16T10:34:34.231881+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:34.231881+0000","last_undegraded":"2024-09-16T10:34:34.231881+0000","last_fullsized":"2024-09-16T10:34:34.231881+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29074,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.431930+0000","last_change":"2024-09-16T10:34:35.233140+0000","last_active":"2024-09-16T10:34:34.431930+0000","last_peered":"2024-09-16T10:34:34.431930+0000","last_clean":"2024-09-16T10:34:34.431930+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:34.431930+0000","last_undegraded":"2024-09-16T10:34:34.431930+0000","last_fullsized":"2024-09-16T10:34:34.431930+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20297,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229898+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:34.229898+0000","last_peered":"2024-09-16T10:34:34.229898+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:34.229898+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231927+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:34.231927+0000","last_peered":"2024-09-16T10:34:34.231927+0000","last_clean":"2024-09-16T10:34:34.231927+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:34.231927+0000","last_undegraded":"2024-09-16T10:34:34.231927+0000","last_fullsized":"2024-09-16T10:34:34.231927+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":28482,"reported_epoch":56,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:49.297826+0000","last_change":"2024-09-16T10:34:33.677585+0000","last_active":"2024-09-16T10:34:49.297826+0000","last_peered":"2024-09-16T10:34:49.297826+0000","last_clean":"2024-09-16T10:23:54.332560+0000","last_became_active":"2024-09-16T10:23:56.921710+0000","last_became_peered":"2024-09-16T10:23:56.921710+0000","last_unstale":"2024-09-16T10:34:49.297826+0000","last_undegraded":"2024-09-16T10:23:56.476308+0000","last_fullsized":"2024-09-16T10:23:56.367209+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":631,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-628}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":72,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.231221+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:34.231221+0000","last_peered":"2024-09-16T10:34:34.231221+0000","last_clean":"2024-09-16T10:34:34.231221+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:34.231221+0000","last_undegraded":"2024-09-16T10:34:34.231221+0000","last_fullsized":"2024-09-16T10:34:34.231221+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24511,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232000+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:34.232000+0000","last_peered":"2024-09-16T10:34:34.232000+0000","last_clean":"2024-09-16T10:34:34.232000+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:34.232000+0000","last_undegraded":"2024-09-16T10:34:34.232000+0000","last_fullsized":"2024-09-16T10:34:34.232000+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":70,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-16T10:34:34.232069+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:34.232069+0000","last_peered":"2024-09-16T10:34:34.232069+0000","last_clean":"2024-09-16T10:34:34.232069+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:34.232069+0000","last_undegraded":"2024-09-16T10:34:34.232069+0000","last_fullsized":"2024-09-16T10:34:34.232069+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-16T10:34:35.232276+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:35.232276+0000","last_peered":"2024-09-16T10:34:35.232276+0000","last_clean":"2024-09-16T10:34:35.232276+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:35.232276+0000","last_undegraded":"2024-09-16T10:34:35.232276+0000","last_fullsized":"2024-09-16T10:34:35.232276+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21150,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:34.229518+0000","last_change":"2024-09-16T10:23:56.934794+0000","last_active":"2024-09-16T10:34:34.229518+0000","last_peered":"2024-09-16T10:34:34.229518+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:34.229518+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4318}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17660,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33686,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":158679040,"data_stored":158639297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":27,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019028,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":429916,"kb_used_data":87372,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93941924,"statfs":{"total":96636764160,"available":96196530176,"internally_reserved":0,"allocated":89468928,"data_stored":89155951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.22800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.249},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.26200000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691972,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487416,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884424,"statfs":{"total":96636764160,"available":96137650176,"internally_reserved":0,"allocated":507904,"data_stored":161946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":498593450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.21099999999999999},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.249}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.33000000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215255,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523328,"kb_used_data":68464,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848512,"statfs":{"total":96636764160,"available":96100876288,"internally_reserved":0,"allocated":70107136,"data_stored":69793696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.30099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89001984,"data_stored":88998371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:55.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:56.076 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:56.077 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:56.089 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-16T10:34:54.775258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34948,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1430716,"kb_used_data":150492,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281684804,"statfs":{"total":289910292480,"available":288445239296,"internally_reserved":0,"allocated":154103808,"data_stored":153110037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001350"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20636,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111194+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:51.111194+0000","last_peered":"2024-09-16T10:34:51.111194+0000","last_clean":"2024-09-16T10:34:51.111194+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:51.111194+0000","last_undegraded":"2024-09-16T10:34:51.111194+0000","last_fullsized":"2024-09-16T10:34:51.111194+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29113,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:52.115764+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:34:52.115764+0000","last_peered":"2024-09-16T10:34:52.115764+0000","last_clean":"2024-09-16T10:34:52.115764+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:52.115764+0000","last_undegraded":"2024-09-16T10:34:52.115764+0000","last_fullsized":"2024-09-16T10:34:52.115764+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29078,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.313433+0000","last_change":"2024-09-16T10:34:52.115526+0000","last_active":"2024-09-16T10:34:51.313433+0000","last_peered":"2024-09-16T10:34:51.313433+0000","last_clean":"2024-09-16T10:34:51.313433+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:51.313433+0000","last_undegraded":"2024-09-16T10:34:51.313433+0000","last_fullsized":"2024-09-16T10:34:51.313433+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":22413,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:54.301812+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:34:54.301812+0000","last_peered":"2024-09-16T10:34:54.301812+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:54.301812+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34947,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019029,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":424052,"kb_used_data":81512,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93947788,"statfs":{"total":96636764160,"available":96202534912,"internally_reserved":0,"allocated":83468288,"data_stored":83144566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.23200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.38200000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691973,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483324,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888516,"statfs":{"total":96636764160,"available":96141840384,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.182},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.223}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.251}]}]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.184},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.40699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:56.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:56.405 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:56.406 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:56.419 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-16T10:34:54.775258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34948,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1430716,"kb_used_data":150492,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281684804,"statfs":{"total":289910292480,"available":288445239296,"internally_reserved":0,"allocated":154103808,"data_stored":153110037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001350"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20636,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111194+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:51.111194+0000","last_peered":"2024-09-16T10:34:51.111194+0000","last_clean":"2024-09-16T10:34:51.111194+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:51.111194+0000","last_undegraded":"2024-09-16T10:34:51.111194+0000","last_fullsized":"2024-09-16T10:34:51.111194+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29113,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:52.115764+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:34:52.115764+0000","last_peered":"2024-09-16T10:34:52.115764+0000","last_clean":"2024-09-16T10:34:52.115764+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:52.115764+0000","last_undegraded":"2024-09-16T10:34:52.115764+0000","last_fullsized":"2024-09-16T10:34:52.115764+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29078,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.313433+0000","last_change":"2024-09-16T10:34:52.115526+0000","last_active":"2024-09-16T10:34:51.313433+0000","last_peered":"2024-09-16T10:34:51.313433+0000","last_clean":"2024-09-16T10:34:51.313433+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:51.313433+0000","last_undegraded":"2024-09-16T10:34:51.313433+0000","last_fullsized":"2024-09-16T10:34:51.313433+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":22413,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:54.301812+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:34:54.301812+0000","last_peered":"2024-09-16T10:34:54.301812+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:54.301812+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34947,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019029,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":424052,"kb_used_data":81512,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93947788,"statfs":{"total":96636764160,"available":96202534912,"internally_reserved":0,"allocated":83468288,"data_stored":83144566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.23200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.38200000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691973,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483324,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888516,"statfs":{"total":96636764160,"available":96141840384,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.182},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.223}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.251}]}]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.184},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.40699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:34:56.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:34:56.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21343, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113340+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:51.113340+0000', 'last_peered': '2024-09-16T10:34:51.113340+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:51.113340+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:56.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:34:56.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21330, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113391+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:51.113391+0000', 'last_peered': '2024-09-16T10:34:51.113391+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:51.113391+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:56.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:34:56.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29113, 'reported_epoch': 58, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:34:52.115764+0000', 'last_change': '2024-09-16T10:34:52.292549+0000', 'last_active': '2024-09-16T10:34:52.115764+0000', 'last_peered': '2024-09-16T10:34:52.115764+0000', 'last_clean': '2024-09-16T10:34:52.115764+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:34:52.115764+0000', 'last_undegraded': '2024-09-16T10:34:52.115764+0000', 'last_fullsized': '2024-09-16T10:34:52.115764+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:34:56.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:34:56.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20299, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113387+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:51.113387+0000', 'last_peered': '2024-09-16T10:34:51.113387+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:51.113387+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:56.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:34:56.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 22413, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:54.301812+0000', 'last_change': '2024-09-16T10:34:50.835164+0000', 'last_active': '2024-09-16T10:34:54.301812+0000', 'last_peered': '2024-09-16T10:34:54.301812+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:54.301812+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3695, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 708, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3691}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:34:56.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:34:56.871 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:56.871 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16399/40 objects degraded (40997.500%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":785,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":58,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1465053184,"bytes_avail":288445239296,"bytes_total":289910292480,"degraded_objects":16399,"degraded_total":40,"degraded_ratio":409.97500000000002,"recovering_objects_per_sec":345,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2762,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:34:56.886 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:34:56.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:34:57.247 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:34:57.247 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:34:57.261 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-16T10:34:56.775600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34948,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1430716,"kb_used_data":150492,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281684804,"statfs":{"total":289910292480,"available":288445239296,"internally_reserved":0,"allocated":154103808,"data_stored":153110037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2762,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001393"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20636,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111194+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:51.111194+0000","last_peered":"2024-09-16T10:34:51.111194+0000","last_clean":"2024-09-16T10:34:51.111194+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:51.111194+0000","last_undegraded":"2024-09-16T10:34:51.111194+0000","last_fullsized":"2024-09-16T10:34:51.111194+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29113,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:52.115764+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:34:52.115764+0000","last_peered":"2024-09-16T10:34:52.115764+0000","last_clean":"2024-09-16T10:34:52.115764+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:52.115764+0000","last_undegraded":"2024-09-16T10:34:52.115764+0000","last_fullsized":"2024-09-16T10:34:52.115764+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29078,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.313433+0000","last_change":"2024-09-16T10:34:52.115526+0000","last_active":"2024-09-16T10:34:51.313433+0000","last_peered":"2024-09-16T10:34:51.313433+0000","last_clean":"2024-09-16T10:34:51.313433+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:51.313433+0000","last_undegraded":"2024-09-16T10:34:51.313433+0000","last_fullsized":"2024-09-16T10:34:51.313433+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":22413,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:54.301812+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:34:54.301812+0000","last_peered":"2024-09-16T10:34:54.301812+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:54.301812+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34947,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019029,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":424052,"kb_used_data":81512,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93947788,"statfs":{"total":96636764160,"available":96202534912,"internally_reserved":0,"allocated":83468288,"data_stored":83144566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.23200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.38200000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691973,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483324,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888516,"statfs":{"total":96636764160,"available":96141840384,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.182},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.223}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.251}]}]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.184},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.40699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:00.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:00.613 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:00.613 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:00.628 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2024-09-16T10:34:58.775987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34948,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1427436,"kb_used_data":139020,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281688084,"statfs":{"total":289910292480,"available":288448598016,"internally_reserved":0,"allocated":142356480,"data_stored":141362709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001445"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20636,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111194+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:51.111194+0000","last_peered":"2024-09-16T10:34:51.111194+0000","last_clean":"2024-09-16T10:34:51.111194+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:51.111194+0000","last_undegraded":"2024-09-16T10:34:51.111194+0000","last_fullsized":"2024-09-16T10:34:51.111194+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29113,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:52.115764+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:34:52.115764+0000","last_peered":"2024-09-16T10:34:52.115764+0000","last_clean":"2024-09-16T10:34:52.115764+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:52.115764+0000","last_undegraded":"2024-09-16T10:34:52.115764+0000","last_fullsized":"2024-09-16T10:34:52.115764+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29080,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:56.307144+0000","last_change":"2024-09-16T10:34:57.702355+0000","last_active":"2024-09-16T10:34:56.307144+0000","last_peered":"2024-09-16T10:34:56.307144+0000","last_clean":"2024-09-16T10:34:56.307144+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:56.307144+0000","last_undegraded":"2024-09-16T10:34:56.307144+0000","last_fullsized":"2024-09-16T10:34:56.307144+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":22413,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:54.301812+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:34:54.301812+0000","last_peered":"2024-09-16T10:34:54.301812+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:54.301812+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3691}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34947,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412580,"kb_used_data":70040,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959260,"statfs":{"total":96636764160,"available":96214282240,"internally_reserved":0,"allocated":71720960,"data_stored":71397238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.23000000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.29899999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691974,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491516,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880324,"statfs":{"total":96636764160,"available":96133451776,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20699999999999999},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.318}]}]},{"osd":0,"up_from":24,"seq":103079215256,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.184},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.40699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:00.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:00.966 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:00.966 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:00.980 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-16T10:35:00.776337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36002,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1427436,"kb_used_data":139020,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281688084,"statfs":{"total":289910292480,"available":288448598016,"internally_reserved":0,"allocated":142356480,"data_stored":141362709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2315,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.667145"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20636,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111194+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:51.111194+0000","last_peered":"2024-09-16T10:34:51.111194+0000","last_clean":"2024-09-16T10:34:51.111194+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:51.111194+0000","last_undegraded":"2024-09-16T10:34:51.111194+0000","last_fullsized":"2024-09-16T10:34:51.111194+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29113,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:52.115764+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:34:52.115764+0000","last_peered":"2024-09-16T10:34:52.115764+0000","last_clean":"2024-09-16T10:34:52.115764+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:52.115764+0000","last_undegraded":"2024-09-16T10:34:52.115764+0000","last_fullsized":"2024-09-16T10:34:52.115764+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29080,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:56.307144+0000","last_change":"2024-09-16T10:34:57.702355+0000","last_active":"2024-09-16T10:34:56.307144+0000","last_peered":"2024-09-16T10:34:56.307144+0000","last_clean":"2024-09-16T10:34:56.307144+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:56.307144+0000","last_undegraded":"2024-09-16T10:34:56.307144+0000","last_fullsized":"2024-09-16T10:34:56.307144+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":24521,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:59.300501+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:34:59.300501+0000","last_peered":"2024-09-16T10:34:59.300501+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:59.300501+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1762,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36001,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412580,"kb_used_data":70040,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959260,"statfs":{"total":96636764160,"available":96214282240,"internally_reserved":0,"allocated":71720960,"data_stored":71397238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.23000000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.29899999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691974,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491516,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880324,"statfs":{"total":96636764160,"available":96133451776,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20699999999999999},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.318}]}]},{"osd":0,"up_from":24,"seq":103079215257,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.186},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.248},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:00.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:00.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21343, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113340+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:51.113340+0000', 'last_peered': '2024-09-16T10:34:51.113340+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:51.113340+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:00.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:35:00.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21330, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113391+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:51.113391+0000', 'last_peered': '2024-09-16T10:34:51.113391+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:51.113391+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:00.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:35:00.981 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29113, 'reported_epoch': 58, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:34:52.115764+0000', 'last_change': '2024-09-16T10:34:52.292549+0000', 'last_active': '2024-09-16T10:34:52.115764+0000', 'last_peered': '2024-09-16T10:34:52.115764+0000', 'last_clean': '2024-09-16T10:34:52.115764+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:34:52.115764+0000', 'last_undegraded': '2024-09-16T10:34:52.115764+0000', 'last_fullsized': '2024-09-16T10:34:52.115764+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:00.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:00.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20299, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113387+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:51.113387+0000', 'last_peered': '2024-09-16T10:34:51.113387+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:51.113387+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:00.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:35:00.982 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 24521, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:59.300501+0000', 'last_change': '2024-09-16T10:34:50.835164+0000', 'last_active': '2024-09-16T10:34:59.300501+0000', 'last_peered': '2024-09-16T10:34:59.300501+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:34:59.300501+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2641, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1762, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2637}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:00.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:01.390 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:01.390 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15345/40 objects degraded (38362.500%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":790,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":58,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1461694464,"bytes_avail":288448598016,"bytes_total":289910292480,"degraded_objects":15345,"degraded_total":40,"degraded_ratio":383.625,"recovering_objects_per_sec":239,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2315,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:01.404 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:01.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:01.753 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:01.753 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:01.766 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-16T10:35:00.776337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36002,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1427436,"kb_used_data":139020,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281688084,"statfs":{"total":289910292480,"available":288448598016,"internally_reserved":0,"allocated":142356480,"data_stored":141362709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2315,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2315,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.667145"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20636,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111194+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:51.111194+0000","last_peered":"2024-09-16T10:34:51.111194+0000","last_clean":"2024-09-16T10:34:51.111194+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:51.111194+0000","last_undegraded":"2024-09-16T10:34:51.111194+0000","last_fullsized":"2024-09-16T10:34:51.111194+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29113,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:52.115764+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:34:52.115764+0000","last_peered":"2024-09-16T10:34:52.115764+0000","last_clean":"2024-09-16T10:34:52.115764+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:52.115764+0000","last_undegraded":"2024-09-16T10:34:52.115764+0000","last_fullsized":"2024-09-16T10:34:52.115764+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29080,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:56.307144+0000","last_change":"2024-09-16T10:34:57.702355+0000","last_active":"2024-09-16T10:34:56.307144+0000","last_peered":"2024-09-16T10:34:56.307144+0000","last_clean":"2024-09-16T10:34:56.307144+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:56.307144+0000","last_undegraded":"2024-09-16T10:34:56.307144+0000","last_fullsized":"2024-09-16T10:34:56.307144+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":24521,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:59.300501+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:34:59.300501+0000","last_peered":"2024-09-16T10:34:59.300501+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:34:59.300501+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1762,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36001,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019030,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412580,"kb_used_data":70040,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959260,"statfs":{"total":96636764160,"available":96214282240,"internally_reserved":0,"allocated":71720960,"data_stored":71397238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.23000000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.29899999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691974,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491516,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880324,"statfs":{"total":96636764160,"available":96133451776,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20699999999999999},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.318}]}]},{"osd":0,"up_from":24,"seq":103079215257,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.186},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.248},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:04.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:05.058 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:05.059 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:05.070 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-16T10:35:04.777072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36962,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1419244,"kb_used_data":139020,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281696276,"statfs":{"total":289910292480,"available":288456986624,"internally_reserved":0,"allocated":142356480,"data_stored":141362709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002141"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20637,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115669+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:52.115669+0000","last_peered":"2024-09-16T10:34:52.115669+0000","last_clean":"2024-09-16T10:34:52.115669+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:52.115669+0000","last_undegraded":"2024-09-16T10:34:52.115669+0000","last_fullsized":"2024-09-16T10:34:52.115669+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29114,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:52.292549+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:34:52.292549+0000","last_peered":"2024-09-16T10:34:52.292549+0000","last_clean":"2024-09-16T10:34:52.292549+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:52.292549+0000","last_undegraded":"2024-09-16T10:34:52.292549+0000","last_fullsized":"2024-09-16T10:34:52.292549+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29081,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:57.702355+0000","last_change":"2024-09-16T10:35:02.404173+0000","last_active":"2024-09-16T10:34:57.702355+0000","last_peered":"2024-09-16T10:34:57.702355+0000","last_clean":"2024-09-16T10:34:57.702355+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:57.702355+0000","last_undegraded":"2024-09-16T10:34:57.702355+0000","last_fullsized":"2024-09-16T10:34:57.702355+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":26441,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:04.299689+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:35:04.299689+0000","last_peered":"2024-09-16T10:35:04.299689+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:35:04.299689+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2722,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36961,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019031,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412580,"kb_used_data":70040,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959260,"statfs":{"total":96636764160,"available":96214282240,"internally_reserved":0,"allocated":71720960,"data_stored":71397238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.23499999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.224},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.34200000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691975,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483324,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888516,"statfs":{"total":96636764160,"available":96141840384,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.17100000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.249}]}]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.182},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.191}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.217},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:05.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:05.402 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:05.402 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:05.414 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-16T10:35:04.777072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36962,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1419244,"kb_used_data":139020,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281696276,"statfs":{"total":289910292480,"available":288456986624,"internally_reserved":0,"allocated":142356480,"data_stored":141362709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002141"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20637,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115669+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:52.115669+0000","last_peered":"2024-09-16T10:34:52.115669+0000","last_clean":"2024-09-16T10:34:52.115669+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:52.115669+0000","last_undegraded":"2024-09-16T10:34:52.115669+0000","last_fullsized":"2024-09-16T10:34:52.115669+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29114,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:52.292549+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:34:52.292549+0000","last_peered":"2024-09-16T10:34:52.292549+0000","last_clean":"2024-09-16T10:34:52.292549+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:52.292549+0000","last_undegraded":"2024-09-16T10:34:52.292549+0000","last_fullsized":"2024-09-16T10:34:52.292549+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29081,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:57.702355+0000","last_change":"2024-09-16T10:35:02.404173+0000","last_active":"2024-09-16T10:34:57.702355+0000","last_peered":"2024-09-16T10:34:57.702355+0000","last_clean":"2024-09-16T10:34:57.702355+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:57.702355+0000","last_undegraded":"2024-09-16T10:34:57.702355+0000","last_fullsized":"2024-09-16T10:34:57.702355+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":26441,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:04.299689+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:35:04.299689+0000","last_peered":"2024-09-16T10:35:04.299689+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:35:04.299689+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2722,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36961,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019031,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412580,"kb_used_data":70040,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959260,"statfs":{"total":96636764160,"available":96214282240,"internally_reserved":0,"allocated":71720960,"data_stored":71397238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.23499999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.224},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.34200000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691975,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483324,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888516,"statfs":{"total":96636764160,"available":96141840384,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.17100000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.249}]}]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.182},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.191}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.217},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:05.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:05.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21343, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113340+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:51.113340+0000', 'last_peered': '2024-09-16T10:34:51.113340+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:51.113340+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:05.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:35:05.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21330, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113391+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:51.113391+0000', 'last_peered': '2024-09-16T10:34:51.113391+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:51.113391+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:05.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:35:05.416 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29114, 'reported_epoch': 58, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:34:52.292549+0000', 'last_change': '2024-09-16T10:34:52.292549+0000', 'last_active': '2024-09-16T10:34:52.292549+0000', 'last_peered': '2024-09-16T10:34:52.292549+0000', 'last_clean': '2024-09-16T10:34:52.292549+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:34:52.292549+0000', 'last_undegraded': '2024-09-16T10:34:52.292549+0000', 'last_fullsized': '2024-09-16T10:34:52.292549+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:05.417 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:05.417 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29081, 'reported_epoch': 58, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:34:57.702355+0000', 'last_change': '2024-09-16T10:35:02.404173+0000', 'last_active': '2024-09-16T10:34:57.702355+0000', 'last_peered': '2024-09-16T10:34:57.702355+0000', 'last_clean': '2024-09-16T10:34:57.702355+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:34:57.702355+0000', 'last_undegraded': '2024-09-16T10:34:57.702355+0000', 'last_fullsized': '2024-09-16T10:34:57.702355+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:05.417 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:05.417 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20299, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113387+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:51.113387+0000', 'last_peered': '2024-09-16T10:34:51.113387+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:51.113387+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:05.417 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:35:05.417 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 26441, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:04.299689+0000', 'last_change': '2024-09-16T10:34:50.835164+0000', 'last_active': '2024-09-16T10:35:04.299689+0000', 'last_peered': '2024-09-16T10:35:04.299689+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:35:04.299689+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1681, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2722, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1677}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:05.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:05.783 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:05.783 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14385/40 objects degraded (35962.500%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":794,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":58,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+clean+scrubbing","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1453305856,"bytes_avail":288456986624,"bytes_total":289910292480,"degraded_objects":14385,"degraded_total":40,"degraded_ratio":359.625,"recovering_objects_per_sec":272,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":3275,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:05.796 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:05.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:06.135 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:06.135 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:06.149 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-16T10:35:04.777072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36962,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1419244,"kb_used_data":139020,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281696276,"statfs":{"total":289910292480,"available":288456986624,"internally_reserved":0,"allocated":142356480,"data_stored":141362709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002141"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20637,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115669+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:52.115669+0000","last_peered":"2024-09-16T10:34:52.115669+0000","last_clean":"2024-09-16T10:34:52.115669+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:52.115669+0000","last_undegraded":"2024-09-16T10:34:52.115669+0000","last_fullsized":"2024-09-16T10:34:52.115669+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29114,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:52.292549+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:34:52.292549+0000","last_peered":"2024-09-16T10:34:52.292549+0000","last_clean":"2024-09-16T10:34:52.292549+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:34:52.292549+0000","last_undegraded":"2024-09-16T10:34:52.292549+0000","last_fullsized":"2024-09-16T10:34:52.292549+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29081,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:34:57.702355+0000","last_change":"2024-09-16T10:35:02.404173+0000","last_active":"2024-09-16T10:34:57.702355+0000","last_peered":"2024-09-16T10:34:57.702355+0000","last_clean":"2024-09-16T10:34:57.702355+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:34:57.702355+0000","last_undegraded":"2024-09-16T10:34:57.702355+0000","last_fullsized":"2024-09-16T10:34:57.702355+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":26441,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:04.299689+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:35:04.299689+0000","last_peered":"2024-09-16T10:35:04.299689+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:35:04.299689+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2722,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36961,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019031,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412580,"kb_used_data":70040,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959260,"statfs":{"total":96636764160,"available":96214282240,"internally_reserved":0,"allocated":71720960,"data_stored":71397238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.23499999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.224},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.34200000000000003}]}]},{"osd":1,"up_from":40,"seq":171798691975,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483324,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888516,"statfs":{"total":96636764160,"available":96141840384,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.17100000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.20499999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.249}]}]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.182},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.191}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.217},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:09.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:09.472 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:09.472 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:09.485 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-16T10:35:08.777706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36962,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1427436,"kb_used_data":139020,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281688084,"statfs":{"total":289910292480,"available":288448598016,"internally_reserved":0,"allocated":142356480,"data_stored":141362709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2014,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002106"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20637,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115669+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:52.115669+0000","last_peered":"2024-09-16T10:34:52.115669+0000","last_clean":"2024-09-16T10:34:52.115669+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:52.115669+0000","last_undegraded":"2024-09-16T10:34:52.115669+0000","last_fullsized":"2024-09-16T10:34:52.115669+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29115,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:01.411904+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:35:01.411904+0000","last_peered":"2024-09-16T10:35:01.411904+0000","last_clean":"2024-09-16T10:35:01.411904+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:01.411904+0000","last_undegraded":"2024-09-16T10:35:01.411904+0000","last_fullsized":"2024-09-16T10:35:01.411904+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29082,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:02.404173+0000","last_change":"2024-09-16T10:35:02.404173+0000","last_active":"2024-09-16T10:35:02.404173+0000","last_peered":"2024-09-16T10:35:02.404173+0000","last_clean":"2024-09-16T10:35:02.404173+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:02.404173+0000","last_undegraded":"2024-09-16T10:35:02.404173+0000","last_fullsized":"2024-09-16T10:35:02.404173+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":26441,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:04.299689+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:35:04.299689+0000","last_peered":"2024-09-16T10:35:04.299689+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:35:04.299689+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2722,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36961,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412580,"kb_used_data":70040,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959260,"statfs":{"total":96636764160,"available":96214282240,"internally_reserved":0,"allocated":71720960,"data_stored":71397238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.20899999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.193},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.22700000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691976,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491516,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880324,"statfs":{"total":96636764160,"available":96133451776,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.22800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.218},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.182},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.191}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.217},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:09.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:09.792 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:09.804 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-16T10:35:08.777706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36962,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1427436,"kb_used_data":139020,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281688084,"statfs":{"total":289910292480,"available":288448598016,"internally_reserved":0,"allocated":142356480,"data_stored":141362709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2014,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002106"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20637,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115669+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:52.115669+0000","last_peered":"2024-09-16T10:34:52.115669+0000","last_clean":"2024-09-16T10:34:52.115669+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:52.115669+0000","last_undegraded":"2024-09-16T10:34:52.115669+0000","last_fullsized":"2024-09-16T10:34:52.115669+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29115,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:01.411904+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:35:01.411904+0000","last_peered":"2024-09-16T10:35:01.411904+0000","last_clean":"2024-09-16T10:35:01.411904+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:01.411904+0000","last_undegraded":"2024-09-16T10:35:01.411904+0000","last_fullsized":"2024-09-16T10:35:01.411904+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29082,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:02.404173+0000","last_change":"2024-09-16T10:35:02.404173+0000","last_active":"2024-09-16T10:35:02.404173+0000","last_peered":"2024-09-16T10:35:02.404173+0000","last_clean":"2024-09-16T10:35:02.404173+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:02.404173+0000","last_undegraded":"2024-09-16T10:35:02.404173+0000","last_fullsized":"2024-09-16T10:35:02.404173+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":26441,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:04.299689+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:35:04.299689+0000","last_peered":"2024-09-16T10:35:04.299689+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:35:04.299689+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2722,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36961,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412580,"kb_used_data":70040,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959260,"statfs":{"total":96636764160,"available":96214282240,"internally_reserved":0,"allocated":71720960,"data_stored":71397238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.20899999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.193},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.22700000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691976,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491516,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880324,"statfs":{"total":96636764160,"available":96133451776,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.22800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.218},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.182},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.191}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.217},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:09.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:09.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21343, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113340+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:34:51.113340+0000', 'last_peered': '2024-09-16T10:34:51.113340+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:34:51.113340+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:09.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:35:09.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 21330, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113391+0000', 'last_change': '2024-09-16T10:23:56.935130+0000', 'last_active': '2024-09-16T10:34:51.113391+0000', 'last_peered': '2024-09-16T10:34:51.113391+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:34:51.113391+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4339, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 99, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4335}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:09.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:35:09.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29115, 'reported_epoch': 58, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:01.411904+0000', 'last_change': '2024-09-16T10:34:52.292549+0000', 'last_active': '2024-09-16T10:35:01.411904+0000', 'last_peered': '2024-09-16T10:35:01.411904+0000', 'last_clean': '2024-09-16T10:35:01.411904+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:35:01.411904+0000', 'last_undegraded': '2024-09-16T10:35:01.411904+0000', 'last_fullsized': '2024-09-16T10:35:01.411904+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:09.806 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:09.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29082, 'reported_epoch': 58, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:02.404173+0000', 'last_change': '2024-09-16T10:35:02.404173+0000', 'last_active': '2024-09-16T10:35:02.404173+0000', 'last_peered': '2024-09-16T10:35:02.404173+0000', 'last_clean': '2024-09-16T10:35:02.404173+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:02.404173+0000', 'last_undegraded': '2024-09-16T10:35:02.404173+0000', 'last_fullsized': '2024-09-16T10:35:02.404173+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:09.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:09.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20299, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:34:51.113387+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:34:51.113387+0000', 'last_peered': '2024-09-16T10:34:51.113387+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:34:51.113387+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:09.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.4 is not active+clean 2024-09-16T10:35:09.807 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.4', 'version': "48'20896", 'reported_seq': 26441, 'reported_epoch': 58, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:04.299689+0000', 'last_change': '2024-09-16T10:34:50.835164+0000', 'last_active': '2024-09-16T10:35:04.299689+0000', 'last_peered': '2024-09-16T10:35:04.299689+0000', 'last_clean': '2024-09-16T10:23:54.364340+0000', 'last_became_active': '2024-09-16T10:23:56.922603+0000', 'last_became_peered': '2024-09-16T10:23:56.922603+0000', 'last_unstale': '2024-09-16T10:35:04.299689+0000', 'last_undegraded': '2024-09-16T10:23:56.466851+0000', 'last_fullsized': '2024-09-16T10:23:56.368666+0000', 'mapping_epoch': 41, 'log_start': "44'10800", 'ondisk_log_start': "44'10800", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1932, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1681, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20893, 'num_write_kb': 41775, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2722, 'num_bytes_recovered': 4579, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1677}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:09.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:10.205 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:10.206 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 14385/40 objects degraded (35962.500%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":798,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":58,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":18},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+clean+scrubbing","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1461694464,"bytes_avail":288448598016,"bytes_total":289910292480,"degraded_objects":14385,"degraded_total":40,"degraded_ratio":359.625,"recovering_objects_per_sec":167,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2014,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:10.219 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:10.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:10.518 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:10.519 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:10.530 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-16T10:35:08.777706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36962,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1427436,"kb_used_data":139020,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281688084,"statfs":{"total":289910292480,"available":288448598016,"internally_reserved":0,"allocated":142356480,"data_stored":141362709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2014,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002106"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20637,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115669+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:34:52.115669+0000","last_peered":"2024-09-16T10:34:52.115669+0000","last_clean":"2024-09-16T10:34:52.115669+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:34:52.115669+0000","last_undegraded":"2024-09-16T10:34:52.115669+0000","last_fullsized":"2024-09-16T10:34:52.115669+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20536,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110873+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:51.110873+0000","last_peered":"2024-09-16T10:34:51.110873+0000","last_clean":"2024-09-16T10:34:51.110873+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:51.110873+0000","last_undegraded":"2024-09-16T10:34:51.110873+0000","last_fullsized":"2024-09-16T10:34:51.110873+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110962+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:51.110962+0000","last_peered":"2024-09-16T10:34:51.110962+0000","last_clean":"2024-09-16T10:34:51.110962+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:51.110962+0000","last_undegraded":"2024-09-16T10:34:51.110962+0000","last_fullsized":"2024-09-16T10:34:51.110962+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21330,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113391+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:51.113391+0000","last_peered":"2024-09-16T10:34:51.113391+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:51.113391+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":105,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112828+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:51.112828+0000","last_peered":"2024-09-16T10:34:51.112828+0000","last_clean":"2024-09-16T10:34:51.112828+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:51.112828+0000","last_undegraded":"2024-09-16T10:34:51.112828+0000","last_fullsized":"2024-09-16T10:34:51.112828+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29115,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:01.411904+0000","last_change":"2024-09-16T10:34:52.292549+0000","last_active":"2024-09-16T10:35:01.411904+0000","last_peered":"2024-09-16T10:35:01.411904+0000","last_clean":"2024-09-16T10:35:01.411904+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:01.411904+0000","last_undegraded":"2024-09-16T10:35:01.411904+0000","last_fullsized":"2024-09-16T10:35:01.411904+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29082,"reported_epoch":58,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:02.404173+0000","last_change":"2024-09-16T10:35:02.404173+0000","last_active":"2024-09-16T10:35:02.404173+0000","last_peered":"2024-09-16T10:35:02.404173+0000","last_clean":"2024-09-16T10:35:02.404173+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:02.404173+0000","last_undegraded":"2024-09-16T10:35:02.404173+0000","last_fullsized":"2024-09-16T10:35:02.404173+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":74,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111016+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:51.111016+0000","last_peered":"2024-09-16T10:34:51.111016+0000","last_clean":"2024-09-16T10:34:51.111016+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:51.111016+0000","last_undegraded":"2024-09-16T10:34:51.111016+0000","last_fullsized":"2024-09-16T10:34:51.111016+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":26441,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:04.299689+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:35:04.299689+0000","last_peered":"2024-09-16T10:35:04.299689+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:35:04.299689+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2722,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14385,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36961,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412580,"kb_used_data":70040,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959260,"statfs":{"total":96636764160,"available":96214282240,"internally_reserved":0,"allocated":71720960,"data_stored":71397238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":2.4969999999999999,"15min":1.6240000000000001},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":5.2229999999999999,"5min":13.74,"15min":13.74},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.9039999999999999,"15min":1.7370000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17899999999999999},"max":{"1min":6.9580000000000002,"5min":13.917,"15min":13.917},"last":0.20899999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:34:08 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":2.8279999999999998,"15min":2.851},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":2.798,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.193},{"interface":"front","average":{"1min":1.105,"5min":3.1400000000000001,"15min":2.7090000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.7930000000000001,"5min":17.219999999999999,"15min":17.914000000000001},"last":0.22700000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691976,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491516,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880324,"statfs":{"total":96636764160,"available":96133451776,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.22800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.218},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215258,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.182},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.191}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.217},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.24099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:13.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:13.848 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:13.849 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:13.861 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-09-16T10:35:13.154394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37953,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":44,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1427444,"kb_used_data":139028,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281688076,"statfs":{"total":289910292480,"available":288448589824,"internally_reserved":0,"allocated":142364672,"data_stored":141365514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-991,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":991,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.377322"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28947,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.113229+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:34:51.113229+0000","last_peered":"2024-09-16T10:34:51.113229+0000","last_clean":"2024-09-16T10:34:51.113229+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:34:51.113229+0000","last_undegraded":"2024-09-16T10:34:51.113229+0000","last_fullsized":"2024-09-16T10:34:51.113229+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20638,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:35:02.702808+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:02.702808+0000","last_peered":"2024-09-16T10:35:02.702808+0000","last_clean":"2024-09-16T10:35:02.702808+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:02.702808+0000","last_undegraded":"2024-09-16T10:35:02.702808+0000","last_fullsized":"2024-09-16T10:35:02.702808+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20939,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110656+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:34:51.110656+0000","last_peered":"2024-09-16T10:34:51.110656+0000","last_clean":"2024-09-16T10:34:51.110656+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:34:51.110656+0000","last_undegraded":"2024-09-16T10:34:51.110656+0000","last_fullsized":"2024-09-16T10:34:51.110656+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20537,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115443+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:52.115443+0000","last_peered":"2024-09-16T10:34:52.115443+0000","last_clean":"2024-09-16T10:34:52.115443+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:52.115443+0000","last_undegraded":"2024-09-16T10:34:52.115443+0000","last_fullsized":"2024-09-16T10:34:52.115443+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20679,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112886+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:34:51.112886+0000","last_peered":"2024-09-16T10:34:51.112886+0000","last_clean":"2024-09-16T10:34:51.112886+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:34:51.112886+0000","last_undegraded":"2024-09-16T10:34:51.112886+0000","last_fullsized":"2024-09-16T10:34:51.112886+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20718,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110616+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:34:51.110616+0000","last_peered":"2024-09-16T10:34:51.110616+0000","last_clean":"2024-09-16T10:34:51.110616+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:34:51.110616+0000","last_undegraded":"2024-09-16T10:34:51.110616+0000","last_fullsized":"2024-09-16T10:34:51.110616+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21343,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113340+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:34:51.113340+0000","last_peered":"2024-09-16T10:34:51.113340+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:34:51.113340+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20583,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.112816+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:34:51.112816+0000","last_peered":"2024-09-16T10:34:51.112816+0000","last_clean":"2024-09-16T10:34:51.112816+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:34:51.112816+0000","last_undegraded":"2024-09-16T10:34:51.112816+0000","last_fullsized":"2024-09-16T10:34:51.112816+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110395+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:34:51.110395+0000","last_peered":"2024-09-16T10:34:51.110395+0000","last_clean":"2024-09-16T10:34:51.110395+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:34:51.110395+0000","last_undegraded":"2024-09-16T10:34:51.110395+0000","last_fullsized":"2024-09-16T10:34:51.110395+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28982,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110563+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:34:51.110563+0000","last_peered":"2024-09-16T10:34:51.110563+0000","last_clean":"2024-09-16T10:34:51.110563+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:34:51.110563+0000","last_undegraded":"2024-09-16T10:34:51.110563+0000","last_fullsized":"2024-09-16T10:34:51.110563+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115690+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:52.115690+0000","last_peered":"2024-09-16T10:34:52.115690+0000","last_clean":"2024-09-16T10:34:52.115690+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:52.115690+0000","last_undegraded":"2024-09-16T10:34:52.115690+0000","last_fullsized":"2024-09-16T10:34:52.115690+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":21331,"reported_epoch":58,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:52.119099+0000","last_change":"2024-09-16T10:23:56.935130+0000","last_active":"2024-09-16T10:34:52.119099+0000","last_peered":"2024-09-16T10:34:52.119099+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:34:52.119099+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4339,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4335}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":106,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.118547+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:34:52.118547+0000","last_peered":"2024-09-16T10:34:52.118547+0000","last_clean":"2024-09-16T10:34:52.118547+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:34:52.118547+0000","last_undegraded":"2024-09-16T10:34:52.118547+0000","last_fullsized":"2024-09-16T10:34:52.118547+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29117,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:35:10.426193+0000","last_change":"2024-09-16T10:35:12.152023+0000","last_active":"2024-09-16T10:35:10.426193+0000","last_peered":"2024-09-16T10:35:10.426193+0000","last_clean":"2024-09-16T10:35:10.426193+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:10.426193+0000","last_undegraded":"2024-09-16T10:35:10.426193+0000","last_fullsized":"2024-09-16T10:35:10.426193+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":86,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110484+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:34:51.110484+0000","last_peered":"2024-09-16T10:34:51.110484+0000","last_clean":"2024-09-16T10:34:51.110484+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:34:51.110484+0000","last_undegraded":"2024-09-16T10:34:51.110484+0000","last_fullsized":"2024-09-16T10:34:51.110484+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29084,"reported_epoch":59,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:12.151839+0000","last_change":"2024-09-16T10:35:12.381230+0000","last_active":"2024-09-16T10:35:12.151839+0000","last_peered":"2024-09-16T10:35:12.151839+0000","last_clean":"2024-09-16T10:35:12.151839+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:12.151839+0000","last_undegraded":"2024-09-16T10:35:12.151839+0000","last_fullsized":"2024-09-16T10:35:12.151839+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:34:51.113387+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:34:51.113387+0000","last_peered":"2024-09-16T10:34:51.113387+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:34:51.113387+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110615+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:34:51.110615+0000","last_peered":"2024-09-16T10:34:51.110615+0000","last_clean":"2024-09-16T10:34:51.110615+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:34:51.110615+0000","last_undegraded":"2024-09-16T10:34:51.110615+0000","last_fullsized":"2024-09-16T10:34:51.110615+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21206,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.221564+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:34:52.221564+0000","last_peered":"2024-09-16T10:34:52.221564+0000","last_clean":"2024-09-16T10:34:52.221564+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:34:52.221564+0000","last_undegraded":"2024-09-16T10:34:52.221564+0000","last_fullsized":"2024-09-16T10:34:52.221564+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115904+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:52.115904+0000","last_peered":"2024-09-16T10:34:52.115904+0000","last_clean":"2024-09-16T10:34:52.115904+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:52.115904+0000","last_undegraded":"2024-09-16T10:34:52.115904+0000","last_fullsized":"2024-09-16T10:34:52.115904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24513,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.111214+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:34:51.111214+0000","last_peered":"2024-09-16T10:34:51.111214+0000","last_clean":"2024-09-16T10:34:51.111214+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:34:51.111214+0000","last_undegraded":"2024-09-16T10:34:51.111214+0000","last_fullsized":"2024-09-16T10:34:51.111214+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":72,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110450+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:34:51.110450+0000","last_peered":"2024-09-16T10:34:51.110450+0000","last_clean":"2024-09-16T10:34:51.110450+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:34:51.110450+0000","last_undegraded":"2024-09-16T10:34:51.110450+0000","last_fullsized":"2024-09-16T10:34:51.110450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":81,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-16T10:34:51.110791+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:34:51.110791+0000","last_peered":"2024-09-16T10:34:51.110791+0000","last_clean":"2024-09-16T10:34:51.110791+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:34:51.110791+0000","last_undegraded":"2024-09-16T10:34:51.110791+0000","last_fullsized":"2024-09-16T10:34:51.110791+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":28423,"reported_epoch":58,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:09.301879+0000","last_change":"2024-09-16T10:34:50.835164+0000","last_active":"2024-09-16T10:35:09.301879+0000","last_peered":"2024-09-16T10:35:09.301879+0000","last_clean":"2024-09-16T10:23:54.364340+0000","last_became_active":"2024-09-16T10:23:56.922603+0000","last_became_peered":"2024-09-16T10:23:56.922603+0000","last_unstale":"2024-09-16T10:35:09.301879+0000","last_undegraded":"2024-09-16T10:23:56.466851+0000","last_fullsized":"2024-09-16T10:23:56.368666+0000","mapping_epoch":41,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":690,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3713,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13394,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37952,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140902400,"data_stored":140873496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412588,"kb_used_data":70048,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959252,"statfs":{"total":96636764160,"available":96214274048,"internally_reserved":0,"allocated":71729152,"data_stored":71400043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.219}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.26400000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691976,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491516,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880324,"statfs":{"total":96636764160,"available":96133451776,"internally_reserved":0,"allocated":512000,"data_stored":166136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.22800000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.218},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28299999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215259,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523340,"kb_used_data":68480,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848500,"statfs":{"total":96636764160,"available":96100864000,"internally_reserved":0,"allocated":70123520,"data_stored":69799335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:13.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:14.810 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:14.825 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-16T10:35:14.778936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39049,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1419272,"kb_used_data":139044,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281696248,"statfs":{"total":289910292480,"available":288456957952,"internally_reserved":0,"allocated":142381056,"data_stored":141372942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001529"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20638,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:35:02.702808+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:02.702808+0000","last_peered":"2024-09-16T10:35:02.702808+0000","last_clean":"2024-09-16T10:35:02.702808+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:02.702808+0000","last_undegraded":"2024-09-16T10:35:02.702808+0000","last_fullsized":"2024-09-16T10:35:02.702808+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20537,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115443+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:52.115443+0000","last_peered":"2024-09-16T10:34:52.115443+0000","last_clean":"2024-09-16T10:34:52.115443+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:52.115443+0000","last_undegraded":"2024-09-16T10:34:52.115443+0000","last_fullsized":"2024-09-16T10:34:52.115443+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115690+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:52.115690+0000","last_peered":"2024-09-16T10:34:52.115690+0000","last_clean":"2024-09-16T10:34:52.115690+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:52.115690+0000","last_undegraded":"2024-09-16T10:34:52.115690+0000","last_fullsized":"2024-09-16T10:34:52.115690+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":22146,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:14.301055+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:14.301055+0000","last_peered":"2024-09-16T10:35:14.301055+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:14.301055+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":505,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3931}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29117,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:35:10.426193+0000","last_change":"2024-09-16T10:35:12.152023+0000","last_active":"2024-09-16T10:35:10.426193+0000","last_peered":"2024-09-16T10:35:10.426193+0000","last_clean":"2024-09-16T10:35:10.426193+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:10.426193+0000","last_undegraded":"2024-09-16T10:35:10.426193+0000","last_fullsized":"2024-09-16T10:35:10.426193+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29084,"reported_epoch":59,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:12.151839+0000","last_change":"2024-09-16T10:35:12.381230+0000","last_active":"2024-09-16T10:35:12.151839+0000","last_peered":"2024-09-16T10:35:12.151839+0000","last_clean":"2024-09-16T10:35:12.151839+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:12.151839+0000","last_undegraded":"2024-09-16T10:35:12.151839+0000","last_fullsized":"2024-09-16T10:35:12.151839+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115904+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:52.115904+0000","last_peered":"2024-09-16T10:34:52.115904+0000","last_clean":"2024-09-16T10:34:52.115904+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:52.115904+0000","last_undegraded":"2024-09-16T10:34:52.115904+0000","last_fullsized":"2024-09-16T10:34:52.115904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39048,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140894208,"data_stored":140872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412588,"kb_used_data":70048,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959252,"statfs":{"total":96636764160,"available":96214274048,"internally_reserved":0,"allocated":71729152,"data_stored":71400043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.219}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.26400000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691977,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483328,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888512,"statfs":{"total":96636764160,"available":96141836288,"internally_reserved":0,"allocated":512000,"data_stored":167975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.189},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.223}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.25900000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:14.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:14.826 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21345, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154693+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:35:12.154693+0000', 'last_peered': '2024-09-16T10:35:12.154693+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:12.154693+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:14.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:35:14.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 22146, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:14.301055+0000', 'last_change': '2024-09-16T10:35:11.932839+0000', 'last_active': '2024-09-16T10:35:14.301055+0000', 'last_peered': '2024-09-16T10:35:14.301055+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:35:14.301055+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3933, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 505, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3931}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:14.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:14.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29084, 'reported_epoch': 59, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:12.151839+0000', 'last_change': '2024-09-16T10:35:12.381230+0000', 'last_active': '2024-09-16T10:35:12.151839+0000', 'last_peered': '2024-09-16T10:35:12.151839+0000', 'last_clean': '2024-09-16T10:35:12.151839+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:12.151839+0000', 'last_undegraded': '2024-09-16T10:35:12.151839+0000', 'last_fullsized': '2024-09-16T10:35:12.151839+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:14.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:14.827 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20301, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154880+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:35:12.154880+0000', 'last_peered': '2024-09-16T10:35:12.154880+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:35:12.154880+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:14.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:15.298 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:15.298 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12298/40 objects degraded (30745.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":804,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":60,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1453334528,"bytes_avail":288456957952,"bytes_total":289910292480,"degraded_objects":12298,"degraded_total":40,"degraded_ratio":307.44999999999999,"recovering_objects_per_sec":260,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2087,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:15.311 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:15.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:15.637 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:15.637 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:15.651 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-16T10:35:14.778936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39049,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1419272,"kb_used_data":139044,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281696248,"statfs":{"total":289910292480,"available":288456957952,"internally_reserved":0,"allocated":142381056,"data_stored":141372942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001529"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20638,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:35:02.702808+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:02.702808+0000","last_peered":"2024-09-16T10:35:02.702808+0000","last_clean":"2024-09-16T10:35:02.702808+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:02.702808+0000","last_undegraded":"2024-09-16T10:35:02.702808+0000","last_fullsized":"2024-09-16T10:35:02.702808+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20537,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115443+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:34:52.115443+0000","last_peered":"2024-09-16T10:34:52.115443+0000","last_clean":"2024-09-16T10:34:52.115443+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:34:52.115443+0000","last_undegraded":"2024-09-16T10:34:52.115443+0000","last_fullsized":"2024-09-16T10:34:52.115443+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115690+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:34:52.115690+0000","last_peered":"2024-09-16T10:34:52.115690+0000","last_clean":"2024-09-16T10:34:52.115690+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:34:52.115690+0000","last_undegraded":"2024-09-16T10:34:52.115690+0000","last_fullsized":"2024-09-16T10:34:52.115690+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":22146,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:14.301055+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:14.301055+0000","last_peered":"2024-09-16T10:35:14.301055+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:14.301055+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":505,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3931}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29117,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:35:10.426193+0000","last_change":"2024-09-16T10:35:12.152023+0000","last_active":"2024-09-16T10:35:10.426193+0000","last_peered":"2024-09-16T10:35:10.426193+0000","last_clean":"2024-09-16T10:35:10.426193+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:10.426193+0000","last_undegraded":"2024-09-16T10:35:10.426193+0000","last_fullsized":"2024-09-16T10:35:10.426193+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29084,"reported_epoch":59,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:12.151839+0000","last_change":"2024-09-16T10:35:12.381230+0000","last_active":"2024-09-16T10:35:12.151839+0000","last_peered":"2024-09-16T10:35:12.151839+0000","last_clean":"2024-09-16T10:35:12.151839+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:12.151839+0000","last_undegraded":"2024-09-16T10:35:12.151839+0000","last_fullsized":"2024-09-16T10:35:12.151839+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":75,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-16T10:34:52.115904+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:34:52.115904+0000","last_peered":"2024-09-16T10:34:52.115904+0000","last_clean":"2024-09-16T10:34:52.115904+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:34:52.115904+0000","last_undegraded":"2024-09-16T10:34:52.115904+0000","last_fullsized":"2024-09-16T10:34:52.115904+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39048,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":140894208,"data_stored":140872530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019033,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":412588,"kb_used_data":70048,"kb_used_omap":4,"kb_used_meta":342523,"kb_avail":93959252,"statfs":{"total":96636764160,"available":96214274048,"internally_reserved":0,"allocated":71729152,"data_stored":71400043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":350744240},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.17899999999999999},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.219}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.26400000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691977,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483328,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888512,"statfs":{"total":96636764160,"available":96141836288,"internally_reserved":0,"allocated":512000,"data_stored":167975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.189},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.223}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.25900000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":71237632,"data_stored":71234019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:18.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:19.046 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:19.047 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:19.061 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-16T10:35:18.779581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39049,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1413988,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1292467,"kb_avail":281701532,"statfs":{"total":289910292480,"available":288462368768,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1323486210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1096,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001443"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.151942+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:12.151942+0000","last_peered":"2024-09-16T10:35:12.151942+0000","last_clean":"2024-09-16T10:35:12.151942+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:12.151942+0000","last_undegraded":"2024-09-16T10:35:12.151942+0000","last_fullsized":"2024-09-16T10:35:12.151942+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":22146,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:14.301055+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:14.301055+0000","last_peered":"2024-09-16T10:35:14.301055+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:14.301055+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":505,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3931}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29120,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:13.424319+0000","last_change":"2024-09-16T10:35:17.451542+0000","last_active":"2024-09-16T10:35:13.424319+0000","last_peered":"2024-09-16T10:35:13.424319+0000","last_clean":"2024-09-16T10:35:13.424319+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:13.424319+0000","last_undegraded":"2024-09-16T10:35:13.424319+0000","last_fullsized":"2024-09-16T10:35:13.424319+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29086,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:13.251649+0000","last_change":"2024-09-16T10:35:16.493467+0000","last_active":"2024-09-16T10:35:13.251649+0000","last_peered":"2024-09-16T10:35:13.251649+0000","last_clean":"2024-09-16T10:35:13.251649+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:13.251649+0000","last_undegraded":"2024-09-16T10:35:13.251649+0000","last_fullsized":"2024-09-16T10:35:13.251649+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152188+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:12.152188+0000","last_peered":"2024-09-16T10:35:12.152188+0000","last_clean":"2024-09-16T10:35:12.152188+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:12.152188+0000","last_undegraded":"2024-09-16T10:35:12.152188+0000","last_fullsized":"2024-09-16T10:35:12.152188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39048,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019034,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.252}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.28899999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691978,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491524,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880316,"statfs":{"total":96636764160,"available":96133443584,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.214},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:19.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:19.404 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:19.404 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:19.419 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-16T10:35:18.779581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39049,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1413988,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1292467,"kb_avail":281701532,"statfs":{"total":289910292480,"available":288462368768,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1323486210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1096,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001443"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.151942+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:12.151942+0000","last_peered":"2024-09-16T10:35:12.151942+0000","last_clean":"2024-09-16T10:35:12.151942+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:12.151942+0000","last_undegraded":"2024-09-16T10:35:12.151942+0000","last_fullsized":"2024-09-16T10:35:12.151942+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":22146,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:14.301055+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:14.301055+0000","last_peered":"2024-09-16T10:35:14.301055+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:14.301055+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":505,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3931}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29120,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:13.424319+0000","last_change":"2024-09-16T10:35:17.451542+0000","last_active":"2024-09-16T10:35:13.424319+0000","last_peered":"2024-09-16T10:35:13.424319+0000","last_clean":"2024-09-16T10:35:13.424319+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:13.424319+0000","last_undegraded":"2024-09-16T10:35:13.424319+0000","last_fullsized":"2024-09-16T10:35:13.424319+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29086,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:13.251649+0000","last_change":"2024-09-16T10:35:16.493467+0000","last_active":"2024-09-16T10:35:13.251649+0000","last_peered":"2024-09-16T10:35:13.251649+0000","last_clean":"2024-09-16T10:35:13.251649+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:13.251649+0000","last_undegraded":"2024-09-16T10:35:13.251649+0000","last_fullsized":"2024-09-16T10:35:13.251649+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152188+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:12.152188+0000","last_peered":"2024-09-16T10:35:12.152188+0000","last_clean":"2024-09-16T10:35:12.152188+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:12.152188+0000","last_undegraded":"2024-09-16T10:35:12.152188+0000","last_fullsized":"2024-09-16T10:35:12.152188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39048,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019034,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.252}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.28899999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691978,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491524,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880316,"statfs":{"total":96636764160,"available":96133443584,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.214},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:19.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:19.420 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21345, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154693+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:35:12.154693+0000', 'last_peered': '2024-09-16T10:35:12.154693+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:12.154693+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:19.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:35:19.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 22146, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:14.301055+0000', 'last_change': '2024-09-16T10:35:11.932839+0000', 'last_active': '2024-09-16T10:35:14.301055+0000', 'last_peered': '2024-09-16T10:35:14.301055+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:35:14.301055+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3933, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 505, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3931}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:19.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:35:19.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29120, 'reported_epoch': 60, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:13.424319+0000', 'last_change': '2024-09-16T10:35:17.451542+0000', 'last_active': '2024-09-16T10:35:13.424319+0000', 'last_peered': '2024-09-16T10:35:13.424319+0000', 'last_clean': '2024-09-16T10:35:13.424319+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:35:13.424319+0000', 'last_undegraded': '2024-09-16T10:35:13.424319+0000', 'last_fullsized': '2024-09-16T10:35:13.424319+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:19.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:19.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29086, 'reported_epoch': 60, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:13.251649+0000', 'last_change': '2024-09-16T10:35:16.493467+0000', 'last_active': '2024-09-16T10:35:13.251649+0000', 'last_peered': '2024-09-16T10:35:13.251649+0000', 'last_clean': '2024-09-16T10:35:13.251649+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:13.251649+0000', 'last_undegraded': '2024-09-16T10:35:13.251649+0000', 'last_fullsized': '2024-09-16T10:35:13.251649+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:19.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:19.421 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20301, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154880+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:35:12.154880+0000', 'last_peered': '2024-09-16T10:35:12.154880+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:35:12.154880+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:19.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:19.854 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:19.854 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12298/40 objects degraded (30745.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":808,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":60,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+clean+scrubbing","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1447923712,"bytes_avail":288462368768,"bytes_total":289910292480,"degraded_objects":12298,"degraded_total":40,"degraded_ratio":307.44999999999999,"recovering_objects_per_sec":136,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1096,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:19.867 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:19.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:20.209 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:20.223 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-16T10:35:18.779581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39049,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1413988,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1292467,"kb_avail":281701532,"statfs":{"total":289910292480,"available":288462368768,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1323486210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1096,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001443"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.151942+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:12.151942+0000","last_peered":"2024-09-16T10:35:12.151942+0000","last_clean":"2024-09-16T10:35:12.151942+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:12.151942+0000","last_undegraded":"2024-09-16T10:35:12.151942+0000","last_fullsized":"2024-09-16T10:35:12.151942+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":22146,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:14.301055+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:14.301055+0000","last_peered":"2024-09-16T10:35:14.301055+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:14.301055+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3933,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":505,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3931}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29120,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:13.424319+0000","last_change":"2024-09-16T10:35:17.451542+0000","last_active":"2024-09-16T10:35:13.424319+0000","last_peered":"2024-09-16T10:35:13.424319+0000","last_clean":"2024-09-16T10:35:13.424319+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:13.424319+0000","last_undegraded":"2024-09-16T10:35:13.424319+0000","last_fullsized":"2024-09-16T10:35:13.424319+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29086,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:13.251649+0000","last_change":"2024-09-16T10:35:16.493467+0000","last_active":"2024-09-16T10:35:13.251649+0000","last_peered":"2024-09-16T10:35:13.251649+0000","last_clean":"2024-09-16T10:35:13.251649+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:13.251649+0000","last_undegraded":"2024-09-16T10:35:13.251649+0000","last_fullsized":"2024-09-16T10:35:13.251649+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152188+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:12.152188+0000","last_peered":"2024-09-16T10:35:12.152188+0000","last_clean":"2024-09-16T10:35:12.152188+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:12.152188+0000","last_undegraded":"2024-09-16T10:35:12.152188+0000","last_fullsized":"2024-09-16T10:35:12.152188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39048,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019034,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.252}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.28899999999999998}]}]},{"osd":1,"up_from":40,"seq":171798691978,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491524,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880316,"statfs":{"total":96636764160,"available":96133443584,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.214},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215260,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.27900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:23.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:23.569 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:23.570 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:23.582 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-16T10:35:22.780358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40035,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1413988,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1292467,"kb_avail":281701532,"statfs":{"total":289910292480,"available":288462368768,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1323486210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001812"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":24118,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:19.305535+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:19.305535+0000","last_peered":"2024-09-16T10:35:19.305535+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:19.305535+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:17.451542+0000","last_change":"2024-09-16T10:35:22.704269+0000","last_active":"2024-09-16T10:35:17.451542+0000","last_peered":"2024-09-16T10:35:17.451542+0000","last_clean":"2024-09-16T10:35:17.451542+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:17.451542+0000","last_undegraded":"2024-09-16T10:35:17.451542+0000","last_fullsized":"2024-09-16T10:35:17.451542+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152188+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:12.152188+0000","last_peered":"2024-09-16T10:35:12.152188+0000","last_clean":"2024-09-16T10:35:12.152188+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:12.152188+0000","last_undegraded":"2024-09-16T10:35:12.152188+0000","last_fullsized":"2024-09-16T10:35:12.152188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40034,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019035,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.217},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.27200000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691978,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491524,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880316,"statfs":{"total":96636764160,"available":96133443584,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.214},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215261,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.219},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.246}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:23.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:23.924 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:23.925 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:23.937 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-16T10:35:22.780358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40035,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1413988,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1292467,"kb_avail":281701532,"statfs":{"total":289910292480,"available":288462368768,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1323486210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001812"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":24118,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:19.305535+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:19.305535+0000","last_peered":"2024-09-16T10:35:19.305535+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:19.305535+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:17.451542+0000","last_change":"2024-09-16T10:35:22.704269+0000","last_active":"2024-09-16T10:35:17.451542+0000","last_peered":"2024-09-16T10:35:17.451542+0000","last_clean":"2024-09-16T10:35:17.451542+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:17.451542+0000","last_undegraded":"2024-09-16T10:35:17.451542+0000","last_fullsized":"2024-09-16T10:35:17.451542+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152188+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:12.152188+0000","last_peered":"2024-09-16T10:35:12.152188+0000","last_clean":"2024-09-16T10:35:12.152188+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:12.152188+0000","last_undegraded":"2024-09-16T10:35:12.152188+0000","last_fullsized":"2024-09-16T10:35:12.152188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40034,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019035,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.217},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.27200000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691978,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491524,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880316,"statfs":{"total":96636764160,"available":96133443584,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.214},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215261,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.219},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.246}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:23.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:23.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21345, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154693+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:35:12.154693+0000', 'last_peered': '2024-09-16T10:35:12.154693+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:12.154693+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:23.941 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:35:23.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 24118, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:19.305535+0000', 'last_change': '2024-09-16T10:35:11.932839+0000', 'last_active': '2024-09-16T10:35:19.305535+0000', 'last_peered': '2024-09-16T10:35:19.305535+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:35:19.305535+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2947, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1491, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2945}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:23.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:23.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29087, 'reported_epoch': 60, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-09-16T10:35:16.493467+0000', 'last_change': '2024-09-16T10:35:22.476392+0000', 'last_active': '2024-09-16T10:35:16.493467+0000', 'last_peered': '2024-09-16T10:35:16.493467+0000', 'last_clean': '2024-09-16T10:35:16.493467+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:16.493467+0000', 'last_undegraded': '2024-09-16T10:35:16.493467+0000', 'last_fullsized': '2024-09-16T10:35:16.493467+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:23.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:23.942 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20301, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154880+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:35:12.154880+0000', 'last_peered': '2024-09-16T10:35:12.154880+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:35:12.154880+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:23.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:24.317 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:24.317 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11312/40 objects degraded (28280.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":813,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":60,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+scrubbing+deep","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1447923712,"bytes_avail":288462368768,"bytes_total":289910292480,"degraded_objects":11312,"degraded_total":40,"degraded_ratio":282.80000000000001,"recovering_objects_per_sec":208,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2082,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:24.329 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:24.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:24.651 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:24.651 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:24.664 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-16T10:35:22.780358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40035,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1413988,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1292467,"kb_avail":281701532,"statfs":{"total":289910292480,"available":288462368768,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1323486210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001812"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":24118,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:19.305535+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:19.305535+0000","last_peered":"2024-09-16T10:35:19.305535+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:19.305535+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2945}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:17.451542+0000","last_change":"2024-09-16T10:35:22.704269+0000","last_active":"2024-09-16T10:35:17.451542+0000","last_peered":"2024-09-16T10:35:17.451542+0000","last_clean":"2024-09-16T10:35:17.451542+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:17.451542+0000","last_undegraded":"2024-09-16T10:35:17.451542+0000","last_fullsized":"2024-09-16T10:35:17.451542+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152188+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:12.152188+0000","last_peered":"2024-09-16T10:35:12.152188+0000","last_clean":"2024-09-16T10:35:12.152188+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:12.152188+0000","last_undegraded":"2024-09-16T10:35:12.152188+0000","last_fullsized":"2024-09-16T10:35:12.152188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40034,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019035,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.217},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.27200000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691978,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491524,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880316,"statfs":{"total":96636764160,"available":96133443584,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":2.9180000000000001,"15min":3.2639999999999998},"min":{"1min":0.156,"5min":0.156,"15min":0.156},"max":{"1min":5.5810000000000004,"5min":12.798,"15min":17.675000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":1.528,"5min":2.911,"15min":2.7730000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":15.151,"5min":15.151,"15min":16.850999999999999},"last":0.23899999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":2.3180000000000001,"15min":2.492},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":4.4329999999999998,"5min":14.089,"15min":18.638999999999999},"last":0.214},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.4809999999999999,"15min":2.7389999999999999},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":5.1920000000000002,"5min":11.074,"15min":11.412000000000001},"last":0.28100000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215261,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":2.2509999999999999,"15min":2.6110000000000002},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.17100000000000001},"max":{"1min":8.9629999999999992,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.219},{"interface":"front","average":{"1min":0.34699999999999998,"5min":2.5939999999999999,"15min":2.2730000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":1.7629999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.246}]},{"osd":2,"last update":"Mon Sep 16 10:34:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":2.3919999999999999,"15min":1.9159999999999999},"min":{"1min":0.20000000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":1.7010000000000001,"5min":15.534000000000001,"15min":18.093},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.75,"5min":2.1389999999999998,"15min":1.913},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":8.9149999999999991,"5min":11.863,"15min":13.752000000000001},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:27.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:27.968 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:27.968 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:27.981 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-16T10:35:26.781011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40784,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1405796,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1284275,"kb_avail":281709724,"statfs":{"total":289910292480,"available":288470757376,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1315097602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1735,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002074"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":25616,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:24.308663+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:24.308663+0000","last_peered":"2024-09-16T10:35:24.308663+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:24.308663+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2240,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2196}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:17.451542+0000","last_change":"2024-09-16T10:35:22.704269+0000","last_active":"2024-09-16T10:35:17.451542+0000","last_peered":"2024-09-16T10:35:17.451542+0000","last_clean":"2024-09-16T10:35:17.451542+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:17.451542+0000","last_undegraded":"2024-09-16T10:35:17.451542+0000","last_fullsized":"2024-09-16T10:35:17.451542+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152188+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:12.152188+0000","last_peered":"2024-09-16T10:35:12.152188+0000","last_clean":"2024-09-16T10:35:12.152188+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:12.152188+0000","last_undegraded":"2024-09-16T10:35:12.152188+0000","last_fullsized":"2024-09-16T10:35:12.152188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40783,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019035,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.217},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.27200000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691979,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483332,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888508,"statfs":{"total":96636764160,"available":96141832192,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.193},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.26300000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:27.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:28.308 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:28.309 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:28.320 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-16T10:35:26.781011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40784,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1405796,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1284275,"kb_avail":281709724,"statfs":{"total":289910292480,"available":288470757376,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1315097602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1735,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002074"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":25616,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:24.308663+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:24.308663+0000","last_peered":"2024-09-16T10:35:24.308663+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:24.308663+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2240,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2196}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:17.451542+0000","last_change":"2024-09-16T10:35:22.704269+0000","last_active":"2024-09-16T10:35:17.451542+0000","last_peered":"2024-09-16T10:35:17.451542+0000","last_clean":"2024-09-16T10:35:17.451542+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:17.451542+0000","last_undegraded":"2024-09-16T10:35:17.451542+0000","last_fullsized":"2024-09-16T10:35:17.451542+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":76,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152188+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:12.152188+0000","last_peered":"2024-09-16T10:35:12.152188+0000","last_clean":"2024-09-16T10:35:12.152188+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:12.152188+0000","last_undegraded":"2024-09-16T10:35:12.152188+0000","last_fullsized":"2024-09-16T10:35:12.152188+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40783,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019035,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.217},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.27200000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691979,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483332,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888508,"statfs":{"total":96636764160,"available":96141832192,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.20599999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.193},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.26300000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:28.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:28.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21345, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154693+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:35:12.154693+0000', 'last_peered': '2024-09-16T10:35:12.154693+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:12.154693+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:28.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:35:28.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 25616, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:24.308663+0000', 'last_change': '2024-09-16T10:35:11.932839+0000', 'last_active': '2024-09-16T10:35:24.308663+0000', 'last_peered': '2024-09-16T10:35:24.308663+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:35:24.308663+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2198, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2240, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2196}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:28.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:28.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29087, 'reported_epoch': 60, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-09-16T10:35:16.493467+0000', 'last_change': '2024-09-16T10:35:22.476392+0000', 'last_active': '2024-09-16T10:35:16.493467+0000', 'last_peered': '2024-09-16T10:35:16.493467+0000', 'last_clean': '2024-09-16T10:35:16.493467+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:16.493467+0000', 'last_undegraded': '2024-09-16T10:35:16.493467+0000', 'last_fullsized': '2024-09-16T10:35:16.493467+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:28.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:28.322 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20301, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154880+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:35:12.154880+0000', 'last_peered': '2024-09-16T10:35:12.154880+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:35:12.154880+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:28.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:28.682 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:28.682 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10563/40 objects degraded (26407.500%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":817,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":60,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+clean+scrubbing+deep","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1439535104,"bytes_avail":288470757376,"bytes_total":289910292480,"degraded_objects":10563,"degraded_total":40,"degraded_ratio":264.07499999999999,"recovering_objects_per_sec":144,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1735,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:28.699 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:28.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:28.998 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:29.011 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-16T10:35:28.781367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40784,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1409892,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281705628,"statfs":{"total":289910292480,"available":288466563072,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1735,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002079"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":25616,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:24.308663+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:24.308663+0000","last_peered":"2024-09-16T10:35:24.308663+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:24.308663+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2240,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2196}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:17.451542+0000","last_change":"2024-09-16T10:35:22.704269+0000","last_active":"2024-09-16T10:35:17.451542+0000","last_peered":"2024-09-16T10:35:17.451542+0000","last_clean":"2024-09-16T10:35:17.451542+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:17.451542+0000","last_undegraded":"2024-09-16T10:35:17.451542+0000","last_fullsized":"2024-09-16T10:35:17.451542+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251971+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:13.251971+0000","last_peered":"2024-09-16T10:35:13.251971+0000","last_clean":"2024-09-16T10:35:13.251971+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:13.251971+0000","last_undegraded":"2024-09-16T10:35:13.251971+0000","last_fullsized":"2024-09-16T10:35:13.251971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40783,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019036,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.251}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.23799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691980,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487428,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884412,"statfs":{"total":96636764160,"available":96137637888,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":498593450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16200000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.187}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.247},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.26800000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215262,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.26500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:32.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:32.313 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:32.314 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:32.325 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-16T10:35:30.781700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41602,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1409892,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281705628,"statfs":{"total":289910292480,"available":288466563072,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2553,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002119"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":27252,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:29.309434+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:29.309434+0000","last_peered":"2024-09-16T10:35:29.309434+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:29.309434+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3058,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1378}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:17.451542+0000","last_change":"2024-09-16T10:35:22.704269+0000","last_active":"2024-09-16T10:35:17.451542+0000","last_peered":"2024-09-16T10:35:17.451542+0000","last_clean":"2024-09-16T10:35:17.451542+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:17.451542+0000","last_undegraded":"2024-09-16T10:35:17.451542+0000","last_fullsized":"2024-09-16T10:35:17.451542+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251971+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:13.251971+0000","last_peered":"2024-09-16T10:35:13.251971+0000","last_clean":"2024-09-16T10:35:13.251971+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:13.251971+0000","last_undegraded":"2024-09-16T10:35:13.251971+0000","last_fullsized":"2024-09-16T10:35:13.251971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41601,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019036,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.251}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.23799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691980,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487428,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884412,"statfs":{"total":96636764160,"available":96137637888,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":498593450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16200000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.187}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.247},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.26800000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215263,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.16600000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:32.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:32.625 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:32.626 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:32.641 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-16T10:35:30.781700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41602,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1409892,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281705628,"statfs":{"total":289910292480,"available":288466563072,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2553,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002119"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":27252,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:29.309434+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:29.309434+0000","last_peered":"2024-09-16T10:35:29.309434+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:29.309434+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3058,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1378}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:17.451542+0000","last_change":"2024-09-16T10:35:22.704269+0000","last_active":"2024-09-16T10:35:17.451542+0000","last_peered":"2024-09-16T10:35:17.451542+0000","last_clean":"2024-09-16T10:35:17.451542+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:17.451542+0000","last_undegraded":"2024-09-16T10:35:17.451542+0000","last_fullsized":"2024-09-16T10:35:17.451542+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251971+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:13.251971+0000","last_peered":"2024-09-16T10:35:13.251971+0000","last_clean":"2024-09-16T10:35:13.251971+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:13.251971+0000","last_undegraded":"2024-09-16T10:35:13.251971+0000","last_fullsized":"2024-09-16T10:35:13.251971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41601,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019036,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.251}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.23799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691980,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487428,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884412,"statfs":{"total":96636764160,"available":96137637888,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":498593450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16200000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.187}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.247},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.26800000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215263,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.16600000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:32.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:32.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21345, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154693+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:35:12.154693+0000', 'last_peered': '2024-09-16T10:35:12.154693+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:12.154693+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:32.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:35:32.642 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 27252, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:29.309434+0000', 'last_change': '2024-09-16T10:35:11.932839+0000', 'last_active': '2024-09-16T10:35:29.309434+0000', 'last_peered': '2024-09-16T10:35:29.309434+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:35:29.309434+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1380, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3058, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1378}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:32.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:32.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29087, 'reported_epoch': 60, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-09-16T10:35:16.493467+0000', 'last_change': '2024-09-16T10:35:22.476392+0000', 'last_active': '2024-09-16T10:35:16.493467+0000', 'last_peered': '2024-09-16T10:35:16.493467+0000', 'last_clean': '2024-09-16T10:35:16.493467+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:16.493467+0000', 'last_undegraded': '2024-09-16T10:35:16.493467+0000', 'last_fullsized': '2024-09-16T10:35:16.493467+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:32.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:32.643 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20301, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154880+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:35:12.154880+0000', 'last_peered': '2024-09-16T10:35:12.154880+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:35:12.154880+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:32.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:33.014 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:33.015 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9745/40 objects degraded (24362.500%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":821,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":60,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+clean+scrubbing+deep","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1443729408,"bytes_avail":288466563072,"bytes_total":289910292480,"degraded_objects":9745,"degraded_total":40,"degraded_ratio":243.625,"recovering_objects_per_sec":212,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2553,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:33.029 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:33.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:33.329 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:33.329 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:33.341 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-09-16T10:35:32.782052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41602,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1409892,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1288371,"kb_avail":281705628,"statfs":{"total":289910292480,"available":288466563072,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1319291906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002090"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":27252,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:29.309434+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:29.309434+0000","last_peered":"2024-09-16T10:35:29.309434+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:29.309434+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3058,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1378}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29122,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:22.704269+0000","last_change":"2024-09-16T10:35:29.534982+0000","last_active":"2024-09-16T10:35:22.704269+0000","last_peered":"2024-09-16T10:35:22.704269+0000","last_clean":"2024-09-16T10:35:22.704269+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:22.704269+0000","last_undegraded":"2024-09-16T10:35:22.704269+0000","last_fullsized":"2024-09-16T10:35:22.704269+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251971+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:13.251971+0000","last_peered":"2024-09-16T10:35:13.251971+0000","last_clean":"2024-09-16T10:35:13.251971+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:13.251971+0000","last_undegraded":"2024-09-16T10:35:13.251971+0000","last_fullsized":"2024-09-16T10:35:13.251971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41601,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019037,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.251}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.23799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691980,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487428,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":486907,"kb_avail":93884412,"statfs":{"total":96636764160,"available":96137637888,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":498593450},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16200000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.187}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.247},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.26800000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215263,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.16600000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.216}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:36.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:36.707 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:36.721 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-09-16T10:35:36.248963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42815,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1401700,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281713820,"statfs":{"total":289910292480,"available":288474951680,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2031,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.467952"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":29678,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:34.309349+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:34.309349+0000","last_peered":"2024-09-16T10:35:34.309349+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:34.309349+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4271,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-165}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29122,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:22.704269+0000","last_change":"2024-09-16T10:35:29.534982+0000","last_active":"2024-09-16T10:35:22.704269+0000","last_peered":"2024-09-16T10:35:22.704269+0000","last_clean":"2024-09-16T10:35:22.704269+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:22.704269+0000","last_undegraded":"2024-09-16T10:35:22.704269+0000","last_fullsized":"2024-09-16T10:35:22.704269+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251971+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:13.251971+0000","last_peered":"2024-09-16T10:35:13.251971+0000","last_clean":"2024-09-16T10:35:13.251971+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:13.251971+0000","last_undegraded":"2024-09-16T10:35:13.251971+0000","last_fullsized":"2024-09-16T10:35:13.251971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42814,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019037,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.251}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.23799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691981,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479236,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93892604,"statfs":{"total":96636764160,"available":96146026496,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":490204842},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16200000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.187}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.219},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.28399999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19900000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.249},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:36.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:37.054 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:37.068 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-16T10:35:36.782690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42815,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1401700,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281713820,"statfs":{"total":289910292480,"available":288474951680,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2031,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001323"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":29678,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:34.309349+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:34.309349+0000","last_peered":"2024-09-16T10:35:34.309349+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:34.309349+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4271,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-165}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29122,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:22.704269+0000","last_change":"2024-09-16T10:35:29.534982+0000","last_active":"2024-09-16T10:35:22.704269+0000","last_peered":"2024-09-16T10:35:22.704269+0000","last_clean":"2024-09-16T10:35:22.704269+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:22.704269+0000","last_undegraded":"2024-09-16T10:35:22.704269+0000","last_fullsized":"2024-09-16T10:35:22.704269+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251971+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:13.251971+0000","last_peered":"2024-09-16T10:35:13.251971+0000","last_clean":"2024-09-16T10:35:13.251971+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:13.251971+0000","last_undegraded":"2024-09-16T10:35:13.251971+0000","last_fullsized":"2024-09-16T10:35:13.251971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42814,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019037,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.251}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.23799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691981,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479236,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93892604,"statfs":{"total":96636764160,"available":96146026496,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":490204842},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16200000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.187}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.219},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.28399999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19900000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.249},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:37.069 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:37.069 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21345, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154693+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:35:12.154693+0000', 'last_peered': '2024-09-16T10:35:12.154693+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:12.154693+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:37.069 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.2 is not active+clean 2024-09-16T10:35:37.069 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.2', 'version': "48'21040", 'reported_seq': 29678, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:34.309349+0000', 'last_change': '2024-09-16T10:35:11.932839+0000', 'last_active': '2024-09-16T10:35:34.309349+0000', 'last_peered': '2024-09-16T10:35:34.309349+0000', 'last_clean': '2024-09-16T10:23:54.364497+0000', 'last_became_active': '2024-09-16T10:23:56.922579+0000', 'last_became_peered': '2024-09-16T10:23:56.922579+0000', 'last_unstale': '2024-09-16T10:35:34.309349+0000', 'last_undegraded': '2024-09-16T10:23:56.485041+0000', 'last_fullsized': '2024-09-16T10:23:56.366767+0000', 'mapping_epoch': 41, 'log_start': "44'11000", 'ondisk_log_start': "44'11000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 167, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 21038, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4271, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -165}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:37.070 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:35:37.070 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29122, 'reported_epoch': 60, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:22.704269+0000', 'last_change': '2024-09-16T10:35:29.534982+0000', 'last_active': '2024-09-16T10:35:22.704269+0000', 'last_peered': '2024-09-16T10:35:22.704269+0000', 'last_clean': '2024-09-16T10:35:22.704269+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:35:22.704269+0000', 'last_undegraded': '2024-09-16T10:35:22.704269+0000', 'last_fullsized': '2024-09-16T10:35:22.704269+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:37.070 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:37.070 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29087, 'reported_epoch': 60, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-09-16T10:35:16.493467+0000', 'last_change': '2024-09-16T10:35:22.476392+0000', 'last_active': '2024-09-16T10:35:16.493467+0000', 'last_peered': '2024-09-16T10:35:16.493467+0000', 'last_clean': '2024-09-16T10:35:16.493467+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:16.493467+0000', 'last_undegraded': '2024-09-16T10:35:16.493467+0000', 'last_fullsized': '2024-09-16T10:35:16.493467+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:37.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:37.071 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 20301, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:12.154880+0000', 'last_change': '2024-09-16T10:23:56.946192+0000', 'last_active': '2024-09-16T10:35:12.154880+0000', 'last_peered': '2024-09-16T10:35:12.154880+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:35:12.154880+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4076, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:37.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:37.483 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:37.483 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8532/40 objects degraded (21330.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":826,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":62,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":19},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+clean+scrubbing+deep","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1435340800,"bytes_avail":288474951680,"bytes_total":289910292480,"degraded_objects":8532,"degraded_total":40,"degraded_ratio":213.30000000000001,"recovering_objects_per_sec":253,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2031,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:37.497 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:37.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:37.822 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:37.822 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:37.835 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-16T10:35:36.782690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42815,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":45,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1401700,"kb_used_data":121476,"kb_used_omap":12,"kb_used_meta":1280179,"kb_avail":281713820,"statfs":{"total":289910292480,"available":288474951680,"internally_reserved":0,"allocated":124391424,"data_stored":123376107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1310903298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2031,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2031,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001323"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28949,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154645+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:12.154645+0000","last_peered":"2024-09-16T10:35:12.154645+0000","last_clean":"2024-09-16T10:35:12.154645+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:12.154645+0000","last_undegraded":"2024-09-16T10:35:12.154645+0000","last_fullsized":"2024-09-16T10:35:12.154645+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20639,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152410+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:12.152410+0000","last_peered":"2024-09-16T10:35:12.152410+0000","last_clean":"2024-09-16T10:35:12.152410+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:12.152410+0000","last_undegraded":"2024-09-16T10:35:12.152410+0000","last_fullsized":"2024-09-16T10:35:12.152410+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20941,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153327+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:12.153327+0000","last_peered":"2024-09-16T10:35:12.153327+0000","last_clean":"2024-09-16T10:35:12.153327+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:12.153327+0000","last_undegraded":"2024-09-16T10:35:12.153327+0000","last_fullsized":"2024-09-16T10:35:12.153327+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20538,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152272+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:12.152272+0000","last_peered":"2024-09-16T10:35:12.152272+0000","last_clean":"2024-09-16T10:35:12.152272+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:12.152272+0000","last_undegraded":"2024-09-16T10:35:12.152272+0000","last_fullsized":"2024-09-16T10:35:12.152272+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20681,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154393+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:12.154393+0000","last_peered":"2024-09-16T10:35:12.154393+0000","last_clean":"2024-09-16T10:35:12.154393+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:12.154393+0000","last_undegraded":"2024-09-16T10:35:12.154393+0000","last_fullsized":"2024-09-16T10:35:12.154393+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20720,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153089+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:12.153089+0000","last_peered":"2024-09-16T10:35:12.153089+0000","last_clean":"2024-09-16T10:35:12.153089+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:12.153089+0000","last_undegraded":"2024-09-16T10:35:12.153089+0000","last_fullsized":"2024-09-16T10:35:12.153089+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21345,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154693+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:12.154693+0000","last_peered":"2024-09-16T10:35:12.154693+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:12.154693+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20585,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154284+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:12.154284+0000","last_peered":"2024-09-16T10:35:12.154284+0000","last_clean":"2024-09-16T10:35:12.154284+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:12.154284+0000","last_undegraded":"2024-09-16T10:35:12.154284+0000","last_fullsized":"2024-09-16T10:35:12.154284+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.152903+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:12.152903+0000","last_peered":"2024-09-16T10:35:12.152903+0000","last_clean":"2024-09-16T10:35:12.152903+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:12.152903+0000","last_undegraded":"2024-09-16T10:35:12.152903+0000","last_fullsized":"2024-09-16T10:35:12.152903+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28984,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153250+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:12.153250+0000","last_peered":"2024-09-16T10:35:12.153250+0000","last_clean":"2024-09-16T10:35:12.153250+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:12.153250+0000","last_undegraded":"2024-09-16T10:35:12.153250+0000","last_fullsized":"2024-09-16T10:35:12.153250+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251765+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:13.251765+0000","last_peered":"2024-09-16T10:35:13.251765+0000","last_clean":"2024-09-16T10:35:13.251765+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:13.251765+0000","last_undegraded":"2024-09-16T10:35:13.251765+0000","last_fullsized":"2024-09-16T10:35:13.251765+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":29678,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:34.309349+0000","last_change":"2024-09-16T10:35:11.932839+0000","last_active":"2024-09-16T10:35:34.309349+0000","last_peered":"2024-09-16T10:35:34.309349+0000","last_clean":"2024-09-16T10:23:54.364497+0000","last_became_active":"2024-09-16T10:23:56.922579+0000","last_became_peered":"2024-09-16T10:23:56.922579+0000","last_unstale":"2024-09-16T10:35:34.309349+0000","last_undegraded":"2024-09-16T10:23:56.485041+0000","last_fullsized":"2024-09-16T10:23:56.366767+0000","mapping_epoch":41,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":167,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4271,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-165}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":108,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.154360+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:12.154360+0000","last_peered":"2024-09-16T10:35:12.154360+0000","last_clean":"2024-09-16T10:35:12.154360+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:12.154360+0000","last_undegraded":"2024-09-16T10:35:12.154360+0000","last_fullsized":"2024-09-16T10:35:12.154360+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29122,"reported_epoch":60,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:22.704269+0000","last_change":"2024-09-16T10:35:29.534982+0000","last_active":"2024-09-16T10:35:22.704269+0000","last_peered":"2024-09-16T10:35:22.704269+0000","last_clean":"2024-09-16T10:35:22.704269+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:22.704269+0000","last_undegraded":"2024-09-16T10:35:22.704269+0000","last_fullsized":"2024-09-16T10:35:22.704269+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":88,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153406+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:12.153406+0000","last_peered":"2024-09-16T10:35:12.153406+0000","last_clean":"2024-09-16T10:35:12.153406+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:12.153406+0000","last_undegraded":"2024-09-16T10:35:12.153406+0000","last_fullsized":"2024-09-16T10:35:12.153406+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29087,"reported_epoch":60,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:16.493467+0000","last_change":"2024-09-16T10:35:22.476392+0000","last_active":"2024-09-16T10:35:16.493467+0000","last_peered":"2024-09-16T10:35:16.493467+0000","last_clean":"2024-09-16T10:35:16.493467+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:16.493467+0000","last_undegraded":"2024-09-16T10:35:16.493467+0000","last_fullsized":"2024-09-16T10:35:16.493467+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:12.154880+0000","last_change":"2024-09-16T10:23:56.946192+0000","last_active":"2024-09-16T10:35:12.154880+0000","last_peered":"2024-09-16T10:35:12.154880+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:12.154880+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4076,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153934+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:12.153934+0000","last_peered":"2024-09-16T10:35:12.153934+0000","last_clean":"2024-09-16T10:35:12.153934+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:12.153934+0000","last_undegraded":"2024-09-16T10:35:12.153934+0000","last_fullsized":"2024-09-16T10:35:12.153934+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21208,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153281+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:12.153281+0000","last_peered":"2024-09-16T10:35:12.153281+0000","last_clean":"2024-09-16T10:35:12.153281+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:12.153281+0000","last_undegraded":"2024-09-16T10:35:12.153281+0000","last_fullsized":"2024-09-16T10:35:12.153281+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":77,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.251971+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:13.251971+0000","last_peered":"2024-09-16T10:35:13.251971+0000","last_clean":"2024-09-16T10:35:13.251971+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:13.251971+0000","last_undegraded":"2024-09-16T10:35:13.251971+0000","last_fullsized":"2024-09-16T10:35:13.251971+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24515,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153309+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:12.153309+0000","last_peered":"2024-09-16T10:35:12.153309+0000","last_clean":"2024-09-16T10:35:12.153309+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:12.153309+0000","last_undegraded":"2024-09-16T10:35:12.153309+0000","last_fullsized":"2024-09-16T10:35:12.153309+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":74,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153200+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:12.153200+0000","last_peered":"2024-09-16T10:35:12.153200+0000","last_clean":"2024-09-16T10:35:12.153200+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:12.153200+0000","last_undegraded":"2024-09-16T10:35:12.153200+0000","last_fullsized":"2024-09-16T10:35:12.153200+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":83,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-16T10:35:12.153374+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:12.153374+0000","last_peered":"2024-09-16T10:35:12.153374+0000","last_clean":"2024-09-16T10:35:12.153374+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:12.153374+0000","last_undegraded":"2024-09-16T10:35:12.153374+0000","last_fullsized":"2024-09-16T10:35:12.153374+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21153,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-16T10:35:13.239086+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:13.239086+0000","last_peered":"2024-09-16T10:35:13.239086+0000","last_clean":"2024-09-16T10:35:13.239086+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:13.239086+0000","last_undegraded":"2024-09-16T10:35:13.239086+0000","last_fullsized":"2024-09-16T10:35:13.239086+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42814,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122888192,"data_stored":122870127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":29,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019037,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399108,"kb_used_data":52472,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93972732,"statfs":{"total":96636764160,"available":96228077568,"internally_reserved":0,"allocated":53731328,"data_stored":53400424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19500000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.251}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.23799999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691981,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479236,"kb_used_data":508,"kb_used_omap":4,"kb_used_meta":478715,"kb_avail":93892604,"statfs":{"total":96636764160,"available":96146026496,"internally_reserved":0,"allocated":520192,"data_stored":170759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":490204842},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16200000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.187}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.219},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.28399999999999997}]}]},{"osd":0,"up_from":24,"seq":103079215264,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523356,"kb_used_data":68496,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848484,"statfs":{"total":96636764160,"available":96100847616,"internally_reserved":0,"allocated":70139904,"data_stored":69804924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.19900000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.249},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.217}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53231616,"data_stored":53231616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:40.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:41.229 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:41.230 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:41.243 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-16T10:35:40.783510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43826,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1396296,"kb_used_data":103784,"kb_used_omap":12,"kb_used_meta":1292467,"kb_avail":281719224,"statfs":{"total":289910292480,"available":288480485376,"internally_reserved":0,"allocated":106274816,"data_stored":105230577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1323486210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2224,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001457"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28951,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.252058+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:35.252058+0000","last_peered":"2024-09-16T10:35:35.252058+0000","last_clean":"2024-09-16T10:35:35.252058+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:35.252058+0000","last_undegraded":"2024-09-16T10:35:35.252058+0000","last_fullsized":"2024-09-16T10:35:35.252058+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20641,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246198+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:35.246198+0000","last_peered":"2024-09-16T10:35:35.246198+0000","last_clean":"2024-09-16T10:35:35.246198+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:35.246198+0000","last_undegraded":"2024-09-16T10:35:35.246198+0000","last_fullsized":"2024-09-16T10:35:35.246198+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246728+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:35.246728+0000","last_peered":"2024-09-16T10:35:35.246728+0000","last_clean":"2024-09-16T10:35:35.246728+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:35.246728+0000","last_undegraded":"2024-09-16T10:35:35.246728+0000","last_fullsized":"2024-09-16T10:35:35.246728+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20540,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246021+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:35.246021+0000","last_peered":"2024-09-16T10:35:35.246021+0000","last_clean":"2024-09-16T10:35:35.246021+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:35.246021+0000","last_undegraded":"2024-09-16T10:35:35.246021+0000","last_fullsized":"2024-09-16T10:35:35.246021+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20683,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248479+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:35.248479+0000","last_peered":"2024-09-16T10:35:35.248479+0000","last_clean":"2024-09-16T10:35:35.248479+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:35.248479+0000","last_undegraded":"2024-09-16T10:35:35.248479+0000","last_fullsized":"2024-09-16T10:35:35.248479+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20722,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246758+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:35.246758+0000","last_peered":"2024-09-16T10:35:35.246758+0000","last_clean":"2024-09-16T10:35:35.246758+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:35.246758+0000","last_undegraded":"2024-09-16T10:35:35.246758+0000","last_fullsized":"2024-09-16T10:35:35.246758+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21347,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:35.248771+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:35.248771+0000","last_peered":"2024-09-16T10:35:35.248771+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:35.248771+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20587,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248438+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:35.248438+0000","last_peered":"2024-09-16T10:35:35.248438+0000","last_clean":"2024-09-16T10:35:35.248438+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:35.248438+0000","last_undegraded":"2024-09-16T10:35:35.248438+0000","last_fullsized":"2024-09-16T10:35:35.248438+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246752+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:35.246752+0000","last_peered":"2024-09-16T10:35:35.246752+0000","last_clean":"2024-09-16T10:35:35.246752+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:35.246752+0000","last_undegraded":"2024-09-16T10:35:35.246752+0000","last_fullsized":"2024-09-16T10:35:35.246752+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28986,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247082+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:35.247082+0000","last_peered":"2024-09-16T10:35:35.247082+0000","last_clean":"2024-09-16T10:35:35.247082+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:35.247082+0000","last_undegraded":"2024-09-16T10:35:35.247082+0000","last_fullsized":"2024-09-16T10:35:35.247082+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246109+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:35.246109+0000","last_peered":"2024-09-16T10:35:35.246109+0000","last_clean":"2024-09-16T10:35:35.246109+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:35.246109+0000","last_undegraded":"2024-09-16T10:35:35.246109+0000","last_fullsized":"2024-09-16T10:35:35.246109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30023,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.309942+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:36.309942+0000","last_peered":"2024-09-16T10:35:36.309942+0000","last_clean":"2024-09-16T10:35:36.309942+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:36.309942+0000","last_undegraded":"2024-09-16T10:35:36.309942+0000","last_fullsized":"2024-09-16T10:35:36.309942+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":110,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.254700+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:35.254700+0000","last_peered":"2024-09-16T10:35:35.254700+0000","last_clean":"2024-09-16T10:35:35.254700+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:35.254700+0000","last_undegraded":"2024-09-16T10:35:35.254700+0000","last_fullsized":"2024-09-16T10:35:35.254700+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246175+0000","last_change":"2024-09-16T10:35:35.246175+0000","last_active":"2024-09-16T10:35:35.246175+0000","last_peered":"2024-09-16T10:35:35.246175+0000","last_clean":"2024-09-16T10:35:35.246175+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:35.246175+0000","last_undegraded":"2024-09-16T10:35:35.246175+0000","last_fullsized":"2024-09-16T10:35:35.246175+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":90,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247014+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:35.247014+0000","last_peered":"2024-09-16T10:35:35.247014+0000","last_clean":"2024-09-16T10:35:35.247014+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:35.247014+0000","last_undegraded":"2024-09-16T10:35:35.247014+0000","last_fullsized":"2024-09-16T10:35:35.247014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29090,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.245962+0000","last_change":"2024-09-16T10:35:35.245962+0000","last_active":"2024-09-16T10:35:35.245962+0000","last_peered":"2024-09-16T10:35:35.245962+0000","last_clean":"2024-09-16T10:35:35.245962+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:35.245962+0000","last_undegraded":"2024-09-16T10:35:35.245962+0000","last_fullsized":"2024-09-16T10:35:35.245962+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":21992,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:39.313716+0000","last_change":"2024-09-16T10:35:34.794858+0000","last_active":"2024-09-16T10:35:39.313716+0000","last_peered":"2024-09-16T10:35:39.313716+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:39.313716+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":929,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247245+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:35.247245+0000","last_peered":"2024-09-16T10:35:35.247245+0000","last_clean":"2024-09-16T10:35:35.247245+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:35.247245+0000","last_undegraded":"2024-09-16T10:35:35.247245+0000","last_fullsized":"2024-09-16T10:35:35.247245+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21210,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247395+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:35.247395+0000","last_peered":"2024-09-16T10:35:35.247395+0000","last_clean":"2024-09-16T10:35:35.247395+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:35.247395+0000","last_undegraded":"2024-09-16T10:35:35.247395+0000","last_fullsized":"2024-09-16T10:35:35.247395+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246102+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:35.246102+0000","last_peered":"2024-09-16T10:35:35.246102+0000","last_clean":"2024-09-16T10:35:35.246102+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:35.246102+0000","last_undegraded":"2024-09-16T10:35:35.246102+0000","last_fullsized":"2024-09-16T10:35:35.246102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24517,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246820+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:35.246820+0000","last_peered":"2024-09-16T10:35:35.246820+0000","last_clean":"2024-09-16T10:35:35.246820+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:35.246820+0000","last_undegraded":"2024-09-16T10:35:35.246820+0000","last_fullsized":"2024-09-16T10:35:35.246820+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247326+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:35.247326+0000","last_peered":"2024-09-16T10:35:35.247326+0000","last_clean":"2024-09-16T10:35:35.247326+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:35.247326+0000","last_undegraded":"2024-09-16T10:35:35.247326+0000","last_fullsized":"2024-09-16T10:35:35.247326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247481+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:35.247481+0000","last_peered":"2024-09-16T10:35:35.247481+0000","last_clean":"2024-09-16T10:35:35.247481+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:35.247481+0000","last_undegraded":"2024-09-16T10:35:35.247481+0000","last_fullsized":"2024-09-16T10:35:35.247481+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21155,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247180+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:35.247180+0000","last_peered":"2024-09-16T10:35:35.247180+0000","last_clean":"2024-09-16T10:35:35.247180+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:35.247180+0000","last_undegraded":"2024-09-16T10:35:35.247180+0000","last_fullsized":"2024-09-16T10:35:35.247180+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43825,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104722432,"data_stored":104707980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019038,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381384,"kb_used_data":34752,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990456,"statfs":{"total":96636764160,"available":96246226944,"internally_reserved":0,"allocated":35586048,"data_stored":35244299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.214},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.24099999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691982,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491536,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880304,"statfs":{"total":96636764160,"available":96133431296,"internally_reserved":0,"allocated":532480,"data_stored":175815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.189},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27300000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215265,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523376,"kb_used_data":68512,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848464,"statfs":{"total":96636764160,"available":96100827136,"internally_reserved":0,"allocated":70156288,"data_stored":69810463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:41.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:41.597 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:41.598 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:41.612 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-16T10:35:40.783510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43826,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1396296,"kb_used_data":103784,"kb_used_omap":12,"kb_used_meta":1292467,"kb_avail":281719224,"statfs":{"total":289910292480,"available":288480485376,"internally_reserved":0,"allocated":106274816,"data_stored":105230577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1323486210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2224,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001457"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28951,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.252058+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:35.252058+0000","last_peered":"2024-09-16T10:35:35.252058+0000","last_clean":"2024-09-16T10:35:35.252058+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:35.252058+0000","last_undegraded":"2024-09-16T10:35:35.252058+0000","last_fullsized":"2024-09-16T10:35:35.252058+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20641,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246198+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:35.246198+0000","last_peered":"2024-09-16T10:35:35.246198+0000","last_clean":"2024-09-16T10:35:35.246198+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:35.246198+0000","last_undegraded":"2024-09-16T10:35:35.246198+0000","last_fullsized":"2024-09-16T10:35:35.246198+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246728+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:35.246728+0000","last_peered":"2024-09-16T10:35:35.246728+0000","last_clean":"2024-09-16T10:35:35.246728+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:35.246728+0000","last_undegraded":"2024-09-16T10:35:35.246728+0000","last_fullsized":"2024-09-16T10:35:35.246728+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20540,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246021+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:35.246021+0000","last_peered":"2024-09-16T10:35:35.246021+0000","last_clean":"2024-09-16T10:35:35.246021+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:35.246021+0000","last_undegraded":"2024-09-16T10:35:35.246021+0000","last_fullsized":"2024-09-16T10:35:35.246021+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20683,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248479+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:35.248479+0000","last_peered":"2024-09-16T10:35:35.248479+0000","last_clean":"2024-09-16T10:35:35.248479+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:35.248479+0000","last_undegraded":"2024-09-16T10:35:35.248479+0000","last_fullsized":"2024-09-16T10:35:35.248479+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20722,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246758+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:35.246758+0000","last_peered":"2024-09-16T10:35:35.246758+0000","last_clean":"2024-09-16T10:35:35.246758+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:35.246758+0000","last_undegraded":"2024-09-16T10:35:35.246758+0000","last_fullsized":"2024-09-16T10:35:35.246758+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21347,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:35.248771+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:35.248771+0000","last_peered":"2024-09-16T10:35:35.248771+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:35.248771+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20587,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248438+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:35.248438+0000","last_peered":"2024-09-16T10:35:35.248438+0000","last_clean":"2024-09-16T10:35:35.248438+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:35.248438+0000","last_undegraded":"2024-09-16T10:35:35.248438+0000","last_fullsized":"2024-09-16T10:35:35.248438+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246752+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:35.246752+0000","last_peered":"2024-09-16T10:35:35.246752+0000","last_clean":"2024-09-16T10:35:35.246752+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:35.246752+0000","last_undegraded":"2024-09-16T10:35:35.246752+0000","last_fullsized":"2024-09-16T10:35:35.246752+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28986,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247082+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:35.247082+0000","last_peered":"2024-09-16T10:35:35.247082+0000","last_clean":"2024-09-16T10:35:35.247082+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:35.247082+0000","last_undegraded":"2024-09-16T10:35:35.247082+0000","last_fullsized":"2024-09-16T10:35:35.247082+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246109+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:35.246109+0000","last_peered":"2024-09-16T10:35:35.246109+0000","last_clean":"2024-09-16T10:35:35.246109+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:35.246109+0000","last_undegraded":"2024-09-16T10:35:35.246109+0000","last_fullsized":"2024-09-16T10:35:35.246109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30023,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.309942+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:36.309942+0000","last_peered":"2024-09-16T10:35:36.309942+0000","last_clean":"2024-09-16T10:35:36.309942+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:36.309942+0000","last_undegraded":"2024-09-16T10:35:36.309942+0000","last_fullsized":"2024-09-16T10:35:36.309942+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":110,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.254700+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:35.254700+0000","last_peered":"2024-09-16T10:35:35.254700+0000","last_clean":"2024-09-16T10:35:35.254700+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:35.254700+0000","last_undegraded":"2024-09-16T10:35:35.254700+0000","last_fullsized":"2024-09-16T10:35:35.254700+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246175+0000","last_change":"2024-09-16T10:35:35.246175+0000","last_active":"2024-09-16T10:35:35.246175+0000","last_peered":"2024-09-16T10:35:35.246175+0000","last_clean":"2024-09-16T10:35:35.246175+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:35.246175+0000","last_undegraded":"2024-09-16T10:35:35.246175+0000","last_fullsized":"2024-09-16T10:35:35.246175+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":90,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247014+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:35.247014+0000","last_peered":"2024-09-16T10:35:35.247014+0000","last_clean":"2024-09-16T10:35:35.247014+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:35.247014+0000","last_undegraded":"2024-09-16T10:35:35.247014+0000","last_fullsized":"2024-09-16T10:35:35.247014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29090,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.245962+0000","last_change":"2024-09-16T10:35:35.245962+0000","last_active":"2024-09-16T10:35:35.245962+0000","last_peered":"2024-09-16T10:35:35.245962+0000","last_clean":"2024-09-16T10:35:35.245962+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:35.245962+0000","last_undegraded":"2024-09-16T10:35:35.245962+0000","last_fullsized":"2024-09-16T10:35:35.245962+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":21992,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:39.313716+0000","last_change":"2024-09-16T10:35:34.794858+0000","last_active":"2024-09-16T10:35:39.313716+0000","last_peered":"2024-09-16T10:35:39.313716+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:39.313716+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":929,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247245+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:35.247245+0000","last_peered":"2024-09-16T10:35:35.247245+0000","last_clean":"2024-09-16T10:35:35.247245+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:35.247245+0000","last_undegraded":"2024-09-16T10:35:35.247245+0000","last_fullsized":"2024-09-16T10:35:35.247245+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21210,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247395+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:35.247395+0000","last_peered":"2024-09-16T10:35:35.247395+0000","last_clean":"2024-09-16T10:35:35.247395+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:35.247395+0000","last_undegraded":"2024-09-16T10:35:35.247395+0000","last_fullsized":"2024-09-16T10:35:35.247395+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246102+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:35.246102+0000","last_peered":"2024-09-16T10:35:35.246102+0000","last_clean":"2024-09-16T10:35:35.246102+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:35.246102+0000","last_undegraded":"2024-09-16T10:35:35.246102+0000","last_fullsized":"2024-09-16T10:35:35.246102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24517,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246820+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:35.246820+0000","last_peered":"2024-09-16T10:35:35.246820+0000","last_clean":"2024-09-16T10:35:35.246820+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:35.246820+0000","last_undegraded":"2024-09-16T10:35:35.246820+0000","last_fullsized":"2024-09-16T10:35:35.246820+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247326+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:35.247326+0000","last_peered":"2024-09-16T10:35:35.247326+0000","last_clean":"2024-09-16T10:35:35.247326+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:35.247326+0000","last_undegraded":"2024-09-16T10:35:35.247326+0000","last_fullsized":"2024-09-16T10:35:35.247326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247481+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:35.247481+0000","last_peered":"2024-09-16T10:35:35.247481+0000","last_clean":"2024-09-16T10:35:35.247481+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:35.247481+0000","last_undegraded":"2024-09-16T10:35:35.247481+0000","last_fullsized":"2024-09-16T10:35:35.247481+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21155,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247180+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:35.247180+0000","last_peered":"2024-09-16T10:35:35.247180+0000","last_clean":"2024-09-16T10:35:35.247180+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:35.247180+0000","last_undegraded":"2024-09-16T10:35:35.247180+0000","last_fullsized":"2024-09-16T10:35:35.247180+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43825,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104722432,"data_stored":104707980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019038,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381384,"kb_used_data":34752,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990456,"statfs":{"total":96636764160,"available":96246226944,"internally_reserved":0,"allocated":35586048,"data_stored":35244299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.214},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.24099999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691982,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491536,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880304,"statfs":{"total":96636764160,"available":96133431296,"internally_reserved":0,"allocated":532480,"data_stored":175815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.189},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27300000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215265,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523376,"kb_used_data":68512,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848464,"statfs":{"total":96636764160,"available":96100827136,"internally_reserved":0,"allocated":70156288,"data_stored":69810463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:41.613 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:41.613 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21347, 'reported_epoch': 61, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:35.248771+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:35:35.248771+0000', 'last_peered': '2024-09-16T10:35:35.248771+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:35.248771+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:41.613 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:41.613 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 21992, 'reported_epoch': 62, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:39.313716+0000', 'last_change': '2024-09-16T10:35:34.794858+0000', 'last_active': '2024-09-16T10:35:39.313716+0000', 'last_peered': '2024-09-16T10:35:39.313716+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:35:39.313716+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3232, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 929, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:41.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:42.035 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:42.035 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7521/40 objects degraded (18802.500%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":830,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":62,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1429807104,"bytes_avail":288480485376,"bytes_total":289910292480,"degraded_objects":7521,"degraded_total":40,"degraded_ratio":188.02500000000001,"recovering_objects_per_sec":277,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2224,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:42.051 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:42.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:42.429 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:42.430 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:42.443 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-16T10:35:40.783510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43826,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1396296,"kb_used_data":103784,"kb_used_omap":12,"kb_used_meta":1292467,"kb_avail":281719224,"statfs":{"total":289910292480,"available":288480485376,"internally_reserved":0,"allocated":106274816,"data_stored":105230577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1323486210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2224,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2224,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001457"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28951,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.252058+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:35.252058+0000","last_peered":"2024-09-16T10:35:35.252058+0000","last_clean":"2024-09-16T10:35:35.252058+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:35.252058+0000","last_undegraded":"2024-09-16T10:35:35.252058+0000","last_fullsized":"2024-09-16T10:35:35.252058+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20641,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246198+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:35.246198+0000","last_peered":"2024-09-16T10:35:35.246198+0000","last_clean":"2024-09-16T10:35:35.246198+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:35.246198+0000","last_undegraded":"2024-09-16T10:35:35.246198+0000","last_fullsized":"2024-09-16T10:35:35.246198+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246728+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:35.246728+0000","last_peered":"2024-09-16T10:35:35.246728+0000","last_clean":"2024-09-16T10:35:35.246728+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:35.246728+0000","last_undegraded":"2024-09-16T10:35:35.246728+0000","last_fullsized":"2024-09-16T10:35:35.246728+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20540,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246021+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:35.246021+0000","last_peered":"2024-09-16T10:35:35.246021+0000","last_clean":"2024-09-16T10:35:35.246021+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:35.246021+0000","last_undegraded":"2024-09-16T10:35:35.246021+0000","last_fullsized":"2024-09-16T10:35:35.246021+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20683,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248479+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:35.248479+0000","last_peered":"2024-09-16T10:35:35.248479+0000","last_clean":"2024-09-16T10:35:35.248479+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:35.248479+0000","last_undegraded":"2024-09-16T10:35:35.248479+0000","last_fullsized":"2024-09-16T10:35:35.248479+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20722,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246758+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:35.246758+0000","last_peered":"2024-09-16T10:35:35.246758+0000","last_clean":"2024-09-16T10:35:35.246758+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:35.246758+0000","last_undegraded":"2024-09-16T10:35:35.246758+0000","last_fullsized":"2024-09-16T10:35:35.246758+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21347,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:35.248771+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:35.248771+0000","last_peered":"2024-09-16T10:35:35.248771+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:35.248771+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20587,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248438+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:35.248438+0000","last_peered":"2024-09-16T10:35:35.248438+0000","last_clean":"2024-09-16T10:35:35.248438+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:35.248438+0000","last_undegraded":"2024-09-16T10:35:35.248438+0000","last_fullsized":"2024-09-16T10:35:35.248438+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246752+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:35.246752+0000","last_peered":"2024-09-16T10:35:35.246752+0000","last_clean":"2024-09-16T10:35:35.246752+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:35.246752+0000","last_undegraded":"2024-09-16T10:35:35.246752+0000","last_fullsized":"2024-09-16T10:35:35.246752+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28986,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247082+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:35.247082+0000","last_peered":"2024-09-16T10:35:35.247082+0000","last_clean":"2024-09-16T10:35:35.247082+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:35.247082+0000","last_undegraded":"2024-09-16T10:35:35.247082+0000","last_fullsized":"2024-09-16T10:35:35.247082+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246109+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:35.246109+0000","last_peered":"2024-09-16T10:35:35.246109+0000","last_clean":"2024-09-16T10:35:35.246109+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:35.246109+0000","last_undegraded":"2024-09-16T10:35:35.246109+0000","last_fullsized":"2024-09-16T10:35:35.246109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30023,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.309942+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:36.309942+0000","last_peered":"2024-09-16T10:35:36.309942+0000","last_clean":"2024-09-16T10:35:36.309942+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:36.309942+0000","last_undegraded":"2024-09-16T10:35:36.309942+0000","last_fullsized":"2024-09-16T10:35:36.309942+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":110,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.254700+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:35.254700+0000","last_peered":"2024-09-16T10:35:35.254700+0000","last_clean":"2024-09-16T10:35:35.254700+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:35.254700+0000","last_undegraded":"2024-09-16T10:35:35.254700+0000","last_fullsized":"2024-09-16T10:35:35.254700+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29124,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246175+0000","last_change":"2024-09-16T10:35:35.246175+0000","last_active":"2024-09-16T10:35:35.246175+0000","last_peered":"2024-09-16T10:35:35.246175+0000","last_clean":"2024-09-16T10:35:35.246175+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:35.246175+0000","last_undegraded":"2024-09-16T10:35:35.246175+0000","last_fullsized":"2024-09-16T10:35:35.246175+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":90,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247014+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:35.247014+0000","last_peered":"2024-09-16T10:35:35.247014+0000","last_clean":"2024-09-16T10:35:35.247014+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:35.247014+0000","last_undegraded":"2024-09-16T10:35:35.247014+0000","last_fullsized":"2024-09-16T10:35:35.247014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29090,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.245962+0000","last_change":"2024-09-16T10:35:35.245962+0000","last_active":"2024-09-16T10:35:35.245962+0000","last_peered":"2024-09-16T10:35:35.245962+0000","last_clean":"2024-09-16T10:35:35.245962+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:35.245962+0000","last_undegraded":"2024-09-16T10:35:35.245962+0000","last_fullsized":"2024-09-16T10:35:35.245962+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":21992,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:39.313716+0000","last_change":"2024-09-16T10:35:34.794858+0000","last_active":"2024-09-16T10:35:39.313716+0000","last_peered":"2024-09-16T10:35:39.313716+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:39.313716+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":929,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247245+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:35.247245+0000","last_peered":"2024-09-16T10:35:35.247245+0000","last_clean":"2024-09-16T10:35:35.247245+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:35.247245+0000","last_undegraded":"2024-09-16T10:35:35.247245+0000","last_fullsized":"2024-09-16T10:35:35.247245+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21210,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247395+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:35.247395+0000","last_peered":"2024-09-16T10:35:35.247395+0000","last_clean":"2024-09-16T10:35:35.247395+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:35.247395+0000","last_undegraded":"2024-09-16T10:35:35.247395+0000","last_fullsized":"2024-09-16T10:35:35.247395+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246102+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:35.246102+0000","last_peered":"2024-09-16T10:35:35.246102+0000","last_clean":"2024-09-16T10:35:35.246102+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:35.246102+0000","last_undegraded":"2024-09-16T10:35:35.246102+0000","last_fullsized":"2024-09-16T10:35:35.246102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24517,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246820+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:35.246820+0000","last_peered":"2024-09-16T10:35:35.246820+0000","last_clean":"2024-09-16T10:35:35.246820+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:35.246820+0000","last_undegraded":"2024-09-16T10:35:35.246820+0000","last_fullsized":"2024-09-16T10:35:35.246820+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247326+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:35.247326+0000","last_peered":"2024-09-16T10:35:35.247326+0000","last_clean":"2024-09-16T10:35:35.247326+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:35.247326+0000","last_undegraded":"2024-09-16T10:35:35.247326+0000","last_fullsized":"2024-09-16T10:35:35.247326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247481+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:35.247481+0000","last_peered":"2024-09-16T10:35:35.247481+0000","last_clean":"2024-09-16T10:35:35.247481+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:35.247481+0000","last_undegraded":"2024-09-16T10:35:35.247481+0000","last_fullsized":"2024-09-16T10:35:35.247481+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21155,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247180+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:35.247180+0000","last_peered":"2024-09-16T10:35:35.247180+0000","last_clean":"2024-09-16T10:35:35.247180+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:35.247180+0000","last_undegraded":"2024-09-16T10:35:35.247180+0000","last_fullsized":"2024-09-16T10:35:35.247180+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43825,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104722432,"data_stored":104707980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019038,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381384,"kb_used_data":34752,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990456,"statfs":{"total":96636764160,"available":96246226944,"internally_reserved":0,"allocated":35586048,"data_stored":35244299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.214},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.24099999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691982,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":491536,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":491003,"kb_avail":93880304,"statfs":{"total":96636764160,"available":96133431296,"internally_reserved":0,"allocated":532480,"data_stored":175815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":502787754},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.189},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27300000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215265,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523376,"kb_used_data":68512,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848464,"statfs":{"total":96636764160,"available":96100827136,"internally_reserved":0,"allocated":70156288,"data_stored":69810463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.24299999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:45.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:45.796 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:45.797 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:45.811 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-16T10:35:44.784220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44970,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1388104,"kb_used_data":103784,"kb_used_omap":12,"kb_used_meta":1284275,"kb_avail":281727416,"statfs":{"total":289910292480,"available":288488873984,"internally_reserved":0,"allocated":106274816,"data_stored":105230577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1315097602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.539136"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28951,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.252058+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:35.252058+0000","last_peered":"2024-09-16T10:35:35.252058+0000","last_clean":"2024-09-16T10:35:35.252058+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:35.252058+0000","last_undegraded":"2024-09-16T10:35:35.252058+0000","last_fullsized":"2024-09-16T10:35:35.252058+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20641,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246198+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:35.246198+0000","last_peered":"2024-09-16T10:35:35.246198+0000","last_clean":"2024-09-16T10:35:35.246198+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:35.246198+0000","last_undegraded":"2024-09-16T10:35:35.246198+0000","last_fullsized":"2024-09-16T10:35:35.246198+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246728+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:35.246728+0000","last_peered":"2024-09-16T10:35:35.246728+0000","last_clean":"2024-09-16T10:35:35.246728+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:35.246728+0000","last_undegraded":"2024-09-16T10:35:35.246728+0000","last_fullsized":"2024-09-16T10:35:35.246728+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20540,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246021+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:35.246021+0000","last_peered":"2024-09-16T10:35:35.246021+0000","last_clean":"2024-09-16T10:35:35.246021+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:35.246021+0000","last_undegraded":"2024-09-16T10:35:35.246021+0000","last_fullsized":"2024-09-16T10:35:35.246021+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20683,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248479+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:35.248479+0000","last_peered":"2024-09-16T10:35:35.248479+0000","last_clean":"2024-09-16T10:35:35.248479+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:35.248479+0000","last_undegraded":"2024-09-16T10:35:35.248479+0000","last_fullsized":"2024-09-16T10:35:35.248479+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20722,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246758+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:35.246758+0000","last_peered":"2024-09-16T10:35:35.246758+0000","last_clean":"2024-09-16T10:35:35.246758+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:35.246758+0000","last_undegraded":"2024-09-16T10:35:35.246758+0000","last_fullsized":"2024-09-16T10:35:35.246758+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21347,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:35.248771+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:35.248771+0000","last_peered":"2024-09-16T10:35:35.248771+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:35.248771+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20587,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248438+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:35.248438+0000","last_peered":"2024-09-16T10:35:35.248438+0000","last_clean":"2024-09-16T10:35:35.248438+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:35.248438+0000","last_undegraded":"2024-09-16T10:35:35.248438+0000","last_fullsized":"2024-09-16T10:35:35.248438+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246752+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:35.246752+0000","last_peered":"2024-09-16T10:35:35.246752+0000","last_clean":"2024-09-16T10:35:35.246752+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:35.246752+0000","last_undegraded":"2024-09-16T10:35:35.246752+0000","last_fullsized":"2024-09-16T10:35:35.246752+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28986,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247082+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:35.247082+0000","last_peered":"2024-09-16T10:35:35.247082+0000","last_clean":"2024-09-16T10:35:35.247082+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:35.247082+0000","last_undegraded":"2024-09-16T10:35:35.247082+0000","last_fullsized":"2024-09-16T10:35:35.247082+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246109+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:35.246109+0000","last_peered":"2024-09-16T10:35:35.246109+0000","last_clean":"2024-09-16T10:35:35.246109+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:35.246109+0000","last_undegraded":"2024-09-16T10:35:35.246109+0000","last_fullsized":"2024-09-16T10:35:35.246109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30023,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.309942+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:36.309942+0000","last_peered":"2024-09-16T10:35:36.309942+0000","last_clean":"2024-09-16T10:35:36.309942+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:36.309942+0000","last_undegraded":"2024-09-16T10:35:36.309942+0000","last_fullsized":"2024-09-16T10:35:36.309942+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":110,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.254700+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:35.254700+0000","last_peered":"2024-09-16T10:35:35.254700+0000","last_clean":"2024-09-16T10:35:35.254700+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:35.254700+0000","last_undegraded":"2024-09-16T10:35:35.254700+0000","last_fullsized":"2024-09-16T10:35:35.254700+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29126,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:38.436222+0000","last_change":"2024-09-16T10:35:38.436222+0000","last_active":"2024-09-16T10:35:38.436222+0000","last_peered":"2024-09-16T10:35:38.436222+0000","last_clean":"2024-09-16T10:35:38.436222+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:38.436222+0000","last_undegraded":"2024-09-16T10:35:38.436222+0000","last_fullsized":"2024-09-16T10:35:38.436222+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":90,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247014+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:35.247014+0000","last_peered":"2024-09-16T10:35:35.247014+0000","last_clean":"2024-09-16T10:35:35.247014+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:35.247014+0000","last_undegraded":"2024-09-16T10:35:35.247014+0000","last_fullsized":"2024-09-16T10:35:35.247014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29091,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:36.250037+0000","last_change":"2024-09-16T10:35:39.462868+0000","last_active":"2024-09-16T10:35:36.250037+0000","last_peered":"2024-09-16T10:35:36.250037+0000","last_clean":"2024-09-16T10:35:36.250037+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:36.250037+0000","last_undegraded":"2024-09-16T10:35:36.250037+0000","last_fullsized":"2024-09-16T10:35:36.250037+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":24280,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:44.311782+0000","last_change":"2024-09-16T10:35:34.794858+0000","last_active":"2024-09-16T10:35:44.311782+0000","last_peered":"2024-09-16T10:35:44.311782+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:44.311782+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247245+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:35.247245+0000","last_peered":"2024-09-16T10:35:35.247245+0000","last_clean":"2024-09-16T10:35:35.247245+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:35.247245+0000","last_undegraded":"2024-09-16T10:35:35.247245+0000","last_fullsized":"2024-09-16T10:35:35.247245+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21210,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247395+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:35.247395+0000","last_peered":"2024-09-16T10:35:35.247395+0000","last_clean":"2024-09-16T10:35:35.247395+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:35.247395+0000","last_undegraded":"2024-09-16T10:35:35.247395+0000","last_fullsized":"2024-09-16T10:35:35.247395+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246102+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:35.246102+0000","last_peered":"2024-09-16T10:35:35.246102+0000","last_clean":"2024-09-16T10:35:35.246102+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:35.246102+0000","last_undegraded":"2024-09-16T10:35:35.246102+0000","last_fullsized":"2024-09-16T10:35:35.246102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24518,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.251503+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:36.251503+0000","last_peered":"2024-09-16T10:35:36.251503+0000","last_clean":"2024-09-16T10:35:36.251503+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:36.251503+0000","last_undegraded":"2024-09-16T10:35:36.251503+0000","last_fullsized":"2024-09-16T10:35:36.251503+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247326+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:35.247326+0000","last_peered":"2024-09-16T10:35:35.247326+0000","last_clean":"2024-09-16T10:35:35.247326+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:35.247326+0000","last_undegraded":"2024-09-16T10:35:35.247326+0000","last_fullsized":"2024-09-16T10:35:35.247326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247481+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:35.247481+0000","last_peered":"2024-09-16T10:35:35.247481+0000","last_clean":"2024-09-16T10:35:35.247481+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:35.247481+0000","last_undegraded":"2024-09-16T10:35:35.247481+0000","last_fullsized":"2024-09-16T10:35:35.247481+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21155,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247180+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:35.247180+0000","last_peered":"2024-09-16T10:35:35.247180+0000","last_clean":"2024-09-16T10:35:35.247180+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:35.247180+0000","last_undegraded":"2024-09-16T10:35:35.247180+0000","last_fullsized":"2024-09-16T10:35:35.247180+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44969,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104722432,"data_stored":104707980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019039,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381384,"kb_used_data":34752,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990456,"statfs":{"total":96636764160,"available":96246226944,"internally_reserved":0,"allocated":35586048,"data_stored":35244299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.26200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.186}]}]},{"osd":1,"up_from":40,"seq":171798691983,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483344,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888496,"statfs":{"total":96636764160,"available":96141819904,"internally_reserved":0,"allocated":532480,"data_stored":175815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16800000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.20399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.25},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.26400000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523376,"kb_used_data":68512,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848464,"statfs":{"total":96636764160,"available":96100827136,"internally_reserved":0,"allocated":70156288,"data_stored":69810463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.187},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.215}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.247},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.25700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:45.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:46.171 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:46.171 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:46.186 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-16T10:35:44.784220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44970,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1388104,"kb_used_data":103784,"kb_used_omap":12,"kb_used_meta":1284275,"kb_avail":281727416,"statfs":{"total":289910292480,"available":288488873984,"internally_reserved":0,"allocated":106274816,"data_stored":105230577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1315097602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.539136"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28951,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.252058+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:35.252058+0000","last_peered":"2024-09-16T10:35:35.252058+0000","last_clean":"2024-09-16T10:35:35.252058+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:35.252058+0000","last_undegraded":"2024-09-16T10:35:35.252058+0000","last_fullsized":"2024-09-16T10:35:35.252058+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20641,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246198+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:35.246198+0000","last_peered":"2024-09-16T10:35:35.246198+0000","last_clean":"2024-09-16T10:35:35.246198+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:35.246198+0000","last_undegraded":"2024-09-16T10:35:35.246198+0000","last_fullsized":"2024-09-16T10:35:35.246198+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246728+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:35.246728+0000","last_peered":"2024-09-16T10:35:35.246728+0000","last_clean":"2024-09-16T10:35:35.246728+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:35.246728+0000","last_undegraded":"2024-09-16T10:35:35.246728+0000","last_fullsized":"2024-09-16T10:35:35.246728+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20540,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246021+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:35.246021+0000","last_peered":"2024-09-16T10:35:35.246021+0000","last_clean":"2024-09-16T10:35:35.246021+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:35.246021+0000","last_undegraded":"2024-09-16T10:35:35.246021+0000","last_fullsized":"2024-09-16T10:35:35.246021+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20683,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248479+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:35.248479+0000","last_peered":"2024-09-16T10:35:35.248479+0000","last_clean":"2024-09-16T10:35:35.248479+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:35.248479+0000","last_undegraded":"2024-09-16T10:35:35.248479+0000","last_fullsized":"2024-09-16T10:35:35.248479+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20722,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246758+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:35.246758+0000","last_peered":"2024-09-16T10:35:35.246758+0000","last_clean":"2024-09-16T10:35:35.246758+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:35.246758+0000","last_undegraded":"2024-09-16T10:35:35.246758+0000","last_fullsized":"2024-09-16T10:35:35.246758+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21347,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:35.248771+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:35.248771+0000","last_peered":"2024-09-16T10:35:35.248771+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:35.248771+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20587,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248438+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:35.248438+0000","last_peered":"2024-09-16T10:35:35.248438+0000","last_clean":"2024-09-16T10:35:35.248438+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:35.248438+0000","last_undegraded":"2024-09-16T10:35:35.248438+0000","last_fullsized":"2024-09-16T10:35:35.248438+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246752+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:35.246752+0000","last_peered":"2024-09-16T10:35:35.246752+0000","last_clean":"2024-09-16T10:35:35.246752+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:35.246752+0000","last_undegraded":"2024-09-16T10:35:35.246752+0000","last_fullsized":"2024-09-16T10:35:35.246752+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28986,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247082+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:35.247082+0000","last_peered":"2024-09-16T10:35:35.247082+0000","last_clean":"2024-09-16T10:35:35.247082+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:35.247082+0000","last_undegraded":"2024-09-16T10:35:35.247082+0000","last_fullsized":"2024-09-16T10:35:35.247082+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246109+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:35.246109+0000","last_peered":"2024-09-16T10:35:35.246109+0000","last_clean":"2024-09-16T10:35:35.246109+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:35.246109+0000","last_undegraded":"2024-09-16T10:35:35.246109+0000","last_fullsized":"2024-09-16T10:35:35.246109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30023,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.309942+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:36.309942+0000","last_peered":"2024-09-16T10:35:36.309942+0000","last_clean":"2024-09-16T10:35:36.309942+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:36.309942+0000","last_undegraded":"2024-09-16T10:35:36.309942+0000","last_fullsized":"2024-09-16T10:35:36.309942+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":110,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.254700+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:35.254700+0000","last_peered":"2024-09-16T10:35:35.254700+0000","last_clean":"2024-09-16T10:35:35.254700+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:35.254700+0000","last_undegraded":"2024-09-16T10:35:35.254700+0000","last_fullsized":"2024-09-16T10:35:35.254700+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29126,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:38.436222+0000","last_change":"2024-09-16T10:35:38.436222+0000","last_active":"2024-09-16T10:35:38.436222+0000","last_peered":"2024-09-16T10:35:38.436222+0000","last_clean":"2024-09-16T10:35:38.436222+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:38.436222+0000","last_undegraded":"2024-09-16T10:35:38.436222+0000","last_fullsized":"2024-09-16T10:35:38.436222+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":90,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247014+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:35.247014+0000","last_peered":"2024-09-16T10:35:35.247014+0000","last_clean":"2024-09-16T10:35:35.247014+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:35.247014+0000","last_undegraded":"2024-09-16T10:35:35.247014+0000","last_fullsized":"2024-09-16T10:35:35.247014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29091,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:36.250037+0000","last_change":"2024-09-16T10:35:39.462868+0000","last_active":"2024-09-16T10:35:36.250037+0000","last_peered":"2024-09-16T10:35:36.250037+0000","last_clean":"2024-09-16T10:35:36.250037+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:36.250037+0000","last_undegraded":"2024-09-16T10:35:36.250037+0000","last_fullsized":"2024-09-16T10:35:36.250037+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":24280,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:44.311782+0000","last_change":"2024-09-16T10:35:34.794858+0000","last_active":"2024-09-16T10:35:44.311782+0000","last_peered":"2024-09-16T10:35:44.311782+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:44.311782+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247245+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:35.247245+0000","last_peered":"2024-09-16T10:35:35.247245+0000","last_clean":"2024-09-16T10:35:35.247245+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:35.247245+0000","last_undegraded":"2024-09-16T10:35:35.247245+0000","last_fullsized":"2024-09-16T10:35:35.247245+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21210,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247395+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:35.247395+0000","last_peered":"2024-09-16T10:35:35.247395+0000","last_clean":"2024-09-16T10:35:35.247395+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:35.247395+0000","last_undegraded":"2024-09-16T10:35:35.247395+0000","last_fullsized":"2024-09-16T10:35:35.247395+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246102+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:35.246102+0000","last_peered":"2024-09-16T10:35:35.246102+0000","last_clean":"2024-09-16T10:35:35.246102+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:35.246102+0000","last_undegraded":"2024-09-16T10:35:35.246102+0000","last_fullsized":"2024-09-16T10:35:35.246102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24518,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.251503+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:36.251503+0000","last_peered":"2024-09-16T10:35:36.251503+0000","last_clean":"2024-09-16T10:35:36.251503+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:36.251503+0000","last_undegraded":"2024-09-16T10:35:36.251503+0000","last_fullsized":"2024-09-16T10:35:36.251503+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247326+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:35.247326+0000","last_peered":"2024-09-16T10:35:35.247326+0000","last_clean":"2024-09-16T10:35:35.247326+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:35.247326+0000","last_undegraded":"2024-09-16T10:35:35.247326+0000","last_fullsized":"2024-09-16T10:35:35.247326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247481+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:35.247481+0000","last_peered":"2024-09-16T10:35:35.247481+0000","last_clean":"2024-09-16T10:35:35.247481+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:35.247481+0000","last_undegraded":"2024-09-16T10:35:35.247481+0000","last_fullsized":"2024-09-16T10:35:35.247481+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21155,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247180+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:35.247180+0000","last_peered":"2024-09-16T10:35:35.247180+0000","last_clean":"2024-09-16T10:35:35.247180+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:35.247180+0000","last_undegraded":"2024-09-16T10:35:35.247180+0000","last_fullsized":"2024-09-16T10:35:35.247180+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44969,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104722432,"data_stored":104707980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019039,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381384,"kb_used_data":34752,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990456,"statfs":{"total":96636764160,"available":96246226944,"internally_reserved":0,"allocated":35586048,"data_stored":35244299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.26200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.186}]}]},{"osd":1,"up_from":40,"seq":171798691983,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483344,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888496,"statfs":{"total":96636764160,"available":96141819904,"internally_reserved":0,"allocated":532480,"data_stored":175815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16800000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.20399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.25},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.26400000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523376,"kb_used_data":68512,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848464,"statfs":{"total":96636764160,"available":96100827136,"internally_reserved":0,"allocated":70156288,"data_stored":69810463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.187},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.215}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.247},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.25700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:46.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:46.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21347, 'reported_epoch': 61, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:35.248771+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:35:35.248771+0000', 'last_peered': '2024-09-16T10:35:35.248771+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:35.248771+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:46.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:35:46.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29126, 'reported_epoch': 62, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:38.436222+0000', 'last_change': '2024-09-16T10:35:38.436222+0000', 'last_active': '2024-09-16T10:35:38.436222+0000', 'last_peered': '2024-09-16T10:35:38.436222+0000', 'last_clean': '2024-09-16T10:35:38.436222+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:35:38.436222+0000', 'last_undegraded': '2024-09-16T10:35:38.436222+0000', 'last_fullsized': '2024-09-16T10:35:38.436222+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:46.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:46.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29091, 'reported_epoch': 62, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:36.250037+0000', 'last_change': '2024-09-16T10:35:39.462868+0000', 'last_active': '2024-09-16T10:35:36.250037+0000', 'last_peered': '2024-09-16T10:35:36.250037+0000', 'last_clean': '2024-09-16T10:35:36.250037+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:36.250037+0000', 'last_undegraded': '2024-09-16T10:35:36.250037+0000', 'last_fullsized': '2024-09-16T10:35:36.250037+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:46.188 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:46.189 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 24280, 'reported_epoch': 62, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:44.311782+0000', 'last_change': '2024-09-16T10:35:34.794858+0000', 'last_active': '2024-09-16T10:35:44.311782+0000', 'last_peered': '2024-09-16T10:35:44.311782+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:35:44.311782+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2088, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2073, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2087}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:46.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:46.608 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:46.609 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6377/40 objects degraded (15942.500%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":835,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":62,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+clean+scrubbing","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1421418496,"bytes_avail":288488873984,"bytes_total":289910292480,"degraded_objects":6377,"degraded_total":40,"degraded_ratio":159.42500000000001,"recovering_objects_per_sec":225,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2155,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:46.623 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:46.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:46.962 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:46.962 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:46.976 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-09-16T10:35:46.784549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44970,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1388104,"kb_used_data":103784,"kb_used_omap":12,"kb_used_meta":1284275,"kb_avail":281727416,"statfs":{"total":289910292480,"available":288488873984,"internally_reserved":0,"allocated":106274816,"data_stored":105230577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1315097602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.535586"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28951,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.252058+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:35.252058+0000","last_peered":"2024-09-16T10:35:35.252058+0000","last_clean":"2024-09-16T10:35:35.252058+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:35.252058+0000","last_undegraded":"2024-09-16T10:35:35.252058+0000","last_fullsized":"2024-09-16T10:35:35.252058+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20641,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246198+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:35.246198+0000","last_peered":"2024-09-16T10:35:35.246198+0000","last_clean":"2024-09-16T10:35:35.246198+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:35.246198+0000","last_undegraded":"2024-09-16T10:35:35.246198+0000","last_fullsized":"2024-09-16T10:35:35.246198+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246728+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:35.246728+0000","last_peered":"2024-09-16T10:35:35.246728+0000","last_clean":"2024-09-16T10:35:35.246728+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:35.246728+0000","last_undegraded":"2024-09-16T10:35:35.246728+0000","last_fullsized":"2024-09-16T10:35:35.246728+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20540,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246021+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:35.246021+0000","last_peered":"2024-09-16T10:35:35.246021+0000","last_clean":"2024-09-16T10:35:35.246021+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:35.246021+0000","last_undegraded":"2024-09-16T10:35:35.246021+0000","last_fullsized":"2024-09-16T10:35:35.246021+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20683,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248479+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:35.248479+0000","last_peered":"2024-09-16T10:35:35.248479+0000","last_clean":"2024-09-16T10:35:35.248479+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:35.248479+0000","last_undegraded":"2024-09-16T10:35:35.248479+0000","last_fullsized":"2024-09-16T10:35:35.248479+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20722,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246758+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:35.246758+0000","last_peered":"2024-09-16T10:35:35.246758+0000","last_clean":"2024-09-16T10:35:35.246758+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:35.246758+0000","last_undegraded":"2024-09-16T10:35:35.246758+0000","last_fullsized":"2024-09-16T10:35:35.246758+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21347,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:35.248771+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:35.248771+0000","last_peered":"2024-09-16T10:35:35.248771+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:35.248771+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20587,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248438+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:35.248438+0000","last_peered":"2024-09-16T10:35:35.248438+0000","last_clean":"2024-09-16T10:35:35.248438+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:35.248438+0000","last_undegraded":"2024-09-16T10:35:35.248438+0000","last_fullsized":"2024-09-16T10:35:35.248438+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246752+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:35.246752+0000","last_peered":"2024-09-16T10:35:35.246752+0000","last_clean":"2024-09-16T10:35:35.246752+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:35.246752+0000","last_undegraded":"2024-09-16T10:35:35.246752+0000","last_fullsized":"2024-09-16T10:35:35.246752+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28986,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247082+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:35.247082+0000","last_peered":"2024-09-16T10:35:35.247082+0000","last_clean":"2024-09-16T10:35:35.247082+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:35.247082+0000","last_undegraded":"2024-09-16T10:35:35.247082+0000","last_fullsized":"2024-09-16T10:35:35.247082+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246109+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:35.246109+0000","last_peered":"2024-09-16T10:35:35.246109+0000","last_clean":"2024-09-16T10:35:35.246109+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:35.246109+0000","last_undegraded":"2024-09-16T10:35:35.246109+0000","last_fullsized":"2024-09-16T10:35:35.246109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30023,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.309942+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:36.309942+0000","last_peered":"2024-09-16T10:35:36.309942+0000","last_clean":"2024-09-16T10:35:36.309942+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:36.309942+0000","last_undegraded":"2024-09-16T10:35:36.309942+0000","last_fullsized":"2024-09-16T10:35:36.309942+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":110,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.254700+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:35.254700+0000","last_peered":"2024-09-16T10:35:35.254700+0000","last_clean":"2024-09-16T10:35:35.254700+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:35.254700+0000","last_undegraded":"2024-09-16T10:35:35.254700+0000","last_fullsized":"2024-09-16T10:35:35.254700+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29126,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:38.436222+0000","last_change":"2024-09-16T10:35:38.436222+0000","last_active":"2024-09-16T10:35:38.436222+0000","last_peered":"2024-09-16T10:35:38.436222+0000","last_clean":"2024-09-16T10:35:38.436222+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:38.436222+0000","last_undegraded":"2024-09-16T10:35:38.436222+0000","last_fullsized":"2024-09-16T10:35:38.436222+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":90,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247014+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:35.247014+0000","last_peered":"2024-09-16T10:35:35.247014+0000","last_clean":"2024-09-16T10:35:35.247014+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:35.247014+0000","last_undegraded":"2024-09-16T10:35:35.247014+0000","last_fullsized":"2024-09-16T10:35:35.247014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29091,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:36.250037+0000","last_change":"2024-09-16T10:35:39.462868+0000","last_active":"2024-09-16T10:35:36.250037+0000","last_peered":"2024-09-16T10:35:36.250037+0000","last_clean":"2024-09-16T10:35:36.250037+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:36.250037+0000","last_undegraded":"2024-09-16T10:35:36.250037+0000","last_fullsized":"2024-09-16T10:35:36.250037+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":24280,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:44.311782+0000","last_change":"2024-09-16T10:35:34.794858+0000","last_active":"2024-09-16T10:35:44.311782+0000","last_peered":"2024-09-16T10:35:44.311782+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:44.311782+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247245+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:35.247245+0000","last_peered":"2024-09-16T10:35:35.247245+0000","last_clean":"2024-09-16T10:35:35.247245+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:35.247245+0000","last_undegraded":"2024-09-16T10:35:35.247245+0000","last_fullsized":"2024-09-16T10:35:35.247245+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21210,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247395+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:35.247395+0000","last_peered":"2024-09-16T10:35:35.247395+0000","last_clean":"2024-09-16T10:35:35.247395+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:35.247395+0000","last_undegraded":"2024-09-16T10:35:35.247395+0000","last_fullsized":"2024-09-16T10:35:35.247395+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246102+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:35.246102+0000","last_peered":"2024-09-16T10:35:35.246102+0000","last_clean":"2024-09-16T10:35:35.246102+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:35.246102+0000","last_undegraded":"2024-09-16T10:35:35.246102+0000","last_fullsized":"2024-09-16T10:35:35.246102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24518,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.251503+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:36.251503+0000","last_peered":"2024-09-16T10:35:36.251503+0000","last_clean":"2024-09-16T10:35:36.251503+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:36.251503+0000","last_undegraded":"2024-09-16T10:35:36.251503+0000","last_fullsized":"2024-09-16T10:35:36.251503+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247326+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:35.247326+0000","last_peered":"2024-09-16T10:35:35.247326+0000","last_clean":"2024-09-16T10:35:35.247326+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:35.247326+0000","last_undegraded":"2024-09-16T10:35:35.247326+0000","last_fullsized":"2024-09-16T10:35:35.247326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247481+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:35.247481+0000","last_peered":"2024-09-16T10:35:35.247481+0000","last_clean":"2024-09-16T10:35:35.247481+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:35.247481+0000","last_undegraded":"2024-09-16T10:35:35.247481+0000","last_fullsized":"2024-09-16T10:35:35.247481+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21155,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247180+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:35.247180+0000","last_peered":"2024-09-16T10:35:35.247180+0000","last_clean":"2024-09-16T10:35:35.247180+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:35.247180+0000","last_undegraded":"2024-09-16T10:35:35.247180+0000","last_fullsized":"2024-09-16T10:35:35.247180+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44969,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104722432,"data_stored":104707980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019039,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381384,"kb_used_data":34752,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990456,"statfs":{"total":96636764160,"available":96246226944,"internally_reserved":0,"allocated":35586048,"data_stored":35244299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.26200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.186}]}]},{"osd":1,"up_from":40,"seq":171798691983,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":483344,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93888496,"statfs":{"total":96636764160,"available":96141819904,"internally_reserved":0,"allocated":532480,"data_stored":175815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":494399146},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16800000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.20399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.25},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.26400000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523376,"kb_used_data":68512,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848464,"statfs":{"total":96636764160,"available":96100827136,"internally_reserved":0,"allocated":70156288,"data_stored":69810463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.187},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.215}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.247},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.25700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:49.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:50.302 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:50.303 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:50.316 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-16T10:35:48.784835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44970,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1414344,"kb_used_data":103784,"kb_used_omap":12,"kb_used_meta":1310515,"kb_avail":281701176,"statfs":{"total":289910292480,"available":288462004224,"internally_reserved":0,"allocated":106274816,"data_stored":105230577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1341967362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002144"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28951,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.252058+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:35.252058+0000","last_peered":"2024-09-16T10:35:35.252058+0000","last_clean":"2024-09-16T10:35:35.252058+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:35.252058+0000","last_undegraded":"2024-09-16T10:35:35.252058+0000","last_fullsized":"2024-09-16T10:35:35.252058+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20641,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246198+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:35.246198+0000","last_peered":"2024-09-16T10:35:35.246198+0000","last_clean":"2024-09-16T10:35:35.246198+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:35.246198+0000","last_undegraded":"2024-09-16T10:35:35.246198+0000","last_fullsized":"2024-09-16T10:35:35.246198+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246728+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:35.246728+0000","last_peered":"2024-09-16T10:35:35.246728+0000","last_clean":"2024-09-16T10:35:35.246728+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:35.246728+0000","last_undegraded":"2024-09-16T10:35:35.246728+0000","last_fullsized":"2024-09-16T10:35:35.246728+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20540,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246021+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:35.246021+0000","last_peered":"2024-09-16T10:35:35.246021+0000","last_clean":"2024-09-16T10:35:35.246021+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:35.246021+0000","last_undegraded":"2024-09-16T10:35:35.246021+0000","last_fullsized":"2024-09-16T10:35:35.246021+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20683,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248479+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:35.248479+0000","last_peered":"2024-09-16T10:35:35.248479+0000","last_clean":"2024-09-16T10:35:35.248479+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:35.248479+0000","last_undegraded":"2024-09-16T10:35:35.248479+0000","last_fullsized":"2024-09-16T10:35:35.248479+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20722,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246758+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:35.246758+0000","last_peered":"2024-09-16T10:35:35.246758+0000","last_clean":"2024-09-16T10:35:35.246758+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:35.246758+0000","last_undegraded":"2024-09-16T10:35:35.246758+0000","last_fullsized":"2024-09-16T10:35:35.246758+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21347,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:35.248771+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:35.248771+0000","last_peered":"2024-09-16T10:35:35.248771+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:35.248771+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20587,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248438+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:35.248438+0000","last_peered":"2024-09-16T10:35:35.248438+0000","last_clean":"2024-09-16T10:35:35.248438+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:35.248438+0000","last_undegraded":"2024-09-16T10:35:35.248438+0000","last_fullsized":"2024-09-16T10:35:35.248438+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246752+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:35.246752+0000","last_peered":"2024-09-16T10:35:35.246752+0000","last_clean":"2024-09-16T10:35:35.246752+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:35.246752+0000","last_undegraded":"2024-09-16T10:35:35.246752+0000","last_fullsized":"2024-09-16T10:35:35.246752+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28986,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247082+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:35.247082+0000","last_peered":"2024-09-16T10:35:35.247082+0000","last_clean":"2024-09-16T10:35:35.247082+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:35.247082+0000","last_undegraded":"2024-09-16T10:35:35.247082+0000","last_fullsized":"2024-09-16T10:35:35.247082+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246109+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:35.246109+0000","last_peered":"2024-09-16T10:35:35.246109+0000","last_clean":"2024-09-16T10:35:35.246109+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:35.246109+0000","last_undegraded":"2024-09-16T10:35:35.246109+0000","last_fullsized":"2024-09-16T10:35:35.246109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30023,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.309942+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:36.309942+0000","last_peered":"2024-09-16T10:35:36.309942+0000","last_clean":"2024-09-16T10:35:36.309942+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:36.309942+0000","last_undegraded":"2024-09-16T10:35:36.309942+0000","last_fullsized":"2024-09-16T10:35:36.309942+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":110,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.254700+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:35.254700+0000","last_peered":"2024-09-16T10:35:35.254700+0000","last_clean":"2024-09-16T10:35:35.254700+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:35.254700+0000","last_undegraded":"2024-09-16T10:35:35.254700+0000","last_fullsized":"2024-09-16T10:35:35.254700+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29127,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:40.434008+0000","last_change":"2024-09-16T10:35:38.436222+0000","last_active":"2024-09-16T10:35:40.434008+0000","last_peered":"2024-09-16T10:35:40.434008+0000","last_clean":"2024-09-16T10:35:40.434008+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:40.434008+0000","last_undegraded":"2024-09-16T10:35:40.434008+0000","last_fullsized":"2024-09-16T10:35:40.434008+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":90,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247014+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:35.247014+0000","last_peered":"2024-09-16T10:35:35.247014+0000","last_clean":"2024-09-16T10:35:35.247014+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:35.247014+0000","last_undegraded":"2024-09-16T10:35:35.247014+0000","last_fullsized":"2024-09-16T10:35:35.247014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29092,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:39.462868+0000","last_change":"2024-09-16T10:35:39.462868+0000","last_active":"2024-09-16T10:35:39.462868+0000","last_peered":"2024-09-16T10:35:39.462868+0000","last_clean":"2024-09-16T10:35:39.462868+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:39.462868+0000","last_undegraded":"2024-09-16T10:35:39.462868+0000","last_fullsized":"2024-09-16T10:35:39.462868+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":24280,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:44.311782+0000","last_change":"2024-09-16T10:35:34.794858+0000","last_active":"2024-09-16T10:35:44.311782+0000","last_peered":"2024-09-16T10:35:44.311782+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:44.311782+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247245+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:35.247245+0000","last_peered":"2024-09-16T10:35:35.247245+0000","last_clean":"2024-09-16T10:35:35.247245+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:35.247245+0000","last_undegraded":"2024-09-16T10:35:35.247245+0000","last_fullsized":"2024-09-16T10:35:35.247245+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21210,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247395+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:35.247395+0000","last_peered":"2024-09-16T10:35:35.247395+0000","last_clean":"2024-09-16T10:35:35.247395+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:35.247395+0000","last_undegraded":"2024-09-16T10:35:35.247395+0000","last_fullsized":"2024-09-16T10:35:35.247395+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246102+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:35.246102+0000","last_peered":"2024-09-16T10:35:35.246102+0000","last_clean":"2024-09-16T10:35:35.246102+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:35.246102+0000","last_undegraded":"2024-09-16T10:35:35.246102+0000","last_fullsized":"2024-09-16T10:35:35.246102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24518,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.251503+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:36.251503+0000","last_peered":"2024-09-16T10:35:36.251503+0000","last_clean":"2024-09-16T10:35:36.251503+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:36.251503+0000","last_undegraded":"2024-09-16T10:35:36.251503+0000","last_fullsized":"2024-09-16T10:35:36.251503+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247326+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:35.247326+0000","last_peered":"2024-09-16T10:35:35.247326+0000","last_clean":"2024-09-16T10:35:35.247326+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:35.247326+0000","last_undegraded":"2024-09-16T10:35:35.247326+0000","last_fullsized":"2024-09-16T10:35:35.247326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247481+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:35.247481+0000","last_peered":"2024-09-16T10:35:35.247481+0000","last_clean":"2024-09-16T10:35:35.247481+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:35.247481+0000","last_undegraded":"2024-09-16T10:35:35.247481+0000","last_fullsized":"2024-09-16T10:35:35.247481+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21155,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247180+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:35.247180+0000","last_peered":"2024-09-16T10:35:35.247180+0000","last_clean":"2024-09-16T10:35:35.247180+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:35.247180+0000","last_undegraded":"2024-09-16T10:35:35.247180+0000","last_fullsized":"2024-09-16T10:35:35.247180+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44969,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104722432,"data_stored":104707980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019040,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381384,"kb_used_data":34752,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990456,"statfs":{"total":96636764160,"available":96246226944,"internally_reserved":0,"allocated":35586048,"data_stored":35244299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.25}]}]},{"osd":1,"up_from":40,"seq":171798691984,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509584,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862256,"statfs":{"total":96636764160,"available":96114950144,"internally_reserved":0,"allocated":532480,"data_stored":175815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":521268906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.217}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.251}]}]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523376,"kb_used_data":68512,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848464,"statfs":{"total":96636764160,"available":96100827136,"internally_reserved":0,"allocated":70156288,"data_stored":69810463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.187},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.215}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.247},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.25700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:50.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:50.621 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:50.621 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:50.634 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-16T10:35:48.784835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44970,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1414344,"kb_used_data":103784,"kb_used_omap":12,"kb_used_meta":1310515,"kb_avail":281701176,"statfs":{"total":289910292480,"available":288462004224,"internally_reserved":0,"allocated":106274816,"data_stored":105230577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1341967362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2155,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002144"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28951,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.252058+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:35.252058+0000","last_peered":"2024-09-16T10:35:35.252058+0000","last_clean":"2024-09-16T10:35:35.252058+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:35.252058+0000","last_undegraded":"2024-09-16T10:35:35.252058+0000","last_fullsized":"2024-09-16T10:35:35.252058+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20641,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246198+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:35.246198+0000","last_peered":"2024-09-16T10:35:35.246198+0000","last_clean":"2024-09-16T10:35:35.246198+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:35.246198+0000","last_undegraded":"2024-09-16T10:35:35.246198+0000","last_fullsized":"2024-09-16T10:35:35.246198+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246728+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:35.246728+0000","last_peered":"2024-09-16T10:35:35.246728+0000","last_clean":"2024-09-16T10:35:35.246728+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:35.246728+0000","last_undegraded":"2024-09-16T10:35:35.246728+0000","last_fullsized":"2024-09-16T10:35:35.246728+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20540,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246021+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:35.246021+0000","last_peered":"2024-09-16T10:35:35.246021+0000","last_clean":"2024-09-16T10:35:35.246021+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:35.246021+0000","last_undegraded":"2024-09-16T10:35:35.246021+0000","last_fullsized":"2024-09-16T10:35:35.246021+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20683,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248479+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:35.248479+0000","last_peered":"2024-09-16T10:35:35.248479+0000","last_clean":"2024-09-16T10:35:35.248479+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:35.248479+0000","last_undegraded":"2024-09-16T10:35:35.248479+0000","last_fullsized":"2024-09-16T10:35:35.248479+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20722,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246758+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:35.246758+0000","last_peered":"2024-09-16T10:35:35.246758+0000","last_clean":"2024-09-16T10:35:35.246758+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:35.246758+0000","last_undegraded":"2024-09-16T10:35:35.246758+0000","last_fullsized":"2024-09-16T10:35:35.246758+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21347,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:35.248771+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:35.248771+0000","last_peered":"2024-09-16T10:35:35.248771+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:35.248771+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20587,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248438+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:35.248438+0000","last_peered":"2024-09-16T10:35:35.248438+0000","last_clean":"2024-09-16T10:35:35.248438+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:35.248438+0000","last_undegraded":"2024-09-16T10:35:35.248438+0000","last_fullsized":"2024-09-16T10:35:35.248438+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246752+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:35.246752+0000","last_peered":"2024-09-16T10:35:35.246752+0000","last_clean":"2024-09-16T10:35:35.246752+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:35.246752+0000","last_undegraded":"2024-09-16T10:35:35.246752+0000","last_fullsized":"2024-09-16T10:35:35.246752+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28986,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247082+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:35.247082+0000","last_peered":"2024-09-16T10:35:35.247082+0000","last_clean":"2024-09-16T10:35:35.247082+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:35.247082+0000","last_undegraded":"2024-09-16T10:35:35.247082+0000","last_fullsized":"2024-09-16T10:35:35.247082+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246109+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:35.246109+0000","last_peered":"2024-09-16T10:35:35.246109+0000","last_clean":"2024-09-16T10:35:35.246109+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:35.246109+0000","last_undegraded":"2024-09-16T10:35:35.246109+0000","last_fullsized":"2024-09-16T10:35:35.246109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30023,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.309942+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:36.309942+0000","last_peered":"2024-09-16T10:35:36.309942+0000","last_clean":"2024-09-16T10:35:36.309942+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:36.309942+0000","last_undegraded":"2024-09-16T10:35:36.309942+0000","last_fullsized":"2024-09-16T10:35:36.309942+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":110,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.254700+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:35.254700+0000","last_peered":"2024-09-16T10:35:35.254700+0000","last_clean":"2024-09-16T10:35:35.254700+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:35.254700+0000","last_undegraded":"2024-09-16T10:35:35.254700+0000","last_fullsized":"2024-09-16T10:35:35.254700+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29127,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:40.434008+0000","last_change":"2024-09-16T10:35:38.436222+0000","last_active":"2024-09-16T10:35:40.434008+0000","last_peered":"2024-09-16T10:35:40.434008+0000","last_clean":"2024-09-16T10:35:40.434008+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:40.434008+0000","last_undegraded":"2024-09-16T10:35:40.434008+0000","last_fullsized":"2024-09-16T10:35:40.434008+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":90,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247014+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:35.247014+0000","last_peered":"2024-09-16T10:35:35.247014+0000","last_clean":"2024-09-16T10:35:35.247014+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:35.247014+0000","last_undegraded":"2024-09-16T10:35:35.247014+0000","last_fullsized":"2024-09-16T10:35:35.247014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29092,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:39.462868+0000","last_change":"2024-09-16T10:35:39.462868+0000","last_active":"2024-09-16T10:35:39.462868+0000","last_peered":"2024-09-16T10:35:39.462868+0000","last_clean":"2024-09-16T10:35:39.462868+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:39.462868+0000","last_undegraded":"2024-09-16T10:35:39.462868+0000","last_fullsized":"2024-09-16T10:35:39.462868+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":24280,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:44.311782+0000","last_change":"2024-09-16T10:35:34.794858+0000","last_active":"2024-09-16T10:35:44.311782+0000","last_peered":"2024-09-16T10:35:44.311782+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:44.311782+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2087}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247245+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:35.247245+0000","last_peered":"2024-09-16T10:35:35.247245+0000","last_clean":"2024-09-16T10:35:35.247245+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:35.247245+0000","last_undegraded":"2024-09-16T10:35:35.247245+0000","last_fullsized":"2024-09-16T10:35:35.247245+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21210,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247395+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:35.247395+0000","last_peered":"2024-09-16T10:35:35.247395+0000","last_clean":"2024-09-16T10:35:35.247395+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:35.247395+0000","last_undegraded":"2024-09-16T10:35:35.247395+0000","last_fullsized":"2024-09-16T10:35:35.247395+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246102+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:35.246102+0000","last_peered":"2024-09-16T10:35:35.246102+0000","last_clean":"2024-09-16T10:35:35.246102+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:35.246102+0000","last_undegraded":"2024-09-16T10:35:35.246102+0000","last_fullsized":"2024-09-16T10:35:35.246102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24518,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.251503+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:36.251503+0000","last_peered":"2024-09-16T10:35:36.251503+0000","last_clean":"2024-09-16T10:35:36.251503+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:36.251503+0000","last_undegraded":"2024-09-16T10:35:36.251503+0000","last_fullsized":"2024-09-16T10:35:36.251503+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247326+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:35.247326+0000","last_peered":"2024-09-16T10:35:35.247326+0000","last_clean":"2024-09-16T10:35:35.247326+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:35.247326+0000","last_undegraded":"2024-09-16T10:35:35.247326+0000","last_fullsized":"2024-09-16T10:35:35.247326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247481+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:35.247481+0000","last_peered":"2024-09-16T10:35:35.247481+0000","last_clean":"2024-09-16T10:35:35.247481+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:35.247481+0000","last_undegraded":"2024-09-16T10:35:35.247481+0000","last_fullsized":"2024-09-16T10:35:35.247481+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21155,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247180+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:35.247180+0000","last_peered":"2024-09-16T10:35:35.247180+0000","last_clean":"2024-09-16T10:35:35.247180+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:35.247180+0000","last_undegraded":"2024-09-16T10:35:35.247180+0000","last_fullsized":"2024-09-16T10:35:35.247180+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6377,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44969,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104722432,"data_stored":104707980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019040,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381384,"kb_used_data":34752,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990456,"statfs":{"total":96636764160,"available":96246226944,"internally_reserved":0,"allocated":35586048,"data_stored":35244299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.25}]}]},{"osd":1,"up_from":40,"seq":171798691984,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509584,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862256,"statfs":{"total":96636764160,"available":96114950144,"internally_reserved":0,"allocated":532480,"data_stored":175815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":521268906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.217}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.251}]}]},{"osd":0,"up_from":24,"seq":103079215266,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523376,"kb_used_data":68512,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848464,"statfs":{"total":96636764160,"available":96100827136,"internally_reserved":0,"allocated":70156288,"data_stored":69810463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.187},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.215}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.247},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.25700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:50.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:50.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 21347, 'reported_epoch': 61, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:35.248771+0000', 'last_change': '2024-09-16T10:23:56.949164+0000', 'last_active': '2024-09-16T10:35:35.248771+0000', 'last_peered': '2024-09-16T10:35:35.248771+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:35.248771+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4289, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 116, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -4287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:50.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:35:50.635 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29127, 'reported_epoch': 62, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:40.434008+0000', 'last_change': '2024-09-16T10:35:38.436222+0000', 'last_active': '2024-09-16T10:35:40.434008+0000', 'last_peered': '2024-09-16T10:35:40.434008+0000', 'last_clean': '2024-09-16T10:35:40.434008+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:35:40.434008+0000', 'last_undegraded': '2024-09-16T10:35:40.434008+0000', 'last_fullsized': '2024-09-16T10:35:40.434008+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29092, 'reported_epoch': 62, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:39.462868+0000', 'last_change': '2024-09-16T10:35:39.462868+0000', 'last_active': '2024-09-16T10:35:39.462868+0000', 'last_peered': '2024-09-16T10:35:39.462868+0000', 'last_clean': '2024-09-16T10:35:39.462868+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:39.462868+0000', 'last_undegraded': '2024-09-16T10:35:39.462868+0000', 'last_fullsized': '2024-09-16T10:35:39.462868+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.7 is not active+clean 2024-09-16T10:35:50.636 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.7', 'version': "48'20034", 'reported_seq': 24280, 'reported_epoch': 62, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:44.311782+0000', 'last_change': '2024-09-16T10:35:34.794858+0000', 'last_active': '2024-09-16T10:35:44.311782+0000', 'last_peered': '2024-09-16T10:35:44.311782+0000', 'last_clean': '2024-09-16T10:23:54.364704+0000', 'last_became_active': '2024-09-16T10:23:56.929329+0000', 'last_became_peered': '2024-09-16T10:23:56.929329+0000', 'last_unstale': '2024-09-16T10:35:44.311782+0000', 'last_undegraded': '2024-09-16T10:23:56.479925+0000', 'last_fullsized': '2024-09-16T10:23:56.368280+0000', 'mapping_epoch': 41, 'log_start': "44'10000", 'ondisk_log_start': "44'10000", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 483, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2088, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20033, 'num_write_kb': 40065, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2073, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2087}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:50.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:51.046 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:51.046 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6377/40 objects degraded (15942.500%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":839,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":62,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":20},{"state_name":"active+clean+scrubbing","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1448288256,"bytes_avail":288462004224,"bytes_total":289910292480,"degraded_objects":6377,"degraded_total":40,"degraded_ratio":159.42500000000001,"recovering_objects_per_sec":179,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2155,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:51.059 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:51.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:51.373 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:51.373 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:51.385 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-16T10:35:50.785206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46428,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1414344,"kb_used_data":103784,"kb_used_omap":12,"kb_used_meta":1310515,"kb_avail":281701176,"statfs":{"total":289910292480,"available":288462004224,"internally_reserved":0,"allocated":106274816,"data_stored":105230577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1341967362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3613,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002115"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28951,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.252058+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:35.252058+0000","last_peered":"2024-09-16T10:35:35.252058+0000","last_clean":"2024-09-16T10:35:35.252058+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:35.252058+0000","last_undegraded":"2024-09-16T10:35:35.252058+0000","last_fullsized":"2024-09-16T10:35:35.252058+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20641,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246198+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:35.246198+0000","last_peered":"2024-09-16T10:35:35.246198+0000","last_clean":"2024-09-16T10:35:35.246198+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:35.246198+0000","last_undegraded":"2024-09-16T10:35:35.246198+0000","last_fullsized":"2024-09-16T10:35:35.246198+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246728+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:35.246728+0000","last_peered":"2024-09-16T10:35:35.246728+0000","last_clean":"2024-09-16T10:35:35.246728+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:35.246728+0000","last_undegraded":"2024-09-16T10:35:35.246728+0000","last_fullsized":"2024-09-16T10:35:35.246728+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20540,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246021+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:35.246021+0000","last_peered":"2024-09-16T10:35:35.246021+0000","last_clean":"2024-09-16T10:35:35.246021+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:35.246021+0000","last_undegraded":"2024-09-16T10:35:35.246021+0000","last_fullsized":"2024-09-16T10:35:35.246021+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20684,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.252603+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:36.252603+0000","last_peered":"2024-09-16T10:35:36.252603+0000","last_clean":"2024-09-16T10:35:36.252603+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:36.252603+0000","last_undegraded":"2024-09-16T10:35:36.252603+0000","last_fullsized":"2024-09-16T10:35:36.252603+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20722,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246758+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:35.246758+0000","last_peered":"2024-09-16T10:35:35.246758+0000","last_clean":"2024-09-16T10:35:35.246758+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:35.246758+0000","last_undegraded":"2024-09-16T10:35:35.246758+0000","last_fullsized":"2024-09-16T10:35:35.246758+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21347,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:35.248771+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:35.248771+0000","last_peered":"2024-09-16T10:35:35.248771+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:35.248771+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20587,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248438+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:35.248438+0000","last_peered":"2024-09-16T10:35:35.248438+0000","last_clean":"2024-09-16T10:35:35.248438+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:35.248438+0000","last_undegraded":"2024-09-16T10:35:35.248438+0000","last_fullsized":"2024-09-16T10:35:35.248438+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246752+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:35.246752+0000","last_peered":"2024-09-16T10:35:35.246752+0000","last_clean":"2024-09-16T10:35:35.246752+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:35.246752+0000","last_undegraded":"2024-09-16T10:35:35.246752+0000","last_fullsized":"2024-09-16T10:35:35.246752+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28986,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247082+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:35.247082+0000","last_peered":"2024-09-16T10:35:35.247082+0000","last_clean":"2024-09-16T10:35:35.247082+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:35.247082+0000","last_undegraded":"2024-09-16T10:35:35.247082+0000","last_fullsized":"2024-09-16T10:35:35.247082+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246109+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:35.246109+0000","last_peered":"2024-09-16T10:35:35.246109+0000","last_clean":"2024-09-16T10:35:35.246109+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:35.246109+0000","last_undegraded":"2024-09-16T10:35:35.246109+0000","last_fullsized":"2024-09-16T10:35:35.246109+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30023,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.309942+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:36.309942+0000","last_peered":"2024-09-16T10:35:36.309942+0000","last_clean":"2024-09-16T10:35:36.309942+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:36.309942+0000","last_undegraded":"2024-09-16T10:35:36.309942+0000","last_fullsized":"2024-09-16T10:35:36.309942+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":110,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.254700+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:35.254700+0000","last_peered":"2024-09-16T10:35:35.254700+0000","last_clean":"2024-09-16T10:35:35.254700+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:35.254700+0000","last_undegraded":"2024-09-16T10:35:35.254700+0000","last_fullsized":"2024-09-16T10:35:35.254700+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29127,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:40.434008+0000","last_change":"2024-09-16T10:35:38.436222+0000","last_active":"2024-09-16T10:35:40.434008+0000","last_peered":"2024-09-16T10:35:40.434008+0000","last_clean":"2024-09-16T10:35:40.434008+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:40.434008+0000","last_undegraded":"2024-09-16T10:35:40.434008+0000","last_fullsized":"2024-09-16T10:35:40.434008+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":90,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247014+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:35.247014+0000","last_peered":"2024-09-16T10:35:35.247014+0000","last_clean":"2024-09-16T10:35:35.247014+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:35.247014+0000","last_undegraded":"2024-09-16T10:35:35.247014+0000","last_fullsized":"2024-09-16T10:35:35.247014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29092,"reported_epoch":62,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:39.462868+0000","last_change":"2024-09-16T10:35:39.462868+0000","last_active":"2024-09-16T10:35:39.462868+0000","last_peered":"2024-09-16T10:35:39.462868+0000","last_clean":"2024-09-16T10:35:39.462868+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:39.462868+0000","last_undegraded":"2024-09-16T10:35:39.462868+0000","last_fullsized":"2024-09-16T10:35:39.462868+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":27196,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:49.312994+0000","last_change":"2024-09-16T10:35:34.794858+0000","last_active":"2024-09-16T10:35:49.312994+0000","last_peered":"2024-09-16T10:35:49.312994+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:49.312994+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-629}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247245+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:35.247245+0000","last_peered":"2024-09-16T10:35:35.247245+0000","last_clean":"2024-09-16T10:35:35.247245+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:35.247245+0000","last_undegraded":"2024-09-16T10:35:35.247245+0000","last_fullsized":"2024-09-16T10:35:35.247245+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21210,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247395+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:35.247395+0000","last_peered":"2024-09-16T10:35:35.247395+0000","last_clean":"2024-09-16T10:35:35.247395+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:35.247395+0000","last_undegraded":"2024-09-16T10:35:35.247395+0000","last_fullsized":"2024-09-16T10:35:35.247395+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":79,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246102+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:35.246102+0000","last_peered":"2024-09-16T10:35:35.246102+0000","last_clean":"2024-09-16T10:35:35.246102+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:35.246102+0000","last_undegraded":"2024-09-16T10:35:35.246102+0000","last_fullsized":"2024-09-16T10:35:35.246102+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24518,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.251503+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:36.251503+0000","last_peered":"2024-09-16T10:35:36.251503+0000","last_clean":"2024-09-16T10:35:36.251503+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:36.251503+0000","last_undegraded":"2024-09-16T10:35:36.251503+0000","last_fullsized":"2024-09-16T10:35:36.251503+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247326+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:35.247326+0000","last_peered":"2024-09-16T10:35:35.247326+0000","last_clean":"2024-09-16T10:35:35.247326+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:35.247326+0000","last_undegraded":"2024-09-16T10:35:35.247326+0000","last_fullsized":"2024-09-16T10:35:35.247326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247481+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:35.247481+0000","last_peered":"2024-09-16T10:35:35.247481+0000","last_clean":"2024-09-16T10:35:35.247481+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:35.247481+0000","last_undegraded":"2024-09-16T10:35:35.247481+0000","last_fullsized":"2024-09-16T10:35:35.247481+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21155,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247180+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:35.247180+0000","last_peered":"2024-09-16T10:35:35.247180+0000","last_clean":"2024-09-16T10:35:35.247180+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:35.247180+0000","last_undegraded":"2024-09-16T10:35:35.247180+0000","last_fullsized":"2024-09-16T10:35:35.247180+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46427,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104722432,"data_stored":104707980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019040,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381384,"kb_used_data":34752,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990456,"statfs":{"total":96636764160,"available":96246226944,"internally_reserved":0,"allocated":35586048,"data_stored":35244299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23200000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.25}]}]},{"osd":1,"up_from":40,"seq":171798691984,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509584,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862256,"statfs":{"total":96636764160,"available":96114950144,"internally_reserved":0,"allocated":532480,"data_stored":175815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":521268906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.217}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.251}]}]},{"osd":0,"up_from":24,"seq":103079215267,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523376,"kb_used_data":68512,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848464,"statfs":{"total":96636764160,"available":96100827136,"internally_reserved":0,"allocated":70156288,"data_stored":69810463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.22500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:54.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:54.730 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:54.730 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:54.744 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-16T10:35:53.289548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46428,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":46,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1414356,"kb_used_data":103792,"kb_used_omap":12,"kb_used_meta":1310515,"kb_avail":281701164,"statfs":{"total":289910292480,"available":288461991936,"internally_reserved":0,"allocated":106283008,"data_stored":105233332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1341967362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1458,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.505327"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28951,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.252058+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:35.252058+0000","last_peered":"2024-09-16T10:35:35.252058+0000","last_clean":"2024-09-16T10:35:35.252058+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:35.252058+0000","last_undegraded":"2024-09-16T10:35:35.252058+0000","last_fullsized":"2024-09-16T10:35:35.252058+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20642,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250477+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:36.250477+0000","last_peered":"2024-09-16T10:35:36.250477+0000","last_clean":"2024-09-16T10:35:36.250477+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:36.250477+0000","last_undegraded":"2024-09-16T10:35:36.250477+0000","last_fullsized":"2024-09-16T10:35:36.250477+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20943,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246728+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:35.246728+0000","last_peered":"2024-09-16T10:35:35.246728+0000","last_clean":"2024-09-16T10:35:35.246728+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:35.246728+0000","last_undegraded":"2024-09-16T10:35:35.246728+0000","last_fullsized":"2024-09-16T10:35:35.246728+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20541,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250309+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:36.250309+0000","last_peered":"2024-09-16T10:35:36.250309+0000","last_clean":"2024-09-16T10:35:36.250309+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:36.250309+0000","last_undegraded":"2024-09-16T10:35:36.250309+0000","last_fullsized":"2024-09-16T10:35:36.250309+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20684,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.252603+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:36.252603+0000","last_peered":"2024-09-16T10:35:36.252603+0000","last_clean":"2024-09-16T10:35:36.252603+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:36.252603+0000","last_undegraded":"2024-09-16T10:35:36.252603+0000","last_fullsized":"2024-09-16T10:35:36.252603+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20722,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246758+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:35.246758+0000","last_peered":"2024-09-16T10:35:35.246758+0000","last_clean":"2024-09-16T10:35:35.246758+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:35.246758+0000","last_undegraded":"2024-09-16T10:35:35.246758+0000","last_fullsized":"2024-09-16T10:35:35.246758+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21347,"reported_epoch":61,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:35.248771+0000","last_change":"2024-09-16T10:23:56.949164+0000","last_active":"2024-09-16T10:35:35.248771+0000","last_peered":"2024-09-16T10:35:35.248771+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:35.248771+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4289,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-4287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20587,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.248438+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:35.248438+0000","last_peered":"2024-09-16T10:35:35.248438+0000","last_clean":"2024-09-16T10:35:35.248438+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:35.248438+0000","last_undegraded":"2024-09-16T10:35:35.248438+0000","last_fullsized":"2024-09-16T10:35:35.248438+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.246752+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:35.246752+0000","last_peered":"2024-09-16T10:35:35.246752+0000","last_clean":"2024-09-16T10:35:35.246752+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:35.246752+0000","last_undegraded":"2024-09-16T10:35:35.246752+0000","last_fullsized":"2024-09-16T10:35:35.246752+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28986,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247082+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:35.247082+0000","last_peered":"2024-09-16T10:35:35.247082+0000","last_clean":"2024-09-16T10:35:35.247082+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:35.247082+0000","last_undegraded":"2024-09-16T10:35:35.247082+0000","last_fullsized":"2024-09-16T10:35:35.247082+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":80,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250185+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:36.250185+0000","last_peered":"2024-09-16T10:35:36.250185+0000","last_clean":"2024-09-16T10:35:36.250185+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:36.250185+0000","last_undegraded":"2024-09-16T10:35:36.250185+0000","last_fullsized":"2024-09-16T10:35:36.250185+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30023,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.309942+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:36.309942+0000","last_peered":"2024-09-16T10:35:36.309942+0000","last_clean":"2024-09-16T10:35:36.309942+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:36.309942+0000","last_undegraded":"2024-09-16T10:35:36.309942+0000","last_fullsized":"2024-09-16T10:35:36.309942+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":110,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.254700+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:35.254700+0000","last_peered":"2024-09-16T10:35:35.254700+0000","last_clean":"2024-09-16T10:35:35.254700+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:35.254700+0000","last_undegraded":"2024-09-16T10:35:35.254700+0000","last_fullsized":"2024-09-16T10:35:35.254700+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29129,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:50.376681+0000","last_change":"2024-09-16T10:35:52.289743+0000","last_active":"2024-09-16T10:35:50.376681+0000","last_peered":"2024-09-16T10:35:50.376681+0000","last_clean":"2024-09-16T10:35:50.376681+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:50.376681+0000","last_undegraded":"2024-09-16T10:35:50.376681+0000","last_fullsized":"2024-09-16T10:35:50.376681+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":90,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247014+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:35.247014+0000","last_peered":"2024-09-16T10:35:35.247014+0000","last_clean":"2024-09-16T10:35:35.247014+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:35.247014+0000","last_undegraded":"2024-09-16T10:35:35.247014+0000","last_fullsized":"2024-09-16T10:35:35.247014+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29094,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:48.367025+0000","last_change":"2024-09-16T10:35:52.289501+0000","last_active":"2024-09-16T10:35:48.367025+0000","last_peered":"2024-09-16T10:35:48.367025+0000","last_clean":"2024-09-16T10:35:48.367025+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:48.367025+0000","last_undegraded":"2024-09-16T10:35:48.367025+0000","last_fullsized":"2024-09-16T10:35:48.367025+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":27196,"reported_epoch":62,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:49.312994+0000","last_change":"2024-09-16T10:35:34.794858+0000","last_active":"2024-09-16T10:35:49.312994+0000","last_peered":"2024-09-16T10:35:49.312994+0000","last_clean":"2024-09-16T10:23:54.364704+0000","last_became_active":"2024-09-16T10:23:56.929329+0000","last_became_peered":"2024-09-16T10:23:56.929329+0000","last_unstale":"2024-09-16T10:35:49.312994+0000","last_undegraded":"2024-09-16T10:23:56.479925+0000","last_fullsized":"2024-09-16T10:23:56.368280+0000","mapping_epoch":41,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":630,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3531,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-629}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247245+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:35.247245+0000","last_peered":"2024-09-16T10:35:35.247245+0000","last_clean":"2024-09-16T10:35:35.247245+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:35.247245+0000","last_undegraded":"2024-09-16T10:35:35.247245+0000","last_fullsized":"2024-09-16T10:35:35.247245+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21210,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247395+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:35.247395+0000","last_peered":"2024-09-16T10:35:35.247395+0000","last_clean":"2024-09-16T10:35:35.247395+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:35.247395+0000","last_undegraded":"2024-09-16T10:35:35.247395+0000","last_fullsized":"2024-09-16T10:35:35.247395+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":80,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250627+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:36.250627+0000","last_peered":"2024-09-16T10:35:36.250627+0000","last_clean":"2024-09-16T10:35:36.250627+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:36.250627+0000","last_undegraded":"2024-09-16T10:35:36.250627+0000","last_fullsized":"2024-09-16T10:35:36.250627+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24518,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.251503+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:36.251503+0000","last_peered":"2024-09-16T10:35:36.251503+0000","last_clean":"2024-09-16T10:35:36.251503+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:36.251503+0000","last_undegraded":"2024-09-16T10:35:36.251503+0000","last_fullsized":"2024-09-16T10:35:36.251503+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":76,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247326+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:35.247326+0000","last_peered":"2024-09-16T10:35:35.247326+0000","last_clean":"2024-09-16T10:35:35.247326+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:35.247326+0000","last_undegraded":"2024-09-16T10:35:35.247326+0000","last_fullsized":"2024-09-16T10:35:35.247326+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":85,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247481+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:35.247481+0000","last_peered":"2024-09-16T10:35:35.247481+0000","last_clean":"2024-09-16T10:35:35.247481+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:35.247481+0000","last_undegraded":"2024-09-16T10:35:35.247481+0000","last_fullsized":"2024-09-16T10:35:35.247481+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21155,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-16T10:35:35.247180+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:35.247180+0000","last_peered":"2024-09-16T10:35:35.247180+0000","last_clean":"2024-09-16T10:35:35.247180+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:35.247180+0000","last_undegraded":"2024-09-16T10:35:35.247180+0000","last_fullsized":"2024-09-16T10:35:35.247180+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46427,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104722432,"data_stored":104707980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019041,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381396,"kb_used_data":34760,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990444,"statfs":{"total":96636764160,"available":96246214656,"internally_reserved":0,"allocated":35594240,"data_stored":35247054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.22800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.25700000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691984,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":509584,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":509051,"kb_avail":93862256,"statfs":{"total":96636764160,"available":96114950144,"internally_reserved":0,"allocated":532480,"data_stored":175815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":521268906},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.17299999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.217}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20300000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.251}]}]},{"osd":0,"up_from":24,"seq":103079215267,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523376,"kb_used_data":68512,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848464,"statfs":{"total":96636764160,"available":96100827136,"internally_reserved":0,"allocated":70156288,"data_stored":69810463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.22500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:54.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:55.074 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:55.075 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:55.088 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-16T10:35:54.785957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47513,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1410280,"kb_used_data":103808,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281705240,"statfs":{"total":289910292480,"available":288466165760,"internally_reserved":0,"allocated":106299392,"data_stored":105240610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2543,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2543,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001408"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20642,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250477+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:36.250477+0000","last_peered":"2024-09-16T10:35:36.250477+0000","last_clean":"2024-09-16T10:35:36.250477+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:36.250477+0000","last_undegraded":"2024-09-16T10:35:36.250477+0000","last_fullsized":"2024-09-16T10:35:36.250477+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20541,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250309+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:36.250309+0000","last_peered":"2024-09-16T10:35:36.250309+0000","last_clean":"2024-09-16T10:35:36.250309+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:36.250309+0000","last_undegraded":"2024-09-16T10:35:36.250309+0000","last_fullsized":"2024-09-16T10:35:36.250309+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":22261,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:54.323202+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:35:54.323202+0000","last_peered":"2024-09-16T10:35:54.323202+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:54.323202+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286993+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:52.286993+0000","last_peered":"2024-09-16T10:35:52.286993+0000","last_clean":"2024-09-16T10:35:52.286993+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:52.286993+0000","last_undegraded":"2024-09-16T10:35:52.286993+0000","last_fullsized":"2024-09-16T10:35:52.286993+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":80,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250185+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:36.250185+0000","last_peered":"2024-09-16T10:35:36.250185+0000","last_clean":"2024-09-16T10:35:36.250185+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:36.250185+0000","last_undegraded":"2024-09-16T10:35:36.250185+0000","last_fullsized":"2024-09-16T10:35:36.250185+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29129,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:50.376681+0000","last_change":"2024-09-16T10:35:52.289743+0000","last_active":"2024-09-16T10:35:50.376681+0000","last_peered":"2024-09-16T10:35:50.376681+0000","last_clean":"2024-09-16T10:35:50.376681+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:50.376681+0000","last_undegraded":"2024-09-16T10:35:50.376681+0000","last_fullsized":"2024-09-16T10:35:50.376681+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29094,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:48.367025+0000","last_change":"2024-09-16T10:35:52.289501+0000","last_active":"2024-09-16T10:35:48.367025+0000","last_peered":"2024-09-16T10:35:48.367025+0000","last_clean":"2024-09-16T10:35:48.367025+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:48.367025+0000","last_undegraded":"2024-09-16T10:35:48.367025+0000","last_fullsized":"2024-09-16T10:35:48.367025+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":80,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250627+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:36.250627+0000","last_peered":"2024-09-16T10:35:36.250627+0000","last_clean":"2024-09-16T10:35:36.250627+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:36.250627+0000","last_undegraded":"2024-09-16T10:35:36.250627+0000","last_fullsized":"2024-09-16T10:35:36.250627+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47512,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104714240,"data_stored":104707014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019041,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381396,"kb_used_data":34760,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990444,"statfs":{"total":96636764160,"available":96246214656,"internally_reserved":0,"allocated":35594240,"data_stored":35247054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.22800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.25700000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691985,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505492,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866348,"statfs":{"total":96636764160,"available":96119140352,"internally_reserved":0,"allocated":532480,"data_stored":177604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.14899999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.17199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.219}]}]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.254},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.27100000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:55.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:55.089 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 22261, 'reported_epoch': 64, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:54.323202+0000', 'last_change': '2024-09-16T10:35:51.459459+0000', 'last_active': '2024-09-16T10:35:54.323202+0000', 'last_peered': '2024-09-16T10:35:54.323202+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:54.323202+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3834, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 571, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3834}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:55.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:35:55.484 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:55.485 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3834/40 objects degraded (9585.000%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":844,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":64,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1444126720,"bytes_avail":288466165760,"bytes_total":289910292480,"degraded_objects":3834,"degraded_total":40,"degraded_ratio":95.849999999999994,"recovering_objects_per_sec":317,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2543,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:35:55.500 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:35:55.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:55.811 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:55.812 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:55.827 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-16T10:35:54.785957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47513,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":54,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1410280,"kb_used_data":103808,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281705240,"statfs":{"total":289910292480,"available":288466165760,"internally_reserved":0,"allocated":106299392,"data_stored":105240610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2543,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2543,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001408"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20642,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250477+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:36.250477+0000","last_peered":"2024-09-16T10:35:36.250477+0000","last_clean":"2024-09-16T10:35:36.250477+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:36.250477+0000","last_undegraded":"2024-09-16T10:35:36.250477+0000","last_fullsized":"2024-09-16T10:35:36.250477+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20541,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250309+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:36.250309+0000","last_peered":"2024-09-16T10:35:36.250309+0000","last_clean":"2024-09-16T10:35:36.250309+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:36.250309+0000","last_undegraded":"2024-09-16T10:35:36.250309+0000","last_fullsized":"2024-09-16T10:35:36.250309+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":22261,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:54.323202+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:35:54.323202+0000","last_peered":"2024-09-16T10:35:54.323202+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:54.323202+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286993+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:52.286993+0000","last_peered":"2024-09-16T10:35:52.286993+0000","last_clean":"2024-09-16T10:35:52.286993+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:52.286993+0000","last_undegraded":"2024-09-16T10:35:52.286993+0000","last_fullsized":"2024-09-16T10:35:52.286993+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":80,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250185+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:36.250185+0000","last_peered":"2024-09-16T10:35:36.250185+0000","last_clean":"2024-09-16T10:35:36.250185+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:36.250185+0000","last_undegraded":"2024-09-16T10:35:36.250185+0000","last_fullsized":"2024-09-16T10:35:36.250185+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29129,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:50.376681+0000","last_change":"2024-09-16T10:35:52.289743+0000","last_active":"2024-09-16T10:35:50.376681+0000","last_peered":"2024-09-16T10:35:50.376681+0000","last_clean":"2024-09-16T10:35:50.376681+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:50.376681+0000","last_undegraded":"2024-09-16T10:35:50.376681+0000","last_fullsized":"2024-09-16T10:35:50.376681+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29094,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:48.367025+0000","last_change":"2024-09-16T10:35:52.289501+0000","last_active":"2024-09-16T10:35:48.367025+0000","last_peered":"2024-09-16T10:35:48.367025+0000","last_clean":"2024-09-16T10:35:48.367025+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:48.367025+0000","last_undegraded":"2024-09-16T10:35:48.367025+0000","last_fullsized":"2024-09-16T10:35:48.367025+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":80,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-16T10:35:36.250627+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:36.250627+0000","last_peered":"2024-09-16T10:35:36.250627+0000","last_clean":"2024-09-16T10:35:36.250627+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:36.250627+0000","last_undegraded":"2024-09-16T10:35:36.250627+0000","last_fullsized":"2024-09-16T10:35:36.250627+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47512,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104714240,"data_stored":104707014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019041,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381396,"kb_used_data":34760,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":93990444,"statfs":{"total":96636764160,"available":96246214656,"internally_reserved":0,"allocated":35594240,"data_stored":35247054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.22800000000000001}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.25700000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691985,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505492,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866348,"statfs":{"total":96636764160,"available":96119140352,"internally_reserved":0,"allocated":532480,"data_stored":177604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.14899999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.17199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.219}]}]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.254},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.27100000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":35069952,"data_stored":35069952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:58.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:59.195 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:59.196 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:59.209 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-16T10:35:58.786604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47513,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1385468,"kb_used_data":87188,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281730052,"statfs":{"total":289910292480,"available":288491573248,"internally_reserved":0,"allocated":89280512,"data_stored":88210814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1329384450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001398"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20643,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289804+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:52.289804+0000","last_peered":"2024-09-16T10:35:52.289804+0000","last_clean":"2024-09-16T10:35:52.289804+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:52.289804+0000","last_undegraded":"2024-09-16T10:35:52.289804+0000","last_fullsized":"2024-09-16T10:35:52.289804+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20542,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.290019+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:52.290019+0000","last_peered":"2024-09-16T10:35:52.290019+0000","last_clean":"2024-09-16T10:35:52.290019+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:52.290019+0000","last_undegraded":"2024-09-16T10:35:52.290019+0000","last_fullsized":"2024-09-16T10:35:52.290019+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":22261,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:54.323202+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:35:54.323202+0000","last_peered":"2024-09-16T10:35:54.323202+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:54.323202+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289666+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:52.289666+0000","last_peered":"2024-09-16T10:35:52.289666+0000","last_clean":"2024-09-16T10:35:52.289666+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:52.289666+0000","last_undegraded":"2024-09-16T10:35:52.289666+0000","last_fullsized":"2024-09-16T10:35:52.289666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29131,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:53.290737+0000","last_change":"2024-09-16T10:35:56.291139+0000","last_active":"2024-09-16T10:35:53.290737+0000","last_peered":"2024-09-16T10:35:53.290737+0000","last_clean":"2024-09-16T10:35:53.290737+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:53.290737+0000","last_undegraded":"2024-09-16T10:35:53.290737+0000","last_fullsized":"2024-09-16T10:35:53.290737+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29096,"reported_epoch":64,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:53.290513+0000","last_change":"2024-09-16T10:35:53.312862+0000","last_active":"2024-09-16T10:35:53.290513+0000","last_peered":"2024-09-16T10:35:53.290513+0000","last_clean":"2024-09-16T10:35:53.290513+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:53.290513+0000","last_undegraded":"2024-09-16T10:35:53.290513+0000","last_fullsized":"2024-09-16T10:35:53.290513+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289709+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:52.289709+0000","last_peered":"2024-09-16T10:35:52.289709+0000","last_clean":"2024-09-16T10:35:52.289709+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:52.289709+0000","last_undegraded":"2024-09-16T10:35:52.289709+0000","last_fullsized":"2024-09-16T10:35:52.289709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47512,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87678976,"data_stored":87671750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019042,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364772,"kb_used_data":18132,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94007068,"statfs":{"total":96636764160,"available":96263237632,"internally_reserved":0,"allocated":18567168,"data_stored":18214524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.252}]}]},{"osd":1,"up_from":40,"seq":171798691986,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497304,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874536,"statfs":{"total":96636764160,"available":96127524864,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":508685994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.313}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.28999999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.254},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.27100000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18034688,"data_stored":18034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:59.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:35:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:35:59.590 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:35:59.602 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-16T10:35:58.786604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47513,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1385468,"kb_used_data":87188,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281730052,"statfs":{"total":289910292480,"available":288491573248,"internally_reserved":0,"allocated":89280512,"data_stored":88210814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1329384450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001398"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20643,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289804+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:52.289804+0000","last_peered":"2024-09-16T10:35:52.289804+0000","last_clean":"2024-09-16T10:35:52.289804+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:52.289804+0000","last_undegraded":"2024-09-16T10:35:52.289804+0000","last_fullsized":"2024-09-16T10:35:52.289804+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20542,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.290019+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:52.290019+0000","last_peered":"2024-09-16T10:35:52.290019+0000","last_clean":"2024-09-16T10:35:52.290019+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:52.290019+0000","last_undegraded":"2024-09-16T10:35:52.290019+0000","last_fullsized":"2024-09-16T10:35:52.290019+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":22261,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:54.323202+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:35:54.323202+0000","last_peered":"2024-09-16T10:35:54.323202+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:54.323202+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289666+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:52.289666+0000","last_peered":"2024-09-16T10:35:52.289666+0000","last_clean":"2024-09-16T10:35:52.289666+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:52.289666+0000","last_undegraded":"2024-09-16T10:35:52.289666+0000","last_fullsized":"2024-09-16T10:35:52.289666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29131,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:53.290737+0000","last_change":"2024-09-16T10:35:56.291139+0000","last_active":"2024-09-16T10:35:53.290737+0000","last_peered":"2024-09-16T10:35:53.290737+0000","last_clean":"2024-09-16T10:35:53.290737+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:53.290737+0000","last_undegraded":"2024-09-16T10:35:53.290737+0000","last_fullsized":"2024-09-16T10:35:53.290737+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29096,"reported_epoch":64,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:53.290513+0000","last_change":"2024-09-16T10:35:53.312862+0000","last_active":"2024-09-16T10:35:53.290513+0000","last_peered":"2024-09-16T10:35:53.290513+0000","last_clean":"2024-09-16T10:35:53.290513+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:53.290513+0000","last_undegraded":"2024-09-16T10:35:53.290513+0000","last_fullsized":"2024-09-16T10:35:53.290513+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289709+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:52.289709+0000","last_peered":"2024-09-16T10:35:52.289709+0000","last_clean":"2024-09-16T10:35:52.289709+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:52.289709+0000","last_undegraded":"2024-09-16T10:35:52.289709+0000","last_fullsized":"2024-09-16T10:35:52.289709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47512,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87678976,"data_stored":87671750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019042,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364772,"kb_used_data":18132,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94007068,"statfs":{"total":96636764160,"available":96263237632,"internally_reserved":0,"allocated":18567168,"data_stored":18214524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.252}]}]},{"osd":1,"up_from":40,"seq":171798691986,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497304,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874536,"statfs":{"total":96636764160,"available":96127524864,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":508685994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.313}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.28999999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.254},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.27100000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18034688,"data_stored":18034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:35:59.603 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:35:59.604 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 22261, 'reported_epoch': 64, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:54.323202+0000', 'last_change': '2024-09-16T10:35:51.459459+0000', 'last_active': '2024-09-16T10:35:54.323202+0000', 'last_peered': '2024-09-16T10:35:54.323202+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:54.323202+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3834, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 571, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -3834}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:35:59.604 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:35:59.604 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29131, 'reported_epoch': 64, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:53.290737+0000', 'last_change': '2024-09-16T10:35:56.291139+0000', 'last_active': '2024-09-16T10:35:53.290737+0000', 'last_peered': '2024-09-16T10:35:53.290737+0000', 'last_clean': '2024-09-16T10:35:53.290737+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:35:53.290737+0000', 'last_undegraded': '2024-09-16T10:35:53.290737+0000', 'last_fullsized': '2024-09-16T10:35:53.290737+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:59.604 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:35:59.604 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29096, 'reported_epoch': 64, 'state': 'active+clean+scrubbing+deep', 'last_fresh': '2024-09-16T10:35:53.290513+0000', 'last_change': '2024-09-16T10:35:53.312862+0000', 'last_active': '2024-09-16T10:35:53.290513+0000', 'last_peered': '2024-09-16T10:35:53.290513+0000', 'last_clean': '2024-09-16T10:35:53.290513+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:53.290513+0000', 'last_undegraded': '2024-09-16T10:35:53.290513+0000', 'last_fullsized': '2024-09-16T10:35:53.290513+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:35:59.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:36:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:00.070 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3834/40 objects degraded (9585.000%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":848,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":64,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean+scrubbing+deep","count":1},{"state_name":"active+clean+scrubbing","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1418719232,"bytes_avail":288491573248,"bytes_total":289910292480,"degraded_objects":3834,"degraded_total":40,"degraded_ratio":95.849999999999994,"recovering_objects_per_sec":135,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1085,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:36:00.084 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:36:00.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:00.407 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:00.408 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:00.422 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-16T10:35:58.786604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47513,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1385468,"kb_used_data":87188,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281730052,"statfs":{"total":289910292480,"available":288491573248,"internally_reserved":0,"allocated":89280512,"data_stored":88210814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1329384450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1085,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001398"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20643,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289804+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:52.289804+0000","last_peered":"2024-09-16T10:35:52.289804+0000","last_clean":"2024-09-16T10:35:52.289804+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:52.289804+0000","last_undegraded":"2024-09-16T10:35:52.289804+0000","last_fullsized":"2024-09-16T10:35:52.289804+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20542,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.290019+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:52.290019+0000","last_peered":"2024-09-16T10:35:52.290019+0000","last_clean":"2024-09-16T10:35:52.290019+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:52.290019+0000","last_undegraded":"2024-09-16T10:35:52.290019+0000","last_fullsized":"2024-09-16T10:35:52.290019+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":22261,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:54.323202+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:35:54.323202+0000","last_peered":"2024-09-16T10:35:54.323202+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:54.323202+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-3834}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289666+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:52.289666+0000","last_peered":"2024-09-16T10:35:52.289666+0000","last_clean":"2024-09-16T10:35:52.289666+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:52.289666+0000","last_undegraded":"2024-09-16T10:35:52.289666+0000","last_fullsized":"2024-09-16T10:35:52.289666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29131,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:53.290737+0000","last_change":"2024-09-16T10:35:56.291139+0000","last_active":"2024-09-16T10:35:53.290737+0000","last_peered":"2024-09-16T10:35:53.290737+0000","last_clean":"2024-09-16T10:35:53.290737+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:53.290737+0000","last_undegraded":"2024-09-16T10:35:53.290737+0000","last_fullsized":"2024-09-16T10:35:53.290737+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29096,"reported_epoch":64,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-16T10:35:53.290513+0000","last_change":"2024-09-16T10:35:53.312862+0000","last_active":"2024-09-16T10:35:53.290513+0000","last_peered":"2024-09-16T10:35:53.290513+0000","last_clean":"2024-09-16T10:35:53.290513+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:53.290513+0000","last_undegraded":"2024-09-16T10:35:53.290513+0000","last_fullsized":"2024-09-16T10:35:53.290513+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289709+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:52.289709+0000","last_peered":"2024-09-16T10:35:52.289709+0000","last_clean":"2024-09-16T10:35:52.289709+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:52.289709+0000","last_undegraded":"2024-09-16T10:35:52.289709+0000","last_fullsized":"2024-09-16T10:35:52.289709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47512,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87678976,"data_stored":87671750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019042,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364772,"kb_used_data":18132,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94007068,"statfs":{"total":96636764160,"available":96263237632,"internally_reserved":0,"allocated":18567168,"data_stored":18214524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.252}]}]},{"osd":1,"up_from":40,"seq":171798691986,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497304,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874536,"statfs":{"total":96636764160,"available":96127524864,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":508685994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.313}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.28999999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215268,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.254},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.27100000000000002}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18034688,"data_stored":18034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:03.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:03.774 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:03.774 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:03.787 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-16T10:36:02.787318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48550,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1385468,"kb_used_data":87188,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281730052,"statfs":{"total":289910292480,"available":288491573248,"internally_reserved":0,"allocated":89280512,"data_stored":88210814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1329384450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001738"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20643,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289804+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:52.289804+0000","last_peered":"2024-09-16T10:35:52.289804+0000","last_clean":"2024-09-16T10:35:52.289804+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:52.289804+0000","last_undegraded":"2024-09-16T10:35:52.289804+0000","last_fullsized":"2024-09-16T10:35:52.289804+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20542,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.290019+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:52.290019+0000","last_peered":"2024-09-16T10:35:52.290019+0000","last_clean":"2024-09-16T10:35:52.290019+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:52.290019+0000","last_undegraded":"2024-09-16T10:35:52.290019+0000","last_fullsized":"2024-09-16T10:35:52.290019+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":24335,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:59.323085+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:35:59.323085+0000","last_peered":"2024-09-16T10:35:59.323085+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:59.323085+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1608,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2797}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289666+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:52.289666+0000","last_peered":"2024-09-16T10:35:52.289666+0000","last_clean":"2024-09-16T10:35:52.289666+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:52.289666+0000","last_undegraded":"2024-09-16T10:35:52.289666+0000","last_fullsized":"2024-09-16T10:35:52.289666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29132,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:56.291139+0000","last_change":"2024-09-16T10:35:56.291139+0000","last_active":"2024-09-16T10:35:56.291139+0000","last_peered":"2024-09-16T10:35:56.291139+0000","last_clean":"2024-09-16T10:35:56.291139+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:56.291139+0000","last_undegraded":"2024-09-16T10:35:56.291139+0000","last_fullsized":"2024-09-16T10:35:56.291139+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29097,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:53.312862+0000","last_change":"2024-09-16T10:35:59.241543+0000","last_active":"2024-09-16T10:35:53.312862+0000","last_peered":"2024-09-16T10:35:53.312862+0000","last_clean":"2024-09-16T10:35:53.312862+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:53.312862+0000","last_undegraded":"2024-09-16T10:35:53.312862+0000","last_fullsized":"2024-09-16T10:35:53.312862+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289709+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:52.289709+0000","last_peered":"2024-09-16T10:35:52.289709+0000","last_clean":"2024-09-16T10:35:52.289709+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:52.289709+0000","last_undegraded":"2024-09-16T10:35:52.289709+0000","last_fullsized":"2024-09-16T10:35:52.289709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48549,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87678976,"data_stored":87671750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019043,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364772,"kb_used_data":18132,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94007068,"statfs":{"total":96636764160,"available":96263237632,"internally_reserved":0,"allocated":18567168,"data_stored":18214524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.252}]}]},{"osd":1,"up_from":40,"seq":171798691986,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497304,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874536,"statfs":{"total":96636764160,"available":96127524864,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":508685994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.313}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.28999999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215269,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.191}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18034688,"data_stored":18034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:03.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:04.108 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:04.108 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:04.122 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-16T10:36:02.787318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48550,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1385468,"kb_used_data":87188,"kb_used_omap":12,"kb_used_meta":1298227,"kb_avail":281730052,"statfs":{"total":289910292480,"available":288491573248,"internally_reserved":0,"allocated":89280512,"data_stored":88210814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1329384450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001738"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20643,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289804+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:52.289804+0000","last_peered":"2024-09-16T10:35:52.289804+0000","last_clean":"2024-09-16T10:35:52.289804+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:52.289804+0000","last_undegraded":"2024-09-16T10:35:52.289804+0000","last_fullsized":"2024-09-16T10:35:52.289804+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20542,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.290019+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:52.290019+0000","last_peered":"2024-09-16T10:35:52.290019+0000","last_clean":"2024-09-16T10:35:52.290019+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:52.290019+0000","last_undegraded":"2024-09-16T10:35:52.290019+0000","last_fullsized":"2024-09-16T10:35:52.290019+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":24335,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:35:59.323085+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:35:59.323085+0000","last_peered":"2024-09-16T10:35:59.323085+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:35:59.323085+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1608,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-2797}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289666+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:52.289666+0000","last_peered":"2024-09-16T10:35:52.289666+0000","last_clean":"2024-09-16T10:35:52.289666+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:52.289666+0000","last_undegraded":"2024-09-16T10:35:52.289666+0000","last_fullsized":"2024-09-16T10:35:52.289666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29132,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:56.291139+0000","last_change":"2024-09-16T10:35:56.291139+0000","last_active":"2024-09-16T10:35:56.291139+0000","last_peered":"2024-09-16T10:35:56.291139+0000","last_clean":"2024-09-16T10:35:56.291139+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:56.291139+0000","last_undegraded":"2024-09-16T10:35:56.291139+0000","last_fullsized":"2024-09-16T10:35:56.291139+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29097,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:53.312862+0000","last_change":"2024-09-16T10:35:59.241543+0000","last_active":"2024-09-16T10:35:53.312862+0000","last_peered":"2024-09-16T10:35:53.312862+0000","last_clean":"2024-09-16T10:35:53.312862+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:53.312862+0000","last_undegraded":"2024-09-16T10:35:53.312862+0000","last_fullsized":"2024-09-16T10:35:53.312862+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289709+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:52.289709+0000","last_peered":"2024-09-16T10:35:52.289709+0000","last_clean":"2024-09-16T10:35:52.289709+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:52.289709+0000","last_undegraded":"2024-09-16T10:35:52.289709+0000","last_fullsized":"2024-09-16T10:35:52.289709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48549,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87678976,"data_stored":87671750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019043,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364772,"kb_used_data":18132,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94007068,"statfs":{"total":96636764160,"available":96263237632,"internally_reserved":0,"allocated":18567168,"data_stored":18214524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.252}]}]},{"osd":1,"up_from":40,"seq":171798691986,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497304,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93874536,"statfs":{"total":96636764160,"available":96127524864,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":508685994},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.313}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.28999999999999998}]}]},{"osd":0,"up_from":24,"seq":103079215269,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.191}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18034688,"data_stored":18034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:04.123 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:36:04.123 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 24335, 'reported_epoch': 64, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:35:59.323085+0000', 'last_change': '2024-09-16T10:35:51.459459+0000', 'last_active': '2024-09-16T10:35:59.323085+0000', 'last_peered': '2024-09-16T10:35:59.323085+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:35:59.323085+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2797, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1608, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -2797}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:36:04.123 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:36:04.123 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29132, 'reported_epoch': 64, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:56.291139+0000', 'last_change': '2024-09-16T10:35:56.291139+0000', 'last_active': '2024-09-16T10:35:56.291139+0000', 'last_peered': '2024-09-16T10:35:56.291139+0000', 'last_clean': '2024-09-16T10:35:56.291139+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:35:56.291139+0000', 'last_undegraded': '2024-09-16T10:35:56.291139+0000', 'last_fullsized': '2024-09-16T10:35:56.291139+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:36:04.123 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:36:04.123 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29097, 'reported_epoch': 64, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:53.312862+0000', 'last_change': '2024-09-16T10:35:59.241543+0000', 'last_active': '2024-09-16T10:35:53.312862+0000', 'last_peered': '2024-09-16T10:35:53.312862+0000', 'last_clean': '2024-09-16T10:35:53.312862+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:53.312862+0000', 'last_undegraded': '2024-09-16T10:35:53.312862+0000', 'last_fullsized': '2024-09-16T10:35:53.312862+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:36:04.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:36:04.529 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:04.530 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2797/40 objects degraded (6992.500%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":853,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":64,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+clean+scrubbing","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1418719232,"bytes_avail":288491573248,"bytes_total":289910292480,"degraded_objects":2797,"degraded_total":40,"degraded_ratio":69.924999999999997,"recovering_objects_per_sec":212,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2122,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:36:04.544 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:36:04.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:04.881 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:04.881 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:04.895 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-09-16T10:36:04.787609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49760,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1393660,"kb_used_data":87188,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281721860,"statfs":{"total":289910292480,"available":288483184640,"internally_reserved":0,"allocated":89280512,"data_stored":88210814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3332,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.498060"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20643,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289804+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:52.289804+0000","last_peered":"2024-09-16T10:35:52.289804+0000","last_clean":"2024-09-16T10:35:52.289804+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:52.289804+0000","last_undegraded":"2024-09-16T10:35:52.289804+0000","last_fullsized":"2024-09-16T10:35:52.289804+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20542,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.290019+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:52.290019+0000","last_peered":"2024-09-16T10:35:52.290019+0000","last_clean":"2024-09-16T10:35:52.290019+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:52.290019+0000","last_undegraded":"2024-09-16T10:35:52.290019+0000","last_fullsized":"2024-09-16T10:35:52.290019+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":26755,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:36:04.326424+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:36:04.326424+0000","last_peered":"2024-09-16T10:36:04.326424+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:36:04.326424+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2818,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289666+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:52.289666+0000","last_peered":"2024-09-16T10:35:52.289666+0000","last_clean":"2024-09-16T10:35:52.289666+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:52.289666+0000","last_undegraded":"2024-09-16T10:35:52.289666+0000","last_fullsized":"2024-09-16T10:35:52.289666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29132,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:56.291139+0000","last_change":"2024-09-16T10:35:56.291139+0000","last_active":"2024-09-16T10:35:56.291139+0000","last_peered":"2024-09-16T10:35:56.291139+0000","last_clean":"2024-09-16T10:35:56.291139+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:56.291139+0000","last_undegraded":"2024-09-16T10:35:56.291139+0000","last_fullsized":"2024-09-16T10:35:56.291139+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29097,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:53.312862+0000","last_change":"2024-09-16T10:35:59.241543+0000","last_active":"2024-09-16T10:35:53.312862+0000","last_peered":"2024-09-16T10:35:53.312862+0000","last_clean":"2024-09-16T10:35:53.312862+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:53.312862+0000","last_undegraded":"2024-09-16T10:35:53.312862+0000","last_fullsized":"2024-09-16T10:35:53.312862+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289709+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:52.289709+0000","last_peered":"2024-09-16T10:35:52.289709+0000","last_clean":"2024-09-16T10:35:52.289709+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:52.289709+0000","last_undegraded":"2024-09-16T10:35:52.289709+0000","last_fullsized":"2024-09-16T10:35:52.289709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49759,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87678976,"data_stored":87671750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019043,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364772,"kb_used_data":18132,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94007068,"statfs":{"total":96636764160,"available":96263237632,"internally_reserved":0,"allocated":18567168,"data_stored":18214524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.252}]}]},{"osd":1,"up_from":40,"seq":171798691987,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505496,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866344,"statfs":{"total":96636764160,"available":96119136256,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.215},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27700000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215270,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.223},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.21199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18034688,"data_stored":18034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:07.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:08.250 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:08.251 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:08.263 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-16T10:36:06.787931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49760,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1393660,"kb_used_data":87188,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281721860,"statfs":{"total":289910292480,"available":288483184640,"internally_reserved":0,"allocated":89280512,"data_stored":88210814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001973"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20643,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289804+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:52.289804+0000","last_peered":"2024-09-16T10:35:52.289804+0000","last_clean":"2024-09-16T10:35:52.289804+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:52.289804+0000","last_undegraded":"2024-09-16T10:35:52.289804+0000","last_fullsized":"2024-09-16T10:35:52.289804+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20542,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.290019+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:52.290019+0000","last_peered":"2024-09-16T10:35:52.290019+0000","last_clean":"2024-09-16T10:35:52.290019+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:52.290019+0000","last_undegraded":"2024-09-16T10:35:52.290019+0000","last_fullsized":"2024-09-16T10:35:52.290019+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":26755,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:36:04.326424+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:36:04.326424+0000","last_peered":"2024-09-16T10:36:04.326424+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:36:04.326424+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2818,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289666+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:52.289666+0000","last_peered":"2024-09-16T10:35:52.289666+0000","last_clean":"2024-09-16T10:35:52.289666+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:52.289666+0000","last_undegraded":"2024-09-16T10:35:52.289666+0000","last_fullsized":"2024-09-16T10:35:52.289666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29132,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:56.291139+0000","last_change":"2024-09-16T10:35:56.291139+0000","last_active":"2024-09-16T10:35:56.291139+0000","last_peered":"2024-09-16T10:35:56.291139+0000","last_clean":"2024-09-16T10:35:56.291139+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:56.291139+0000","last_undegraded":"2024-09-16T10:35:56.291139+0000","last_fullsized":"2024-09-16T10:35:56.291139+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29097,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:53.312862+0000","last_change":"2024-09-16T10:35:59.241543+0000","last_active":"2024-09-16T10:35:53.312862+0000","last_peered":"2024-09-16T10:35:53.312862+0000","last_clean":"2024-09-16T10:35:53.312862+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:53.312862+0000","last_undegraded":"2024-09-16T10:35:53.312862+0000","last_fullsized":"2024-09-16T10:35:53.312862+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289709+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:52.289709+0000","last_peered":"2024-09-16T10:35:52.289709+0000","last_clean":"2024-09-16T10:35:52.289709+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:52.289709+0000","last_undegraded":"2024-09-16T10:35:52.289709+0000","last_fullsized":"2024-09-16T10:35:52.289709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49759,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87678976,"data_stored":87671750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019043,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364772,"kb_used_data":18132,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94007068,"statfs":{"total":96636764160,"available":96263237632,"internally_reserved":0,"allocated":18567168,"data_stored":18214524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.252}]}]},{"osd":1,"up_from":40,"seq":171798691987,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505496,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866344,"statfs":{"total":96636764160,"available":96119136256,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.215},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27700000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215270,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.223},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.21199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18034688,"data_stored":18034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:08.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:08.586 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:08.586 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:08.598 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-16T10:36:06.787931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49760,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1393660,"kb_used_data":87188,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281721860,"statfs":{"total":289910292480,"available":288483184640,"internally_reserved":0,"allocated":89280512,"data_stored":88210814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001973"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20643,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289804+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:52.289804+0000","last_peered":"2024-09-16T10:35:52.289804+0000","last_clean":"2024-09-16T10:35:52.289804+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:52.289804+0000","last_undegraded":"2024-09-16T10:35:52.289804+0000","last_fullsized":"2024-09-16T10:35:52.289804+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20542,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.290019+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:52.290019+0000","last_peered":"2024-09-16T10:35:52.290019+0000","last_clean":"2024-09-16T10:35:52.290019+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:52.290019+0000","last_undegraded":"2024-09-16T10:35:52.290019+0000","last_fullsized":"2024-09-16T10:35:52.290019+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":26755,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:36:04.326424+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:36:04.326424+0000","last_peered":"2024-09-16T10:36:04.326424+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:36:04.326424+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2818,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289666+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:52.289666+0000","last_peered":"2024-09-16T10:35:52.289666+0000","last_clean":"2024-09-16T10:35:52.289666+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:52.289666+0000","last_undegraded":"2024-09-16T10:35:52.289666+0000","last_fullsized":"2024-09-16T10:35:52.289666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29132,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:56.291139+0000","last_change":"2024-09-16T10:35:56.291139+0000","last_active":"2024-09-16T10:35:56.291139+0000","last_peered":"2024-09-16T10:35:56.291139+0000","last_clean":"2024-09-16T10:35:56.291139+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:35:56.291139+0000","last_undegraded":"2024-09-16T10:35:56.291139+0000","last_fullsized":"2024-09-16T10:35:56.291139+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29097,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:53.312862+0000","last_change":"2024-09-16T10:35:59.241543+0000","last_active":"2024-09-16T10:35:53.312862+0000","last_peered":"2024-09-16T10:35:53.312862+0000","last_clean":"2024-09-16T10:35:53.312862+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:53.312862+0000","last_undegraded":"2024-09-16T10:35:53.312862+0000","last_fullsized":"2024-09-16T10:35:53.312862+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289709+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:52.289709+0000","last_peered":"2024-09-16T10:35:52.289709+0000","last_clean":"2024-09-16T10:35:52.289709+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:52.289709+0000","last_undegraded":"2024-09-16T10:35:52.289709+0000","last_fullsized":"2024-09-16T10:35:52.289709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49759,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87678976,"data_stored":87671750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019043,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364772,"kb_used_data":18132,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94007068,"statfs":{"total":96636764160,"available":96263237632,"internally_reserved":0,"allocated":18567168,"data_stored":18214524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.252}]}]},{"osd":1,"up_from":40,"seq":171798691987,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505496,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866344,"statfs":{"total":96636764160,"available":96119136256,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.28899999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.215},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27700000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215270,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.223},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.21199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18034688,"data_stored":18034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:08.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:36:08.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 26755, 'reported_epoch': 64, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:36:04.326424+0000', 'last_change': '2024-09-16T10:35:51.459459+0000', 'last_active': '2024-09-16T10:36:04.326424+0000', 'last_peered': '2024-09-16T10:36:04.326424+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:36:04.326424+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1587, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2818, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1587}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:36:08.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.1 is not active+clean 2024-09-16T10:36:08.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.1', 'version': "48'20419", 'reported_seq': 29132, 'reported_epoch': 64, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:56.291139+0000', 'last_change': '2024-09-16T10:35:56.291139+0000', 'last_active': '2024-09-16T10:35:56.291139+0000', 'last_peered': '2024-09-16T10:35:56.291139+0000', 'last_clean': '2024-09-16T10:35:56.291139+0000', 'last_became_active': '2024-09-16T10:34:35.234178+0000', 'last_became_peered': '2024-09-16T10:34:35.234178+0000', 'last_unstale': '2024-09-16T10:35:56.291139+0000', 'last_undegraded': '2024-09-16T10:35:56.291139+0000', 'last_fullsized': '2024-09-16T10:35:56.291139+0000', 'mapping_epoch': 55, 'log_start': "44'10400", 'ondisk_log_start': "44'10400", 'created': 15, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 799, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20417, 'num_write_kb': 40831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4281, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:36:08.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.0 is not active+clean 2024-09-16T10:36:08.600 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.0', 'version': "48'20576", 'reported_seq': 29097, 'reported_epoch': 64, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-16T10:35:53.312862+0000', 'last_change': '2024-09-16T10:35:59.241543+0000', 'last_active': '2024-09-16T10:35:53.312862+0000', 'last_peered': '2024-09-16T10:35:53.312862+0000', 'last_clean': '2024-09-16T10:35:53.312862+0000', 'last_became_active': '2024-09-16T10:31:29.304076+0000', 'last_became_peered': '2024-09-16T10:31:29.304076+0000', 'last_unstale': '2024-09-16T10:35:53.312862+0000', 'last_undegraded': '2024-09-16T10:35:53.312862+0000', 'last_fullsized': '2024-09-16T10:35:53.312862+0000', 'mapping_epoch': 47, 'log_start': "44'10500", 'ondisk_log_start': "44'10500", 'created': 15, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'queued for scrub', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 966, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 20574, 'num_write_kb': 41143, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4196, 'num_bytes_recovered': 5328896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-16T10:36:08.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:36:09.003 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:09.003 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1587/40 objects degraded (3967.500%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":857,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":64,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21},{"state_name":"active+clean+scrubbing","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":20,"data_bytes":8713,"bytes_used":1427107840,"bytes_avail":288483184640,"bytes_total":289910292480,"degraded_objects":1587,"degraded_total":40,"degraded_ratio":39.674999999999997,"recovering_objects_per_sec":187,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2247,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:36:09.016 INFO:tasks.rep_lost_unfound_delete.ceph_manager:making progress, resetting timeout 2024-09-16T10:36:09.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:09.385 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:09.386 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:09.400 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-16T10:36:09.317736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49760,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1389564,"kb_used_data":87188,"kb_used_omap":12,"kb_used_meta":1302323,"kb_avail":281725956,"statfs":{"total":289910292480,"available":288487378944,"internally_reserved":0,"allocated":89280512,"data_stored":88210814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1333578754},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2247,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.531131"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28953,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287169+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:35:52.287169+0000","last_peered":"2024-09-16T10:35:52.287169+0000","last_clean":"2024-09-16T10:35:52.287169+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:35:52.287169+0000","last_undegraded":"2024-09-16T10:35:52.287169+0000","last_fullsized":"2024-09-16T10:35:52.287169+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20643,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289804+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:52.289804+0000","last_peered":"2024-09-16T10:35:52.289804+0000","last_clean":"2024-09-16T10:35:52.289804+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:52.289804+0000","last_undegraded":"2024-09-16T10:35:52.289804+0000","last_fullsized":"2024-09-16T10:35:52.289804+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20542,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.290019+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:52.290019+0000","last_peered":"2024-09-16T10:35:52.290019+0000","last_clean":"2024-09-16T10:35:52.290019+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:52.290019+0000","last_undegraded":"2024-09-16T10:35:52.290019+0000","last_fullsized":"2024-09-16T10:35:52.290019+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20686,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287007+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:52.287007+0000","last_peered":"2024-09-16T10:35:52.287007+0000","last_clean":"2024-09-16T10:35:52.287007+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:52.287007+0000","last_undegraded":"2024-09-16T10:35:52.287007+0000","last_fullsized":"2024-09-16T10:35:52.287007+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":26755,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:36:04.326424+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:36:04.326424+0000","last_peered":"2024-09-16T10:36:04.326424+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:36:04.326424+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2818,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20589,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287708+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:52.287708+0000","last_peered":"2024-09-16T10:35:52.287708+0000","last_clean":"2024-09-16T10:35:52.287708+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:52.287708+0000","last_undegraded":"2024-09-16T10:35:52.287708+0000","last_fullsized":"2024-09-16T10:35:52.287708+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289666+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:52.289666+0000","last_peered":"2024-09-16T10:35:52.289666+0000","last_clean":"2024-09-16T10:35:52.289666+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:52.289666+0000","last_undegraded":"2024-09-16T10:35:52.289666+0000","last_fullsized":"2024-09-16T10:35:52.289666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30025,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286968+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:52.286968+0000","last_peered":"2024-09-16T10:35:52.286968+0000","last_clean":"2024-09-16T10:35:52.286968+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:52.286968+0000","last_undegraded":"2024-09-16T10:35:52.286968+0000","last_fullsized":"2024-09-16T10:35:52.286968+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.286824+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:52.286824+0000","last_peered":"2024-09-16T10:35:52.286824+0000","last_clean":"2024-09-16T10:35:52.286824+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:52.286824+0000","last_undegraded":"2024-09-16T10:35:52.286824+0000","last_fullsized":"2024-09-16T10:35:52.286824+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29133,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:36:00.238170+0000","last_change":"2024-09-16T10:35:56.291139+0000","last_active":"2024-09-16T10:36:00.238170+0000","last_peered":"2024-09-16T10:36:00.238170+0000","last_clean":"2024-09-16T10:36:00.238170+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:00.238170+0000","last_undegraded":"2024-09-16T10:36:00.238170+0000","last_fullsized":"2024-09-16T10:36:00.238170+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29098,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:59.241543+0000","last_change":"2024-09-16T10:35:59.241543+0000","last_active":"2024-09-16T10:35:59.241543+0000","last_peered":"2024-09-16T10:35:59.241543+0000","last_clean":"2024-09-16T10:35:59.241543+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:59.241543+0000","last_undegraded":"2024-09-16T10:35:59.241543+0000","last_fullsized":"2024-09-16T10:35:59.241543+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289709+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:52.289709+0000","last_peered":"2024-09-16T10:35:52.289709+0000","last_clean":"2024-09-16T10:35:52.289709+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:52.289709+0000","last_undegraded":"2024-09-16T10:35:52.289709+0000","last_fullsized":"2024-09-16T10:35:52.289709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49759,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87678976,"data_stored":87671750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019044,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364772,"kb_used_data":18132,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94007068,"statfs":{"total":96636764160,"available":96263237632,"internally_reserved":0,"allocated":18567168,"data_stored":18214524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23499999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.214},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.251}]}]},{"osd":1,"up_from":40,"seq":171798691988,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501400,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870440,"statfs":{"total":96636764160,"available":96123330560,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":512880298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.23799999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215270,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.223},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.21199999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69640192,"data_stored":69636579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18034688,"data_stored":18034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:12.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:12.702 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:12.702 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:12.714 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-16T10:36:10.788626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8713,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306289,"num_write_kb":567755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49760,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1389564,"kb_used_data":87188,"kb_used_omap":12,"kb_used_meta":1302323,"kb_avail":281725956,"statfs":{"total":289910292480,"available":288487378944,"internally_reserved":0,"allocated":89280512,"data_stored":88210814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1333578754},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1210,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1210,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001307"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28955,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.326474+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:36:09.326474+0000","last_peered":"2024-09-16T10:36:09.326474+0000","last_clean":"2024-09-16T10:36:09.326474+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:09.326474+0000","last_undegraded":"2024-09-16T10:36:09.326474+0000","last_fullsized":"2024-09-16T10:36:09.326474+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20643,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289804+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:35:52.289804+0000","last_peered":"2024-09-16T10:35:52.289804+0000","last_clean":"2024-09-16T10:35:52.289804+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:35:52.289804+0000","last_undegraded":"2024-09-16T10:35:52.289804+0000","last_fullsized":"2024-09-16T10:35:52.289804+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20542,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.290019+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:35:52.290019+0000","last_peered":"2024-09-16T10:35:52.290019+0000","last_clean":"2024-09-16T10:35:52.290019+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:35:52.290019+0000","last_undegraded":"2024-09-16T10:35:52.290019+0000","last_fullsized":"2024-09-16T10:35:52.290019+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20687,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293649+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:53.293649+0000","last_peered":"2024-09-16T10:35:53.293649+0000","last_clean":"2024-09-16T10:35:53.293649+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:53.293649+0000","last_undegraded":"2024-09-16T10:35:53.293649+0000","last_fullsized":"2024-09-16T10:35:53.293649+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":26755,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:36:04.326424+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:36:04.326424+0000","last_peered":"2024-09-16T10:36:04.326424+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:36:04.326424+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2818,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20590,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293448+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:53.293448+0000","last_peered":"2024-09-16T10:35:53.293448+0000","last_clean":"2024-09-16T10:35:53.293448+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:53.293448+0000","last_undegraded":"2024-09-16T10:35:53.293448+0000","last_fullsized":"2024-09-16T10:35:53.293448+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289666+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:35:52.289666+0000","last_peered":"2024-09-16T10:35:52.289666+0000","last_clean":"2024-09-16T10:35:52.289666+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:35:52.289666+0000","last_undegraded":"2024-09-16T10:35:52.289666+0000","last_fullsized":"2024-09-16T10:35:52.289666+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30026,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293601+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:53.293601+0000","last_peered":"2024-09-16T10:35:53.293601+0000","last_clean":"2024-09-16T10:35:53.293601+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:53.293601+0000","last_undegraded":"2024-09-16T10:35:53.293601+0000","last_fullsized":"2024-09-16T10:35:53.293601+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":113,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293650+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:53.293650+0000","last_peered":"2024-09-16T10:35:53.293650+0000","last_clean":"2024-09-16T10:35:53.293650+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:53.293650+0000","last_undegraded":"2024-09-16T10:35:53.293650+0000","last_fullsized":"2024-09-16T10:35:53.293650+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29133,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:36:00.238170+0000","last_change":"2024-09-16T10:35:56.291139+0000","last_active":"2024-09-16T10:36:00.238170+0000","last_peered":"2024-09-16T10:36:00.238170+0000","last_clean":"2024-09-16T10:36:00.238170+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:00.238170+0000","last_undegraded":"2024-09-16T10:36:00.238170+0000","last_fullsized":"2024-09-16T10:36:00.238170+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":799,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20417,"num_write_kb":40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29098,"reported_epoch":64,"state":"active+clean+scrubbing","last_fresh":"2024-09-16T10:35:59.241543+0000","last_change":"2024-09-16T10:35:59.241543+0000","last_active":"2024-09-16T10:35:59.241543+0000","last_peered":"2024-09-16T10:35:59.241543+0000","last_clean":"2024-09-16T10:35:59.241543+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:35:59.241543+0000","last_undegraded":"2024-09-16T10:35:59.241543+0000","last_fullsized":"2024-09-16T10:35:59.241543+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":81,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.289709+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:35:52.289709+0000","last_peered":"2024-09-16T10:35:52.289709+0000","last_clean":"2024-09-16T10:35:52.289709+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:35:52.289709+0000","last_undegraded":"2024-09-16T10:35:52.289709+0000","last_fullsized":"2024-09-16T10:35:52.289709+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":8694,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":306287,"num_write_kb":567753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49759,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":78356480,"data_stored":78349254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019044,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364772,"kb_used_data":18132,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94007068,"statfs":{"total":96636764160,"available":96263237632,"internally_reserved":0,"allocated":18567168,"data_stored":18214524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.21299999999999999,"5min":2.056,"15min":1.5800000000000001},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.27400000000000002,"5min":13.74,"15min":13.74},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.24099999999999999,"5min":1.5509999999999999,"15min":1.6930000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.41099999999999998,"5min":13.743,"15min":13.917},"last":0.23499999999999999}]},{"osd":1,"last update":"Mon Sep 16 10:35:09 2024","interfaces":[{"interface":"back","average":{"1min":0.249,"5min":2.2589999999999999,"15min":2.6749999999999998},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.46300000000000002,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.214},{"interface":"front","average":{"1min":0.26800000000000002,"5min":2.419,"15min":2.5779999999999998},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.38200000000000001,"5min":11.81,"15min":17.914000000000001},"last":0.251}]}]},{"osd":1,"up_from":40,"seq":171798691988,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501400,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870440,"statfs":{"total":96636764160,"available":96123330560,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":512880298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.23799999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215271,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.185},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.247}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60317696,"data_stored":60314083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":18034688,"data_stored":18034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:12.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:13.002 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:13.002 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:13.014 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-16T10:36:12.789029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7747,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285872,"num_write_kb":526924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45479,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1371964,"kb_used_data":69592,"kb_used_omap":12,"kb_used_meta":1302323,"kb_avail":281743556,"statfs":{"total":289910292480,"available":288505401344,"internally_reserved":0,"allocated":71262208,"data_stored":70181565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1333578754},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-966,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-20417,"num_write_kb":-40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001420"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28955,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.326474+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:36:09.326474+0000","last_peered":"2024-09-16T10:36:09.326474+0000","last_clean":"2024-09-16T10:36:09.326474+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:09.326474+0000","last_undegraded":"2024-09-16T10:36:09.326474+0000","last_fullsized":"2024-09-16T10:36:09.326474+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20645,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319359+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:36:09.319359+0000","last_peered":"2024-09-16T10:36:09.319359+0000","last_clean":"2024-09-16T10:36:09.319359+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:09.319359+0000","last_undegraded":"2024-09-16T10:36:09.319359+0000","last_fullsized":"2024-09-16T10:36:09.319359+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20544,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319239+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:36:09.319239+0000","last_peered":"2024-09-16T10:36:09.319239+0000","last_clean":"2024-09-16T10:36:09.319239+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:09.319239+0000","last_undegraded":"2024-09-16T10:36:09.319239+0000","last_fullsized":"2024-09-16T10:36:09.319239+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20687,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293649+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:53.293649+0000","last_peered":"2024-09-16T10:35:53.293649+0000","last_clean":"2024-09-16T10:35:53.293649+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:53.293649+0000","last_undegraded":"2024-09-16T10:35:53.293649+0000","last_fullsized":"2024-09-16T10:35:53.293649+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":26755,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:36:04.326424+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:36:04.326424+0000","last_peered":"2024-09-16T10:36:04.326424+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:36:04.326424+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2818,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20590,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293448+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:53.293448+0000","last_peered":"2024-09-16T10:35:53.293448+0000","last_clean":"2024-09-16T10:35:53.293448+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:53.293448+0000","last_undegraded":"2024-09-16T10:35:53.293448+0000","last_fullsized":"2024-09-16T10:35:53.293448+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319055+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:36:09.319055+0000","last_peered":"2024-09-16T10:36:09.319055+0000","last_clean":"2024-09-16T10:36:09.319055+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:09.319055+0000","last_undegraded":"2024-09-16T10:36:09.319055+0000","last_fullsized":"2024-09-16T10:36:09.319055+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30026,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293601+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:53.293601+0000","last_peered":"2024-09-16T10:35:53.293601+0000","last_clean":"2024-09-16T10:35:53.293601+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:53.293601+0000","last_undegraded":"2024-09-16T10:35:53.293601+0000","last_fullsized":"2024-09-16T10:35:53.293601+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":113,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293650+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:53.293650+0000","last_peered":"2024-09-16T10:35:53.293650+0000","last_clean":"2024-09-16T10:35:53.293650+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:53.293650+0000","last_undegraded":"2024-09-16T10:35:53.293650+0000","last_fullsized":"2024-09-16T10:35:53.293650+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29147,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:11.186312+0000","last_change":"2024-09-16T10:36:11.186430+0000","last_active":"2024-09-16T10:36:11.186312+0000","last_peered":"2024-09-16T10:36:11.186312+0000","last_clean":"2024-09-16T10:36:11.186312+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:11.186312+0000","last_undegraded":"2024-09-16T10:36:11.186312+0000","last_fullsized":"2024-09-16T10:36:11.186312+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:11.186295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:11.186295+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:59.156525+0000","scrub_duration":0.010167803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29101,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.318922+0000","last_change":"2024-09-16T10:36:09.318922+0000","last_active":"2024-09-16T10:36:09.318922+0000","last_peered":"2024-09-16T10:36:09.318922+0000","last_clean":"2024-09-16T10:36:09.318922+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:09.318922+0000","last_undegraded":"2024-09-16T10:36:09.318922+0000","last_fullsized":"2024-09-16T10:36:09.318922+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319329+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:36:09.319329+0000","last_peered":"2024-09-16T10:36:09.319329+0000","last_clean":"2024-09-16T10:36:09.319329+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:09.319329+0000","last_undegraded":"2024-09-16T10:36:09.319329+0000","last_fullsized":"2024-09-16T10:36:09.319329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":7728,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285870,"num_write_kb":526922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45478,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":60321792,"data_stored":60314566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019045,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347172,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024668,"statfs":{"total":96636764160,"available":96281260032,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691988,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501400,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870440,"statfs":{"total":96636764160,"available":96123330560,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":512880298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.23799999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215271,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.185},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.247}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60317696,"data_stored":60314083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:13.015 INFO:tasks.rep_lost_unfound_delete.ceph_manager:PG 2.9 is not active+clean 2024-09-16T10:36:13.015 INFO:tasks.rep_lost_unfound_delete.ceph_manager:{'pgid': '2.9', 'version': "48'20920", 'reported_seq': 26755, 'reported_epoch': 64, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-16T10:36:04.326424+0000', 'last_change': '2024-09-16T10:35:51.459459+0000', 'last_active': '2024-09-16T10:36:04.326424+0000', 'last_peered': '2024-09-16T10:36:04.326424+0000', 'last_clean': '2024-09-16T10:23:54.351323+0000', 'last_became_active': '2024-09-16T10:23:56.935037+0000', 'last_became_peered': '2024-09-16T10:23:56.935037+0000', 'last_unstale': '2024-09-16T10:36:04.326424+0000', 'last_undegraded': '2024-09-16T10:23:56.460939+0000', 'last_fullsized': '2024-09-16T10:23:56.366817+0000', 'mapping_epoch': 41, 'log_start': "44'10900", 'ondisk_log_start': "44'10900", 'created': 15, 'last_epoch_clean': 38, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "35'3824", 'last_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-16T10:22:12.997533+0000', 'last_clean_scrub_stamp': '2024-09-16T10:23:43.952628+0000', 'objects_scrubbed': 3752, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 2, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000', 'scrub_duration': 1.536242753, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1587, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 17162, 'num_write_kb': 26820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2818, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [0], 'avail_no_missing': ['0', '2'], 'object_location_counts': [{'shards': '0', 'objects': -1587}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-16T10:36:13.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-16T10:36:13.366 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:13.366 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1587/36 objects degraded (4408.333%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":862,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":66,"num_osds":3,"num_up_osds":3,"osd_up_since":1726482234,"num_in_osds":3,"osd_in_since":1726482228,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":23},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":24,"num_pools":2,"num_objects":18,"data_bytes":7747,"bytes_used":1404891136,"bytes_avail":288505401344,"bytes_total":289910292480,"degraded_objects":1587,"degraded_total":36,"degraded_ratio":44.083333333333343},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-16T10:31:52.731920+0000","services":{}},"progress_events":{}} 2024-09-16T10:36:13.379 INFO:tasks.rep_lost_unfound_delete.ceph_manager:no progress seen, keeping timeout for now 2024-09-16T10:36:13.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:13.664 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:13.665 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:13.677 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-16T10:36:12.789029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7747,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285872,"num_write_kb":526924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45479,"num_bytes_recovered":39694838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":47,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1371964,"kb_used_data":69592,"kb_used_omap":12,"kb_used_meta":1302323,"kb_avail":281743556,"statfs":{"total":289910292480,"available":288505401344,"internally_reserved":0,"allocated":71262208,"data_stored":70181565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1333578754},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-966,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-20417,"num_write_kb":-40831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4281,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001420"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28955,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.326474+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:36:09.326474+0000","last_peered":"2024-09-16T10:36:09.326474+0000","last_clean":"2024-09-16T10:36:09.326474+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:09.326474+0000","last_undegraded":"2024-09-16T10:36:09.326474+0000","last_fullsized":"2024-09-16T10:36:09.326474+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20645,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319359+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:36:09.319359+0000","last_peered":"2024-09-16T10:36:09.319359+0000","last_clean":"2024-09-16T10:36:09.319359+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:09.319359+0000","last_undegraded":"2024-09-16T10:36:09.319359+0000","last_fullsized":"2024-09-16T10:36:09.319359+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20945,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287740+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:35:52.287740+0000","last_peered":"2024-09-16T10:35:52.287740+0000","last_clean":"2024-09-16T10:35:52.287740+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:35:52.287740+0000","last_undegraded":"2024-09-16T10:35:52.287740+0000","last_fullsized":"2024-09-16T10:35:52.287740+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20544,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319239+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:36:09.319239+0000","last_peered":"2024-09-16T10:36:09.319239+0000","last_clean":"2024-09-16T10:36:09.319239+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:09.319239+0000","last_undegraded":"2024-09-16T10:36:09.319239+0000","last_fullsized":"2024-09-16T10:36:09.319239+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20687,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293649+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:35:53.293649+0000","last_peered":"2024-09-16T10:35:53.293649+0000","last_clean":"2024-09-16T10:35:53.293649+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:35:53.293649+0000","last_undegraded":"2024-09-16T10:35:53.293649+0000","last_fullsized":"2024-09-16T10:35:53.293649+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20724,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287071+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:35:52.287071+0000","last_peered":"2024-09-16T10:35:52.287071+0000","last_clean":"2024-09-16T10:35:52.287071+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:35:52.287071+0000","last_undegraded":"2024-09-16T10:35:52.287071+0000","last_fullsized":"2024-09-16T10:35:52.287071+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":26755,"reported_epoch":64,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-16T10:36:04.326424+0000","last_change":"2024-09-16T10:35:51.459459+0000","last_active":"2024-09-16T10:36:04.326424+0000","last_peered":"2024-09-16T10:36:04.326424+0000","last_clean":"2024-09-16T10:23:54.351323+0000","last_became_active":"2024-09-16T10:23:56.935037+0000","last_became_peered":"2024-09-16T10:23:56.935037+0000","last_unstale":"2024-09-16T10:36:04.326424+0000","last_undegraded":"2024-09-16T10:23:56.460939+0000","last_fullsized":"2024-09-16T10:23:56.366817+0000","mapping_epoch":41,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:12.416469+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2818,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0","objects":-1587}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20590,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293448+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:35:53.293448+0000","last_peered":"2024-09-16T10:35:53.293448+0000","last_clean":"2024-09-16T10:35:53.293448+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:35:53.293448+0000","last_undegraded":"2024-09-16T10:35:53.293448+0000","last_fullsized":"2024-09-16T10:35:53.293448+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":79,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.291333+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:35:53.291333+0000","last_peered":"2024-09-16T10:35:53.291333+0000","last_clean":"2024-09-16T10:35:53.291333+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:35:53.291333+0000","last_undegraded":"2024-09-16T10:35:53.291333+0000","last_fullsized":"2024-09-16T10:35:53.291333+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":28988,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287464+0000","last_change":"2024-09-16T10:26:01.958970+0000","last_active":"2024-09-16T10:35:52.287464+0000","last_peered":"2024-09-16T10:35:52.287464+0000","last_clean":"2024-09-16T10:35:52.287464+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:35:52.287464+0000","last_undegraded":"2024-09-16T10:35:52.287464+0000","last_fullsized":"2024-09-16T10:35:52.287464+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20459,"num_write_kb":40912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4201,"num_bytes_recovered":17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319055+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:36:09.319055+0000","last_peered":"2024-09-16T10:36:09.319055+0000","last_clean":"2024-09-16T10:36:09.319055+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:09.319055+0000","last_undegraded":"2024-09-16T10:36:09.319055+0000","last_fullsized":"2024-09-16T10:36:09.319055+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30026,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293601+0000","last_change":"2024-09-16T10:35:36.309942+0000","last_active":"2024-09-16T10:35:53.293601+0000","last_peered":"2024-09-16T10:35:53.293601+0000","last_clean":"2024-09-16T10:35:53.293601+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:35:53.293601+0000","last_undegraded":"2024-09-16T10:35:53.293601+0000","last_fullsized":"2024-09-16T10:35:53.293601+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21038,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4438,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":113,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.293650+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:35:53.293650+0000","last_peered":"2024-09-16T10:35:53.293650+0000","last_clean":"2024-09-16T10:35:53.293650+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:35:53.293650+0000","last_undegraded":"2024-09-16T10:35:53.293650+0000","last_fullsized":"2024-09-16T10:35:53.293650+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29147,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:11.186312+0000","last_change":"2024-09-16T10:36:11.186430+0000","last_active":"2024-09-16T10:36:11.186312+0000","last_peered":"2024-09-16T10:36:11.186312+0000","last_clean":"2024-09-16T10:36:11.186312+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:11.186312+0000","last_undegraded":"2024-09-16T10:36:11.186312+0000","last_fullsized":"2024-09-16T10:36:11.186312+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:11.186295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:11.186295+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:59.156525+0000","scrub_duration":0.010167803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":92,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287377+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:35:52.287377+0000","last_peered":"2024-09-16T10:35:52.287377+0000","last_clean":"2024-09-16T10:35:52.287377+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:35:52.287377+0000","last_undegraded":"2024-09-16T10:35:52.287377+0000","last_fullsized":"2024-09-16T10:35:52.287377+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29101,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.318922+0000","last_change":"2024-09-16T10:36:09.318922+0000","last_active":"2024-09-16T10:36:09.318922+0000","last_peered":"2024-09-16T10:36:09.318922+0000","last_clean":"2024-09-16T10:36:09.318922+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:09.318922+0000","last_undegraded":"2024-09-16T10:36:09.318922+0000","last_fullsized":"2024-09-16T10:36:09.318922+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20299,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-16T10:35:53.377851+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:35:53.377851+0000","last_peered":"2024-09-16T10:35:53.377851+0000","last_clean":"2024-09-16T10:35:53.377851+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:35:53.377851+0000","last_undegraded":"2024-09-16T10:35:53.377851+0000","last_fullsized":"2024-09-16T10:35:53.377851+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287642+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:35:52.287642+0000","last_peered":"2024-09-16T10:35:52.287642+0000","last_clean":"2024-09-16T10:35:52.287642+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:35:52.287642+0000","last_undegraded":"2024-09-16T10:35:52.287642+0000","last_fullsized":"2024-09-16T10:35:52.287642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21212,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287831+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:35:52.287831+0000","last_peered":"2024-09-16T10:35:52.287831+0000","last_clean":"2024-09-16T10:35:52.287831+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:35:52.287831+0000","last_undegraded":"2024-09-16T10:35:52.287831+0000","last_fullsized":"2024-09-16T10:35:52.287831+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319329+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:36:09.319329+0000","last_peered":"2024-09-16T10:36:09.319329+0000","last_clean":"2024-09-16T10:36:09.319329+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:09.319329+0000","last_undegraded":"2024-09-16T10:36:09.319329+0000","last_fullsized":"2024-09-16T10:36:09.319329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24520,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287863+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:35:52.287863+0000","last_peered":"2024-09-16T10:35:52.287863+0000","last_clean":"2024-09-16T10:35:52.287863+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:35:52.287863+0000","last_undegraded":"2024-09-16T10:35:52.287863+0000","last_fullsized":"2024-09-16T10:35:52.287863+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":78,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287821+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:35:52.287821+0000","last_peered":"2024-09-16T10:35:52.287821+0000","last_clean":"2024-09-16T10:35:52.287821+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:35:52.287821+0000","last_undegraded":"2024-09-16T10:35:52.287821+0000","last_fullsized":"2024-09-16T10:35:52.287821+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":87,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287444+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:35:52.287444+0000","last_peered":"2024-09-16T10:35:52.287444+0000","last_clean":"2024-09-16T10:35:52.287444+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:35:52.287444+0000","last_undegraded":"2024-09-16T10:35:52.287444+0000","last_fullsized":"2024-09-16T10:35:52.287444+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21157,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-16T10:35:52.287563+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:35:52.287563+0000","last_peered":"2024-09-16T10:35:52.287563+0000","last_clean":"2024-09-16T10:35:52.287563+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:35:52.287563+0000","last_undegraded":"2024-09-16T10:35:52.287563+0000","last_fullsized":"2024-09-16T10:35:52.287563+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":7728,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":285870,"num_write_kb":526922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45478,"num_bytes_recovered":39694819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":60321792,"data_stored":60314566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":31,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019045,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347172,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024668,"statfs":{"total":96636764160,"available":96281260032,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691988,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501400,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93870440,"statfs":{"total":96636764160,"available":96123330560,"internally_reserved":0,"allocated":540672,"data_stored":180338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":512880298},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.16500000000000001},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.20799999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.23799999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215271,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":523392,"kb_used_data":68528,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93848448,"statfs":{"total":96636764160,"available":96100810752,"internally_reserved":0,"allocated":70172672,"data_stored":69815952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.185},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.247}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.218}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60317696,"data_stored":60314083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:16.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:16.977 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:16.977 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:16.989 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T10:36:16.789911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5815,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244375,"num_write_kb":443944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38427,"num_bytes_recovered":22503926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1308088,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281807432,"statfs":{"total":289910292480,"available":288570810368,"internally_reserved":0,"allocated":1658880,"data_stored":556347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2898,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-61914,"num_write_kb":-123811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11333,"num_bytes_recovered":-17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001690"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28956,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.350311+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:36:09.350311+0000","last_peered":"2024-09-16T10:36:09.350311+0000","last_clean":"2024-09-16T10:36:09.350311+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:09.350311+0000","last_undegraded":"2024-09-16T10:36:09.350311+0000","last_fullsized":"2024-09-16T10:36:09.350311+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20645,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319359+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:36:09.319359+0000","last_peered":"2024-09-16T10:36:09.319359+0000","last_clean":"2024-09-16T10:36:09.319359+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:09.319359+0000","last_undegraded":"2024-09-16T10:36:09.319359+0000","last_fullsized":"2024-09-16T10:36:09.319359+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20947,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320879+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:36:09.320879+0000","last_peered":"2024-09-16T10:36:09.320879+0000","last_clean":"2024-09-16T10:36:09.320879+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:36:09.320879+0000","last_undegraded":"2024-09-16T10:36:09.320879+0000","last_fullsized":"2024-09-16T10:36:09.320879+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20544,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319239+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:36:09.319239+0000","last_peered":"2024-09-16T10:36:09.319239+0000","last_clean":"2024-09-16T10:36:09.319239+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:09.319239+0000","last_undegraded":"2024-09-16T10:36:09.319239+0000","last_fullsized":"2024-09-16T10:36:09.319239+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20688,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.328449+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:36:09.328449+0000","last_peered":"2024-09-16T10:36:09.328449+0000","last_clean":"2024-09-16T10:36:09.328449+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:36:09.328449+0000","last_undegraded":"2024-09-16T10:36:09.328449+0000","last_fullsized":"2024-09-16T10:36:09.328449+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20726,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320596+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:36:09.320596+0000","last_peered":"2024-09-16T10:36:09.320596+0000","last_clean":"2024-09-16T10:36:09.320596+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:36:09.320596+0000","last_undegraded":"2024-09-16T10:36:09.320596+0000","last_fullsized":"2024-09-16T10:36:09.320596+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:10.405243+0000","last_change":"2024-09-16T10:36:10.404790+0000","last_active":"2024-09-16T10:36:10.405243+0000","last_peered":"2024-09-16T10:36:10.405243+0000","last_clean":"2024-09-16T10:36:10.405243+0000","last_became_active":"2024-09-16T10:36:10.324803+0000","last_became_peered":"2024-09-16T10:36:10.324803+0000","last_unstale":"2024-09-16T10:36:10.405243+0000","last_undegraded":"2024-09-16T10:36:10.405243+0000","last_fullsized":"2024-09-16T10:36:10.405243+0000","mapping_epoch":65,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:18.250767+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20591,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.326828+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:36:09.326828+0000","last_peered":"2024-09-16T10:36:09.326828+0000","last_clean":"2024-09-16T10:36:09.326828+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:36:09.326828+0000","last_undegraded":"2024-09-16T10:36:09.326828+0000","last_fullsized":"2024-09-16T10:36:09.326828+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":81,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320376+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:36:09.320376+0000","last_peered":"2024-09-16T10:36:09.320376+0000","last_clean":"2024-09-16T10:36:09.320376+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:36:09.320376+0000","last_undegraded":"2024-09-16T10:36:09.320376+0000","last_fullsized":"2024-09-16T10:36:09.320376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":29002,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.817525+0000","last_change":"2024-09-16T10:36:12.817640+0000","last_active":"2024-09-16T10:36:12.817525+0000","last_peered":"2024-09-16T10:36:12.817525+0000","last_clean":"2024-09-16T10:36:12.817525+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:36:12.817525+0000","last_undegraded":"2024-09-16T10:36:12.817525+0000","last_fullsized":"2024-09-16T10:36:12.817525+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20461","last_scrub_stamp":"2024-09-16T10:36:12.817500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.817500+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:34.937908+0000","scrub_duration":0.010570972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319055+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:36:09.319055+0000","last_peered":"2024-09-16T10:36:09.319055+0000","last_clean":"2024-09-16T10:36:09.319055+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:09.319055+0000","last_undegraded":"2024-09-16T10:36:09.319055+0000","last_fullsized":"2024-09-16T10:36:09.319055+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30039,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.660270+0000","last_change":"2024-09-16T10:36:12.660360+0000","last_active":"2024-09-16T10:36:12.660270+0000","last_peered":"2024-09-16T10:36:12.660270+0000","last_clean":"2024-09-16T10:36:12.660270+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:36:12.660270+0000","last_undegraded":"2024-09-16T10:36:12.660270+0000","last_fullsized":"2024-09-16T10:36:12.660270+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'21040","last_scrub_stamp":"2024-09-16T10:36:12.660255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.660255+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:37.144016+0000","scrub_duration":0.0082777990000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":114,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.324719+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:36:09.324719+0000","last_peered":"2024-09-16T10:36:09.324719+0000","last_clean":"2024-09-16T10:36:09.324719+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:36:09.324719+0000","last_undegraded":"2024-09-16T10:36:09.324719+0000","last_fullsized":"2024-09-16T10:36:09.324719+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29147,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:11.186312+0000","last_change":"2024-09-16T10:36:11.186430+0000","last_active":"2024-09-16T10:36:11.186312+0000","last_peered":"2024-09-16T10:36:11.186312+0000","last_clean":"2024-09-16T10:36:11.186312+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:11.186312+0000","last_undegraded":"2024-09-16T10:36:11.186312+0000","last_fullsized":"2024-09-16T10:36:11.186312+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:11.186295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:11.186295+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:59.156525+0000","scrub_duration":0.010167803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":94,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320702+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:36:09.320702+0000","last_peered":"2024-09-16T10:36:09.320702+0000","last_clean":"2024-09-16T10:36:09.320702+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:36:09.320702+0000","last_undegraded":"2024-09-16T10:36:09.320702+0000","last_fullsized":"2024-09-16T10:36:09.320702+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29101,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.318922+0000","last_change":"2024-09-16T10:36:09.318922+0000","last_active":"2024-09-16T10:36:09.318922+0000","last_peered":"2024-09-16T10:36:09.318922+0000","last_clean":"2024-09-16T10:36:09.318922+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:09.318922+0000","last_undegraded":"2024-09-16T10:36:09.318922+0000","last_fullsized":"2024-09-16T10:36:09.318922+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321071+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:36:09.321071+0000","last_peered":"2024-09-16T10:36:09.321071+0000","last_clean":"2024-09-16T10:36:09.321071+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:36:09.321071+0000","last_undegraded":"2024-09-16T10:36:09.321071+0000","last_fullsized":"2024-09-16T10:36:09.321071+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320855+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:36:09.320855+0000","last_peered":"2024-09-16T10:36:09.320855+0000","last_clean":"2024-09-16T10:36:09.320855+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:36:09.320855+0000","last_undegraded":"2024-09-16T10:36:09.320855+0000","last_fullsized":"2024-09-16T10:36:09.320855+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21214,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320982+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:36:09.320982+0000","last_peered":"2024-09-16T10:36:09.320982+0000","last_clean":"2024-09-16T10:36:09.320982+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:36:09.320982+0000","last_undegraded":"2024-09-16T10:36:09.320982+0000","last_fullsized":"2024-09-16T10:36:09.320982+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319329+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:36:09.319329+0000","last_peered":"2024-09-16T10:36:09.319329+0000","last_clean":"2024-09-16T10:36:09.319329+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:09.319329+0000","last_undegraded":"2024-09-16T10:36:09.319329+0000","last_fullsized":"2024-09-16T10:36:09.319329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24522,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320895+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:36:09.320895+0000","last_peered":"2024-09-16T10:36:09.320895+0000","last_clean":"2024-09-16T10:36:09.320895+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:36:09.320895+0000","last_undegraded":"2024-09-16T10:36:09.320895+0000","last_fullsized":"2024-09-16T10:36:09.320895+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320943+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:36:09.320943+0000","last_peered":"2024-09-16T10:36:09.320943+0000","last_clean":"2024-09-16T10:36:09.320943+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:36:09.320943+0000","last_undegraded":"2024-09-16T10:36:09.320943+0000","last_fullsized":"2024-09-16T10:36:09.320943+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":89,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321058+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:36:09.321058+0000","last_peered":"2024-09-16T10:36:09.321058+0000","last_clean":"2024-09-16T10:36:09.321058+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:36:09.321058+0000","last_undegraded":"2024-09-16T10:36:09.321058+0000","last_fullsized":"2024-09-16T10:36:09.321058+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21159,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320799+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:36:09.320799+0000","last_peered":"2024-09-16T10:36:09.320799+0000","last_clean":"2024-09-16T10:36:09.320799+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:36:09.320799+0000","last_undegraded":"2024-09-16T10:36:09.320799+0000","last_fullsized":"2024-09-16T10:36:09.320799+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5796,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244373,"num_write_kb":443942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38426,"num_bytes_recovered":22503907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019045,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347172,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024668,"statfs":{"total":96636764160,"available":96281260032,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505516,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866324,"statfs":{"total":96636764160,"available":96119115776,"internally_reserved":0,"allocated":557056,"data_stored":185777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.218},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27900000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455400,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916440,"statfs":{"total":96636764160,"available":96170434560,"internally_reserved":0,"allocated":552960,"data_stored":185295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.433}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:16.991 INFO:tasks.rep_lost_unfound_delete.ceph_manager:clean! 2024-09-16T10:36:16.991 INFO:tasks.rep_lost_unfound_delete.ceph_manager:waiting for clean 2024-09-16T10:36:16.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:17.270 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:17.270 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:17.282 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T10:36:16.789911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5815,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244375,"num_write_kb":443944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38427,"num_bytes_recovered":22503926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1308088,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281807432,"statfs":{"total":289910292480,"available":288570810368,"internally_reserved":0,"allocated":1658880,"data_stored":556347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2898,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-61914,"num_write_kb":-123811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11333,"num_bytes_recovered":-17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001690"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28956,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.350311+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:36:09.350311+0000","last_peered":"2024-09-16T10:36:09.350311+0000","last_clean":"2024-09-16T10:36:09.350311+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:09.350311+0000","last_undegraded":"2024-09-16T10:36:09.350311+0000","last_fullsized":"2024-09-16T10:36:09.350311+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20645,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319359+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:36:09.319359+0000","last_peered":"2024-09-16T10:36:09.319359+0000","last_clean":"2024-09-16T10:36:09.319359+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:09.319359+0000","last_undegraded":"2024-09-16T10:36:09.319359+0000","last_fullsized":"2024-09-16T10:36:09.319359+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20947,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320879+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:36:09.320879+0000","last_peered":"2024-09-16T10:36:09.320879+0000","last_clean":"2024-09-16T10:36:09.320879+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:36:09.320879+0000","last_undegraded":"2024-09-16T10:36:09.320879+0000","last_fullsized":"2024-09-16T10:36:09.320879+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20544,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319239+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:36:09.319239+0000","last_peered":"2024-09-16T10:36:09.319239+0000","last_clean":"2024-09-16T10:36:09.319239+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:09.319239+0000","last_undegraded":"2024-09-16T10:36:09.319239+0000","last_fullsized":"2024-09-16T10:36:09.319239+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20688,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.328449+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:36:09.328449+0000","last_peered":"2024-09-16T10:36:09.328449+0000","last_clean":"2024-09-16T10:36:09.328449+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:36:09.328449+0000","last_undegraded":"2024-09-16T10:36:09.328449+0000","last_fullsized":"2024-09-16T10:36:09.328449+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20726,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320596+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:36:09.320596+0000","last_peered":"2024-09-16T10:36:09.320596+0000","last_clean":"2024-09-16T10:36:09.320596+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:36:09.320596+0000","last_undegraded":"2024-09-16T10:36:09.320596+0000","last_fullsized":"2024-09-16T10:36:09.320596+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:10.405243+0000","last_change":"2024-09-16T10:36:10.404790+0000","last_active":"2024-09-16T10:36:10.405243+0000","last_peered":"2024-09-16T10:36:10.405243+0000","last_clean":"2024-09-16T10:36:10.405243+0000","last_became_active":"2024-09-16T10:36:10.324803+0000","last_became_peered":"2024-09-16T10:36:10.324803+0000","last_unstale":"2024-09-16T10:36:10.405243+0000","last_undegraded":"2024-09-16T10:36:10.405243+0000","last_fullsized":"2024-09-16T10:36:10.405243+0000","mapping_epoch":65,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:18.250767+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20591,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.326828+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:36:09.326828+0000","last_peered":"2024-09-16T10:36:09.326828+0000","last_clean":"2024-09-16T10:36:09.326828+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:36:09.326828+0000","last_undegraded":"2024-09-16T10:36:09.326828+0000","last_fullsized":"2024-09-16T10:36:09.326828+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":81,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320376+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:36:09.320376+0000","last_peered":"2024-09-16T10:36:09.320376+0000","last_clean":"2024-09-16T10:36:09.320376+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:36:09.320376+0000","last_undegraded":"2024-09-16T10:36:09.320376+0000","last_fullsized":"2024-09-16T10:36:09.320376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":29002,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.817525+0000","last_change":"2024-09-16T10:36:12.817640+0000","last_active":"2024-09-16T10:36:12.817525+0000","last_peered":"2024-09-16T10:36:12.817525+0000","last_clean":"2024-09-16T10:36:12.817525+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:36:12.817525+0000","last_undegraded":"2024-09-16T10:36:12.817525+0000","last_fullsized":"2024-09-16T10:36:12.817525+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20461","last_scrub_stamp":"2024-09-16T10:36:12.817500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.817500+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:34.937908+0000","scrub_duration":0.010570972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319055+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:36:09.319055+0000","last_peered":"2024-09-16T10:36:09.319055+0000","last_clean":"2024-09-16T10:36:09.319055+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:09.319055+0000","last_undegraded":"2024-09-16T10:36:09.319055+0000","last_fullsized":"2024-09-16T10:36:09.319055+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30039,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.660270+0000","last_change":"2024-09-16T10:36:12.660360+0000","last_active":"2024-09-16T10:36:12.660270+0000","last_peered":"2024-09-16T10:36:12.660270+0000","last_clean":"2024-09-16T10:36:12.660270+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:36:12.660270+0000","last_undegraded":"2024-09-16T10:36:12.660270+0000","last_fullsized":"2024-09-16T10:36:12.660270+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'21040","last_scrub_stamp":"2024-09-16T10:36:12.660255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.660255+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:37.144016+0000","scrub_duration":0.0082777990000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":114,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.324719+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:36:09.324719+0000","last_peered":"2024-09-16T10:36:09.324719+0000","last_clean":"2024-09-16T10:36:09.324719+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:36:09.324719+0000","last_undegraded":"2024-09-16T10:36:09.324719+0000","last_fullsized":"2024-09-16T10:36:09.324719+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29147,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:11.186312+0000","last_change":"2024-09-16T10:36:11.186430+0000","last_active":"2024-09-16T10:36:11.186312+0000","last_peered":"2024-09-16T10:36:11.186312+0000","last_clean":"2024-09-16T10:36:11.186312+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:11.186312+0000","last_undegraded":"2024-09-16T10:36:11.186312+0000","last_fullsized":"2024-09-16T10:36:11.186312+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:11.186295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:11.186295+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:59.156525+0000","scrub_duration":0.010167803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":94,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320702+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:36:09.320702+0000","last_peered":"2024-09-16T10:36:09.320702+0000","last_clean":"2024-09-16T10:36:09.320702+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:36:09.320702+0000","last_undegraded":"2024-09-16T10:36:09.320702+0000","last_fullsized":"2024-09-16T10:36:09.320702+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29101,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.318922+0000","last_change":"2024-09-16T10:36:09.318922+0000","last_active":"2024-09-16T10:36:09.318922+0000","last_peered":"2024-09-16T10:36:09.318922+0000","last_clean":"2024-09-16T10:36:09.318922+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:09.318922+0000","last_undegraded":"2024-09-16T10:36:09.318922+0000","last_fullsized":"2024-09-16T10:36:09.318922+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321071+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:36:09.321071+0000","last_peered":"2024-09-16T10:36:09.321071+0000","last_clean":"2024-09-16T10:36:09.321071+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:36:09.321071+0000","last_undegraded":"2024-09-16T10:36:09.321071+0000","last_fullsized":"2024-09-16T10:36:09.321071+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320855+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:36:09.320855+0000","last_peered":"2024-09-16T10:36:09.320855+0000","last_clean":"2024-09-16T10:36:09.320855+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:36:09.320855+0000","last_undegraded":"2024-09-16T10:36:09.320855+0000","last_fullsized":"2024-09-16T10:36:09.320855+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21214,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320982+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:36:09.320982+0000","last_peered":"2024-09-16T10:36:09.320982+0000","last_clean":"2024-09-16T10:36:09.320982+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:36:09.320982+0000","last_undegraded":"2024-09-16T10:36:09.320982+0000","last_fullsized":"2024-09-16T10:36:09.320982+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319329+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:36:09.319329+0000","last_peered":"2024-09-16T10:36:09.319329+0000","last_clean":"2024-09-16T10:36:09.319329+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:09.319329+0000","last_undegraded":"2024-09-16T10:36:09.319329+0000","last_fullsized":"2024-09-16T10:36:09.319329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24522,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320895+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:36:09.320895+0000","last_peered":"2024-09-16T10:36:09.320895+0000","last_clean":"2024-09-16T10:36:09.320895+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:36:09.320895+0000","last_undegraded":"2024-09-16T10:36:09.320895+0000","last_fullsized":"2024-09-16T10:36:09.320895+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320943+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:36:09.320943+0000","last_peered":"2024-09-16T10:36:09.320943+0000","last_clean":"2024-09-16T10:36:09.320943+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:36:09.320943+0000","last_undegraded":"2024-09-16T10:36:09.320943+0000","last_fullsized":"2024-09-16T10:36:09.320943+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":89,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321058+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:36:09.321058+0000","last_peered":"2024-09-16T10:36:09.321058+0000","last_clean":"2024-09-16T10:36:09.321058+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:36:09.321058+0000","last_undegraded":"2024-09-16T10:36:09.321058+0000","last_fullsized":"2024-09-16T10:36:09.321058+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21159,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320799+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:36:09.320799+0000","last_peered":"2024-09-16T10:36:09.320799+0000","last_clean":"2024-09-16T10:36:09.320799+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:36:09.320799+0000","last_undegraded":"2024-09-16T10:36:09.320799+0000","last_fullsized":"2024-09-16T10:36:09.320799+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5796,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244373,"num_write_kb":443942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38426,"num_bytes_recovered":22503907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019045,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347172,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024668,"statfs":{"total":96636764160,"available":96281260032,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505516,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866324,"statfs":{"total":96636764160,"available":96119115776,"internally_reserved":0,"allocated":557056,"data_stored":185777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.218},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27900000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455400,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916440,"statfs":{"total":96636764160,"available":96170434560,"internally_reserved":0,"allocated":552960,"data_stored":185295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.433}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:17.283 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:17.566 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:17.566 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:17.578 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T10:36:16.789911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5815,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244375,"num_write_kb":443944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38427,"num_bytes_recovered":22503926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1308088,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281807432,"statfs":{"total":289910292480,"available":288570810368,"internally_reserved":0,"allocated":1658880,"data_stored":556347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2898,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-61914,"num_write_kb":-123811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11333,"num_bytes_recovered":-17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001690"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28956,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.350311+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:36:09.350311+0000","last_peered":"2024-09-16T10:36:09.350311+0000","last_clean":"2024-09-16T10:36:09.350311+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:09.350311+0000","last_undegraded":"2024-09-16T10:36:09.350311+0000","last_fullsized":"2024-09-16T10:36:09.350311+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20645,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319359+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:36:09.319359+0000","last_peered":"2024-09-16T10:36:09.319359+0000","last_clean":"2024-09-16T10:36:09.319359+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:09.319359+0000","last_undegraded":"2024-09-16T10:36:09.319359+0000","last_fullsized":"2024-09-16T10:36:09.319359+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20947,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320879+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:36:09.320879+0000","last_peered":"2024-09-16T10:36:09.320879+0000","last_clean":"2024-09-16T10:36:09.320879+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:36:09.320879+0000","last_undegraded":"2024-09-16T10:36:09.320879+0000","last_fullsized":"2024-09-16T10:36:09.320879+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20544,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319239+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:36:09.319239+0000","last_peered":"2024-09-16T10:36:09.319239+0000","last_clean":"2024-09-16T10:36:09.319239+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:09.319239+0000","last_undegraded":"2024-09-16T10:36:09.319239+0000","last_fullsized":"2024-09-16T10:36:09.319239+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20688,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.328449+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:36:09.328449+0000","last_peered":"2024-09-16T10:36:09.328449+0000","last_clean":"2024-09-16T10:36:09.328449+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:36:09.328449+0000","last_undegraded":"2024-09-16T10:36:09.328449+0000","last_fullsized":"2024-09-16T10:36:09.328449+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20726,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320596+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:36:09.320596+0000","last_peered":"2024-09-16T10:36:09.320596+0000","last_clean":"2024-09-16T10:36:09.320596+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:36:09.320596+0000","last_undegraded":"2024-09-16T10:36:09.320596+0000","last_fullsized":"2024-09-16T10:36:09.320596+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:10.405243+0000","last_change":"2024-09-16T10:36:10.404790+0000","last_active":"2024-09-16T10:36:10.405243+0000","last_peered":"2024-09-16T10:36:10.405243+0000","last_clean":"2024-09-16T10:36:10.405243+0000","last_became_active":"2024-09-16T10:36:10.324803+0000","last_became_peered":"2024-09-16T10:36:10.324803+0000","last_unstale":"2024-09-16T10:36:10.405243+0000","last_undegraded":"2024-09-16T10:36:10.405243+0000","last_fullsized":"2024-09-16T10:36:10.405243+0000","mapping_epoch":65,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:18.250767+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20591,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.326828+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:36:09.326828+0000","last_peered":"2024-09-16T10:36:09.326828+0000","last_clean":"2024-09-16T10:36:09.326828+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:36:09.326828+0000","last_undegraded":"2024-09-16T10:36:09.326828+0000","last_fullsized":"2024-09-16T10:36:09.326828+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":81,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320376+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:36:09.320376+0000","last_peered":"2024-09-16T10:36:09.320376+0000","last_clean":"2024-09-16T10:36:09.320376+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:36:09.320376+0000","last_undegraded":"2024-09-16T10:36:09.320376+0000","last_fullsized":"2024-09-16T10:36:09.320376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":29002,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.817525+0000","last_change":"2024-09-16T10:36:12.817640+0000","last_active":"2024-09-16T10:36:12.817525+0000","last_peered":"2024-09-16T10:36:12.817525+0000","last_clean":"2024-09-16T10:36:12.817525+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:36:12.817525+0000","last_undegraded":"2024-09-16T10:36:12.817525+0000","last_fullsized":"2024-09-16T10:36:12.817525+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20461","last_scrub_stamp":"2024-09-16T10:36:12.817500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.817500+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:34.937908+0000","scrub_duration":0.010570972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319055+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:36:09.319055+0000","last_peered":"2024-09-16T10:36:09.319055+0000","last_clean":"2024-09-16T10:36:09.319055+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:09.319055+0000","last_undegraded":"2024-09-16T10:36:09.319055+0000","last_fullsized":"2024-09-16T10:36:09.319055+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30039,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.660270+0000","last_change":"2024-09-16T10:36:12.660360+0000","last_active":"2024-09-16T10:36:12.660270+0000","last_peered":"2024-09-16T10:36:12.660270+0000","last_clean":"2024-09-16T10:36:12.660270+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:36:12.660270+0000","last_undegraded":"2024-09-16T10:36:12.660270+0000","last_fullsized":"2024-09-16T10:36:12.660270+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'21040","last_scrub_stamp":"2024-09-16T10:36:12.660255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.660255+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:37.144016+0000","scrub_duration":0.0082777990000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":114,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.324719+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:36:09.324719+0000","last_peered":"2024-09-16T10:36:09.324719+0000","last_clean":"2024-09-16T10:36:09.324719+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:36:09.324719+0000","last_undegraded":"2024-09-16T10:36:09.324719+0000","last_fullsized":"2024-09-16T10:36:09.324719+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29147,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:11.186312+0000","last_change":"2024-09-16T10:36:11.186430+0000","last_active":"2024-09-16T10:36:11.186312+0000","last_peered":"2024-09-16T10:36:11.186312+0000","last_clean":"2024-09-16T10:36:11.186312+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:11.186312+0000","last_undegraded":"2024-09-16T10:36:11.186312+0000","last_fullsized":"2024-09-16T10:36:11.186312+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:11.186295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:11.186295+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:59.156525+0000","scrub_duration":0.010167803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":94,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320702+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:36:09.320702+0000","last_peered":"2024-09-16T10:36:09.320702+0000","last_clean":"2024-09-16T10:36:09.320702+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:36:09.320702+0000","last_undegraded":"2024-09-16T10:36:09.320702+0000","last_fullsized":"2024-09-16T10:36:09.320702+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29101,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.318922+0000","last_change":"2024-09-16T10:36:09.318922+0000","last_active":"2024-09-16T10:36:09.318922+0000","last_peered":"2024-09-16T10:36:09.318922+0000","last_clean":"2024-09-16T10:36:09.318922+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:09.318922+0000","last_undegraded":"2024-09-16T10:36:09.318922+0000","last_fullsized":"2024-09-16T10:36:09.318922+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321071+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:36:09.321071+0000","last_peered":"2024-09-16T10:36:09.321071+0000","last_clean":"2024-09-16T10:36:09.321071+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:36:09.321071+0000","last_undegraded":"2024-09-16T10:36:09.321071+0000","last_fullsized":"2024-09-16T10:36:09.321071+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320855+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:36:09.320855+0000","last_peered":"2024-09-16T10:36:09.320855+0000","last_clean":"2024-09-16T10:36:09.320855+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:36:09.320855+0000","last_undegraded":"2024-09-16T10:36:09.320855+0000","last_fullsized":"2024-09-16T10:36:09.320855+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21214,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320982+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:36:09.320982+0000","last_peered":"2024-09-16T10:36:09.320982+0000","last_clean":"2024-09-16T10:36:09.320982+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:36:09.320982+0000","last_undegraded":"2024-09-16T10:36:09.320982+0000","last_fullsized":"2024-09-16T10:36:09.320982+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319329+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:36:09.319329+0000","last_peered":"2024-09-16T10:36:09.319329+0000","last_clean":"2024-09-16T10:36:09.319329+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:09.319329+0000","last_undegraded":"2024-09-16T10:36:09.319329+0000","last_fullsized":"2024-09-16T10:36:09.319329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24522,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320895+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:36:09.320895+0000","last_peered":"2024-09-16T10:36:09.320895+0000","last_clean":"2024-09-16T10:36:09.320895+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:36:09.320895+0000","last_undegraded":"2024-09-16T10:36:09.320895+0000","last_fullsized":"2024-09-16T10:36:09.320895+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320943+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:36:09.320943+0000","last_peered":"2024-09-16T10:36:09.320943+0000","last_clean":"2024-09-16T10:36:09.320943+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:36:09.320943+0000","last_undegraded":"2024-09-16T10:36:09.320943+0000","last_fullsized":"2024-09-16T10:36:09.320943+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":89,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321058+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:36:09.321058+0000","last_peered":"2024-09-16T10:36:09.321058+0000","last_clean":"2024-09-16T10:36:09.321058+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:36:09.321058+0000","last_undegraded":"2024-09-16T10:36:09.321058+0000","last_fullsized":"2024-09-16T10:36:09.321058+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21159,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320799+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:36:09.320799+0000","last_peered":"2024-09-16T10:36:09.320799+0000","last_clean":"2024-09-16T10:36:09.320799+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:36:09.320799+0000","last_undegraded":"2024-09-16T10:36:09.320799+0000","last_fullsized":"2024-09-16T10:36:09.320799+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5796,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244373,"num_write_kb":443942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38426,"num_bytes_recovered":22503907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019045,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347172,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024668,"statfs":{"total":96636764160,"available":96281260032,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505516,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866324,"statfs":{"total":96636764160,"available":96119115776,"internally_reserved":0,"allocated":557056,"data_stored":185777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.218},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27900000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455400,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916440,"statfs":{"total":96636764160,"available":96170434560,"internally_reserved":0,"allocated":552960,"data_stored":185295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.433}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:17.579 INFO:tasks.rep_lost_unfound_delete.ceph_manager:clean! 2024-09-16T10:36:17.579 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T10:36:17.589 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T10:36:17.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:17.873 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:17.873 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:17.885 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T10:36:16.789911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5815,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244375,"num_write_kb":443944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38427,"num_bytes_recovered":22503926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1308088,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281807432,"statfs":{"total":289910292480,"available":288570810368,"internally_reserved":0,"allocated":1658880,"data_stored":556347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2898,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-61914,"num_write_kb":-123811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11333,"num_bytes_recovered":-17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001690"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28956,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.350311+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:36:09.350311+0000","last_peered":"2024-09-16T10:36:09.350311+0000","last_clean":"2024-09-16T10:36:09.350311+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:09.350311+0000","last_undegraded":"2024-09-16T10:36:09.350311+0000","last_fullsized":"2024-09-16T10:36:09.350311+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20645,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319359+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:36:09.319359+0000","last_peered":"2024-09-16T10:36:09.319359+0000","last_clean":"2024-09-16T10:36:09.319359+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:09.319359+0000","last_undegraded":"2024-09-16T10:36:09.319359+0000","last_fullsized":"2024-09-16T10:36:09.319359+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20947,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320879+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:36:09.320879+0000","last_peered":"2024-09-16T10:36:09.320879+0000","last_clean":"2024-09-16T10:36:09.320879+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:36:09.320879+0000","last_undegraded":"2024-09-16T10:36:09.320879+0000","last_fullsized":"2024-09-16T10:36:09.320879+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20544,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319239+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:36:09.319239+0000","last_peered":"2024-09-16T10:36:09.319239+0000","last_clean":"2024-09-16T10:36:09.319239+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:09.319239+0000","last_undegraded":"2024-09-16T10:36:09.319239+0000","last_fullsized":"2024-09-16T10:36:09.319239+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20688,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.328449+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:36:09.328449+0000","last_peered":"2024-09-16T10:36:09.328449+0000","last_clean":"2024-09-16T10:36:09.328449+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:36:09.328449+0000","last_undegraded":"2024-09-16T10:36:09.328449+0000","last_fullsized":"2024-09-16T10:36:09.328449+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20726,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320596+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:36:09.320596+0000","last_peered":"2024-09-16T10:36:09.320596+0000","last_clean":"2024-09-16T10:36:09.320596+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:36:09.320596+0000","last_undegraded":"2024-09-16T10:36:09.320596+0000","last_fullsized":"2024-09-16T10:36:09.320596+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:10.405243+0000","last_change":"2024-09-16T10:36:10.404790+0000","last_active":"2024-09-16T10:36:10.405243+0000","last_peered":"2024-09-16T10:36:10.405243+0000","last_clean":"2024-09-16T10:36:10.405243+0000","last_became_active":"2024-09-16T10:36:10.324803+0000","last_became_peered":"2024-09-16T10:36:10.324803+0000","last_unstale":"2024-09-16T10:36:10.405243+0000","last_undegraded":"2024-09-16T10:36:10.405243+0000","last_fullsized":"2024-09-16T10:36:10.405243+0000","mapping_epoch":65,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:18.250767+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20591,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.326828+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:36:09.326828+0000","last_peered":"2024-09-16T10:36:09.326828+0000","last_clean":"2024-09-16T10:36:09.326828+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:36:09.326828+0000","last_undegraded":"2024-09-16T10:36:09.326828+0000","last_fullsized":"2024-09-16T10:36:09.326828+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":81,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320376+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:36:09.320376+0000","last_peered":"2024-09-16T10:36:09.320376+0000","last_clean":"2024-09-16T10:36:09.320376+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:36:09.320376+0000","last_undegraded":"2024-09-16T10:36:09.320376+0000","last_fullsized":"2024-09-16T10:36:09.320376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":29002,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.817525+0000","last_change":"2024-09-16T10:36:12.817640+0000","last_active":"2024-09-16T10:36:12.817525+0000","last_peered":"2024-09-16T10:36:12.817525+0000","last_clean":"2024-09-16T10:36:12.817525+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:36:12.817525+0000","last_undegraded":"2024-09-16T10:36:12.817525+0000","last_fullsized":"2024-09-16T10:36:12.817525+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20461","last_scrub_stamp":"2024-09-16T10:36:12.817500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.817500+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:34.937908+0000","scrub_duration":0.010570972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319055+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:36:09.319055+0000","last_peered":"2024-09-16T10:36:09.319055+0000","last_clean":"2024-09-16T10:36:09.319055+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:09.319055+0000","last_undegraded":"2024-09-16T10:36:09.319055+0000","last_fullsized":"2024-09-16T10:36:09.319055+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30039,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.660270+0000","last_change":"2024-09-16T10:36:12.660360+0000","last_active":"2024-09-16T10:36:12.660270+0000","last_peered":"2024-09-16T10:36:12.660270+0000","last_clean":"2024-09-16T10:36:12.660270+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:36:12.660270+0000","last_undegraded":"2024-09-16T10:36:12.660270+0000","last_fullsized":"2024-09-16T10:36:12.660270+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'21040","last_scrub_stamp":"2024-09-16T10:36:12.660255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.660255+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:37.144016+0000","scrub_duration":0.0082777990000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":114,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.324719+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:36:09.324719+0000","last_peered":"2024-09-16T10:36:09.324719+0000","last_clean":"2024-09-16T10:36:09.324719+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:36:09.324719+0000","last_undegraded":"2024-09-16T10:36:09.324719+0000","last_fullsized":"2024-09-16T10:36:09.324719+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29147,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:11.186312+0000","last_change":"2024-09-16T10:36:11.186430+0000","last_active":"2024-09-16T10:36:11.186312+0000","last_peered":"2024-09-16T10:36:11.186312+0000","last_clean":"2024-09-16T10:36:11.186312+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:11.186312+0000","last_undegraded":"2024-09-16T10:36:11.186312+0000","last_fullsized":"2024-09-16T10:36:11.186312+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:11.186295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:11.186295+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:59.156525+0000","scrub_duration":0.010167803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":94,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320702+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:36:09.320702+0000","last_peered":"2024-09-16T10:36:09.320702+0000","last_clean":"2024-09-16T10:36:09.320702+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:36:09.320702+0000","last_undegraded":"2024-09-16T10:36:09.320702+0000","last_fullsized":"2024-09-16T10:36:09.320702+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29101,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.318922+0000","last_change":"2024-09-16T10:36:09.318922+0000","last_active":"2024-09-16T10:36:09.318922+0000","last_peered":"2024-09-16T10:36:09.318922+0000","last_clean":"2024-09-16T10:36:09.318922+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:09.318922+0000","last_undegraded":"2024-09-16T10:36:09.318922+0000","last_fullsized":"2024-09-16T10:36:09.318922+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321071+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:36:09.321071+0000","last_peered":"2024-09-16T10:36:09.321071+0000","last_clean":"2024-09-16T10:36:09.321071+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:36:09.321071+0000","last_undegraded":"2024-09-16T10:36:09.321071+0000","last_fullsized":"2024-09-16T10:36:09.321071+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320855+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:36:09.320855+0000","last_peered":"2024-09-16T10:36:09.320855+0000","last_clean":"2024-09-16T10:36:09.320855+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:36:09.320855+0000","last_undegraded":"2024-09-16T10:36:09.320855+0000","last_fullsized":"2024-09-16T10:36:09.320855+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21214,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320982+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:36:09.320982+0000","last_peered":"2024-09-16T10:36:09.320982+0000","last_clean":"2024-09-16T10:36:09.320982+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:36:09.320982+0000","last_undegraded":"2024-09-16T10:36:09.320982+0000","last_fullsized":"2024-09-16T10:36:09.320982+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319329+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:36:09.319329+0000","last_peered":"2024-09-16T10:36:09.319329+0000","last_clean":"2024-09-16T10:36:09.319329+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:09.319329+0000","last_undegraded":"2024-09-16T10:36:09.319329+0000","last_fullsized":"2024-09-16T10:36:09.319329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24522,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320895+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:36:09.320895+0000","last_peered":"2024-09-16T10:36:09.320895+0000","last_clean":"2024-09-16T10:36:09.320895+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:36:09.320895+0000","last_undegraded":"2024-09-16T10:36:09.320895+0000","last_fullsized":"2024-09-16T10:36:09.320895+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320943+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:36:09.320943+0000","last_peered":"2024-09-16T10:36:09.320943+0000","last_clean":"2024-09-16T10:36:09.320943+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:36:09.320943+0000","last_undegraded":"2024-09-16T10:36:09.320943+0000","last_fullsized":"2024-09-16T10:36:09.320943+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":89,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321058+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:36:09.321058+0000","last_peered":"2024-09-16T10:36:09.321058+0000","last_clean":"2024-09-16T10:36:09.321058+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:36:09.321058+0000","last_undegraded":"2024-09-16T10:36:09.321058+0000","last_fullsized":"2024-09-16T10:36:09.321058+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21159,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320799+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:36:09.320799+0000","last_peered":"2024-09-16T10:36:09.320799+0000","last_clean":"2024-09-16T10:36:09.320799+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:36:09.320799+0000","last_undegraded":"2024-09-16T10:36:09.320799+0000","last_fullsized":"2024-09-16T10:36:09.320799+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5796,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244373,"num_write_kb":443942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38426,"num_bytes_recovered":22503907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019045,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347172,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024668,"statfs":{"total":96636764160,"available":96281260032,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505516,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866324,"statfs":{"total":96636764160,"available":96119115776,"internally_reserved":0,"allocated":557056,"data_stored":185777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.218},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27900000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455400,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916440,"statfs":{"total":96636764160,"available":96170434560,"internally_reserved":0,"allocated":552960,"data_stored":185295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.433}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:17.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:18.172 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:18.173 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:18.181 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T10:36:16.789911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5815,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244375,"num_write_kb":443944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38427,"num_bytes_recovered":22503926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1308088,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281807432,"statfs":{"total":289910292480,"available":288570810368,"internally_reserved":0,"allocated":1658880,"data_stored":556347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2898,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-61914,"num_write_kb":-123811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11333,"num_bytes_recovered":-17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001690"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28956,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.350311+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:36:09.350311+0000","last_peered":"2024-09-16T10:36:09.350311+0000","last_clean":"2024-09-16T10:36:09.350311+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:09.350311+0000","last_undegraded":"2024-09-16T10:36:09.350311+0000","last_fullsized":"2024-09-16T10:36:09.350311+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20645,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319359+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:36:09.319359+0000","last_peered":"2024-09-16T10:36:09.319359+0000","last_clean":"2024-09-16T10:36:09.319359+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:09.319359+0000","last_undegraded":"2024-09-16T10:36:09.319359+0000","last_fullsized":"2024-09-16T10:36:09.319359+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20947,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320879+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:36:09.320879+0000","last_peered":"2024-09-16T10:36:09.320879+0000","last_clean":"2024-09-16T10:36:09.320879+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:36:09.320879+0000","last_undegraded":"2024-09-16T10:36:09.320879+0000","last_fullsized":"2024-09-16T10:36:09.320879+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20544,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319239+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:36:09.319239+0000","last_peered":"2024-09-16T10:36:09.319239+0000","last_clean":"2024-09-16T10:36:09.319239+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:09.319239+0000","last_undegraded":"2024-09-16T10:36:09.319239+0000","last_fullsized":"2024-09-16T10:36:09.319239+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20688,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.328449+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:36:09.328449+0000","last_peered":"2024-09-16T10:36:09.328449+0000","last_clean":"2024-09-16T10:36:09.328449+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:36:09.328449+0000","last_undegraded":"2024-09-16T10:36:09.328449+0000","last_fullsized":"2024-09-16T10:36:09.328449+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20726,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320596+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:36:09.320596+0000","last_peered":"2024-09-16T10:36:09.320596+0000","last_clean":"2024-09-16T10:36:09.320596+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:36:09.320596+0000","last_undegraded":"2024-09-16T10:36:09.320596+0000","last_fullsized":"2024-09-16T10:36:09.320596+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:10.405243+0000","last_change":"2024-09-16T10:36:10.404790+0000","last_active":"2024-09-16T10:36:10.405243+0000","last_peered":"2024-09-16T10:36:10.405243+0000","last_clean":"2024-09-16T10:36:10.405243+0000","last_became_active":"2024-09-16T10:36:10.324803+0000","last_became_peered":"2024-09-16T10:36:10.324803+0000","last_unstale":"2024-09-16T10:36:10.405243+0000","last_undegraded":"2024-09-16T10:36:10.405243+0000","last_fullsized":"2024-09-16T10:36:10.405243+0000","mapping_epoch":65,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:18.250767+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20591,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.326828+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:36:09.326828+0000","last_peered":"2024-09-16T10:36:09.326828+0000","last_clean":"2024-09-16T10:36:09.326828+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:36:09.326828+0000","last_undegraded":"2024-09-16T10:36:09.326828+0000","last_fullsized":"2024-09-16T10:36:09.326828+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":81,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320376+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:36:09.320376+0000","last_peered":"2024-09-16T10:36:09.320376+0000","last_clean":"2024-09-16T10:36:09.320376+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:36:09.320376+0000","last_undegraded":"2024-09-16T10:36:09.320376+0000","last_fullsized":"2024-09-16T10:36:09.320376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":29002,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.817525+0000","last_change":"2024-09-16T10:36:12.817640+0000","last_active":"2024-09-16T10:36:12.817525+0000","last_peered":"2024-09-16T10:36:12.817525+0000","last_clean":"2024-09-16T10:36:12.817525+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:36:12.817525+0000","last_undegraded":"2024-09-16T10:36:12.817525+0000","last_fullsized":"2024-09-16T10:36:12.817525+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20461","last_scrub_stamp":"2024-09-16T10:36:12.817500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.817500+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:34.937908+0000","scrub_duration":0.010570972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319055+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:36:09.319055+0000","last_peered":"2024-09-16T10:36:09.319055+0000","last_clean":"2024-09-16T10:36:09.319055+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:09.319055+0000","last_undegraded":"2024-09-16T10:36:09.319055+0000","last_fullsized":"2024-09-16T10:36:09.319055+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30039,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.660270+0000","last_change":"2024-09-16T10:36:12.660360+0000","last_active":"2024-09-16T10:36:12.660270+0000","last_peered":"2024-09-16T10:36:12.660270+0000","last_clean":"2024-09-16T10:36:12.660270+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:36:12.660270+0000","last_undegraded":"2024-09-16T10:36:12.660270+0000","last_fullsized":"2024-09-16T10:36:12.660270+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'21040","last_scrub_stamp":"2024-09-16T10:36:12.660255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.660255+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:37.144016+0000","scrub_duration":0.0082777990000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":114,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.324719+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:36:09.324719+0000","last_peered":"2024-09-16T10:36:09.324719+0000","last_clean":"2024-09-16T10:36:09.324719+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:36:09.324719+0000","last_undegraded":"2024-09-16T10:36:09.324719+0000","last_fullsized":"2024-09-16T10:36:09.324719+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29147,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:11.186312+0000","last_change":"2024-09-16T10:36:11.186430+0000","last_active":"2024-09-16T10:36:11.186312+0000","last_peered":"2024-09-16T10:36:11.186312+0000","last_clean":"2024-09-16T10:36:11.186312+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:11.186312+0000","last_undegraded":"2024-09-16T10:36:11.186312+0000","last_fullsized":"2024-09-16T10:36:11.186312+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:11.186295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:11.186295+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:59.156525+0000","scrub_duration":0.010167803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":94,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320702+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:36:09.320702+0000","last_peered":"2024-09-16T10:36:09.320702+0000","last_clean":"2024-09-16T10:36:09.320702+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:36:09.320702+0000","last_undegraded":"2024-09-16T10:36:09.320702+0000","last_fullsized":"2024-09-16T10:36:09.320702+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29101,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.318922+0000","last_change":"2024-09-16T10:36:09.318922+0000","last_active":"2024-09-16T10:36:09.318922+0000","last_peered":"2024-09-16T10:36:09.318922+0000","last_clean":"2024-09-16T10:36:09.318922+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:09.318922+0000","last_undegraded":"2024-09-16T10:36:09.318922+0000","last_fullsized":"2024-09-16T10:36:09.318922+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321071+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:36:09.321071+0000","last_peered":"2024-09-16T10:36:09.321071+0000","last_clean":"2024-09-16T10:36:09.321071+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:36:09.321071+0000","last_undegraded":"2024-09-16T10:36:09.321071+0000","last_fullsized":"2024-09-16T10:36:09.321071+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320855+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:36:09.320855+0000","last_peered":"2024-09-16T10:36:09.320855+0000","last_clean":"2024-09-16T10:36:09.320855+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:36:09.320855+0000","last_undegraded":"2024-09-16T10:36:09.320855+0000","last_fullsized":"2024-09-16T10:36:09.320855+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21214,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320982+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:36:09.320982+0000","last_peered":"2024-09-16T10:36:09.320982+0000","last_clean":"2024-09-16T10:36:09.320982+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:36:09.320982+0000","last_undegraded":"2024-09-16T10:36:09.320982+0000","last_fullsized":"2024-09-16T10:36:09.320982+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319329+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:36:09.319329+0000","last_peered":"2024-09-16T10:36:09.319329+0000","last_clean":"2024-09-16T10:36:09.319329+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:09.319329+0000","last_undegraded":"2024-09-16T10:36:09.319329+0000","last_fullsized":"2024-09-16T10:36:09.319329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24522,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320895+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:36:09.320895+0000","last_peered":"2024-09-16T10:36:09.320895+0000","last_clean":"2024-09-16T10:36:09.320895+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:36:09.320895+0000","last_undegraded":"2024-09-16T10:36:09.320895+0000","last_fullsized":"2024-09-16T10:36:09.320895+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320943+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:36:09.320943+0000","last_peered":"2024-09-16T10:36:09.320943+0000","last_clean":"2024-09-16T10:36:09.320943+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:36:09.320943+0000","last_undegraded":"2024-09-16T10:36:09.320943+0000","last_fullsized":"2024-09-16T10:36:09.320943+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":89,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321058+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:36:09.321058+0000","last_peered":"2024-09-16T10:36:09.321058+0000","last_clean":"2024-09-16T10:36:09.321058+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:36:09.321058+0000","last_undegraded":"2024-09-16T10:36:09.321058+0000","last_fullsized":"2024-09-16T10:36:09.321058+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21159,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320799+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:36:09.320799+0000","last_peered":"2024-09-16T10:36:09.320799+0000","last_clean":"2024-09-16T10:36:09.320799+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:36:09.320799+0000","last_undegraded":"2024-09-16T10:36:09.320799+0000","last_fullsized":"2024-09-16T10:36:09.320799+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5796,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244373,"num_write_kb":443942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38426,"num_bytes_recovered":22503907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019045,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347172,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024668,"statfs":{"total":96636764160,"available":96281260032,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505516,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866324,"statfs":{"total":96636764160,"available":96119115776,"internally_reserved":0,"allocated":557056,"data_stored":185777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.218},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27900000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455400,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916440,"statfs":{"total":96636764160,"available":96170434560,"internally_reserved":0,"allocated":552960,"data_stored":185295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.433}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:18.183 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T10:36:18.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:18.465 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:18.466 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:18.477 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-16T10:36:16.789911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5815,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244375,"num_write_kb":443944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38427,"num_bytes_recovered":22503926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1308088,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281807432,"statfs":{"total":289910292480,"available":288570810368,"internally_reserved":0,"allocated":1658880,"data_stored":556347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2898,"num_objects":-6,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-61914,"num_write_kb":-123811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11333,"num_bytes_recovered":-17190912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001690"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28956,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.350311+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:36:09.350311+0000","last_peered":"2024-09-16T10:36:09.350311+0000","last_clean":"2024-09-16T10:36:09.350311+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:09.350311+0000","last_undegraded":"2024-09-16T10:36:09.350311+0000","last_fullsized":"2024-09-16T10:36:09.350311+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20645,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319359+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:36:09.319359+0000","last_peered":"2024-09-16T10:36:09.319359+0000","last_clean":"2024-09-16T10:36:09.319359+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:09.319359+0000","last_undegraded":"2024-09-16T10:36:09.319359+0000","last_fullsized":"2024-09-16T10:36:09.319359+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20947,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320879+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:36:09.320879+0000","last_peered":"2024-09-16T10:36:09.320879+0000","last_clean":"2024-09-16T10:36:09.320879+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:36:09.320879+0000","last_undegraded":"2024-09-16T10:36:09.320879+0000","last_fullsized":"2024-09-16T10:36:09.320879+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20544,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319239+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:36:09.319239+0000","last_peered":"2024-09-16T10:36:09.319239+0000","last_clean":"2024-09-16T10:36:09.319239+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:09.319239+0000","last_undegraded":"2024-09-16T10:36:09.319239+0000","last_fullsized":"2024-09-16T10:36:09.319239+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20688,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.328449+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:36:09.328449+0000","last_peered":"2024-09-16T10:36:09.328449+0000","last_clean":"2024-09-16T10:36:09.328449+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:36:09.328449+0000","last_undegraded":"2024-09-16T10:36:09.328449+0000","last_fullsized":"2024-09-16T10:36:09.328449+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20726,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320596+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:36:09.320596+0000","last_peered":"2024-09-16T10:36:09.320596+0000","last_clean":"2024-09-16T10:36:09.320596+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:36:09.320596+0000","last_undegraded":"2024-09-16T10:36:09.320596+0000","last_fullsized":"2024-09-16T10:36:09.320596+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:10.405243+0000","last_change":"2024-09-16T10:36:10.404790+0000","last_active":"2024-09-16T10:36:10.405243+0000","last_peered":"2024-09-16T10:36:10.405243+0000","last_clean":"2024-09-16T10:36:10.405243+0000","last_became_active":"2024-09-16T10:36:10.324803+0000","last_became_peered":"2024-09-16T10:36:10.324803+0000","last_unstale":"2024-09-16T10:36:10.405243+0000","last_undegraded":"2024-09-16T10:36:10.405243+0000","last_fullsized":"2024-09-16T10:36:10.405243+0000","mapping_epoch":65,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:18.250767+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20591,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.326828+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:36:09.326828+0000","last_peered":"2024-09-16T10:36:09.326828+0000","last_clean":"2024-09-16T10:36:09.326828+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:36:09.326828+0000","last_undegraded":"2024-09-16T10:36:09.326828+0000","last_fullsized":"2024-09-16T10:36:09.326828+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":81,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320376+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:36:09.320376+0000","last_peered":"2024-09-16T10:36:09.320376+0000","last_clean":"2024-09-16T10:36:09.320376+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:36:09.320376+0000","last_undegraded":"2024-09-16T10:36:09.320376+0000","last_fullsized":"2024-09-16T10:36:09.320376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":29002,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.817525+0000","last_change":"2024-09-16T10:36:12.817640+0000","last_active":"2024-09-16T10:36:12.817525+0000","last_peered":"2024-09-16T10:36:12.817525+0000","last_clean":"2024-09-16T10:36:12.817525+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:36:12.817525+0000","last_undegraded":"2024-09-16T10:36:12.817525+0000","last_fullsized":"2024-09-16T10:36:12.817525+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20461","last_scrub_stamp":"2024-09-16T10:36:12.817500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.817500+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:34.937908+0000","scrub_duration":0.010570972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319055+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:36:09.319055+0000","last_peered":"2024-09-16T10:36:09.319055+0000","last_clean":"2024-09-16T10:36:09.319055+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:09.319055+0000","last_undegraded":"2024-09-16T10:36:09.319055+0000","last_fullsized":"2024-09-16T10:36:09.319055+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30039,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.660270+0000","last_change":"2024-09-16T10:36:12.660360+0000","last_active":"2024-09-16T10:36:12.660270+0000","last_peered":"2024-09-16T10:36:12.660270+0000","last_clean":"2024-09-16T10:36:12.660270+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:36:12.660270+0000","last_undegraded":"2024-09-16T10:36:12.660270+0000","last_fullsized":"2024-09-16T10:36:12.660270+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'21040","last_scrub_stamp":"2024-09-16T10:36:12.660255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.660255+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:37.144016+0000","scrub_duration":0.0082777990000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":114,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.324719+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:36:09.324719+0000","last_peered":"2024-09-16T10:36:09.324719+0000","last_clean":"2024-09-16T10:36:09.324719+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:36:09.324719+0000","last_undegraded":"2024-09-16T10:36:09.324719+0000","last_fullsized":"2024-09-16T10:36:09.324719+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29147,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:11.186312+0000","last_change":"2024-09-16T10:36:11.186430+0000","last_active":"2024-09-16T10:36:11.186312+0000","last_peered":"2024-09-16T10:36:11.186312+0000","last_clean":"2024-09-16T10:36:11.186312+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:11.186312+0000","last_undegraded":"2024-09-16T10:36:11.186312+0000","last_fullsized":"2024-09-16T10:36:11.186312+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:11.186295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:11.186295+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:59.156525+0000","scrub_duration":0.010167803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":94,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320702+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:36:09.320702+0000","last_peered":"2024-09-16T10:36:09.320702+0000","last_clean":"2024-09-16T10:36:09.320702+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:36:09.320702+0000","last_undegraded":"2024-09-16T10:36:09.320702+0000","last_fullsized":"2024-09-16T10:36:09.320702+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29101,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.318922+0000","last_change":"2024-09-16T10:36:09.318922+0000","last_active":"2024-09-16T10:36:09.318922+0000","last_peered":"2024-09-16T10:36:09.318922+0000","last_clean":"2024-09-16T10:36:09.318922+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:09.318922+0000","last_undegraded":"2024-09-16T10:36:09.318922+0000","last_fullsized":"2024-09-16T10:36:09.318922+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20574,"num_write_kb":41143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4196,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321071+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:36:09.321071+0000","last_peered":"2024-09-16T10:36:09.321071+0000","last_clean":"2024-09-16T10:36:09.321071+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:36:09.321071+0000","last_undegraded":"2024-09-16T10:36:09.321071+0000","last_fullsized":"2024-09-16T10:36:09.321071+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320855+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:36:09.320855+0000","last_peered":"2024-09-16T10:36:09.320855+0000","last_clean":"2024-09-16T10:36:09.320855+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:36:09.320855+0000","last_undegraded":"2024-09-16T10:36:09.320855+0000","last_fullsized":"2024-09-16T10:36:09.320855+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21214,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320982+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:36:09.320982+0000","last_peered":"2024-09-16T10:36:09.320982+0000","last_clean":"2024-09-16T10:36:09.320982+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:36:09.320982+0000","last_undegraded":"2024-09-16T10:36:09.320982+0000","last_fullsized":"2024-09-16T10:36:09.320982+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319329+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:36:09.319329+0000","last_peered":"2024-09-16T10:36:09.319329+0000","last_clean":"2024-09-16T10:36:09.319329+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:09.319329+0000","last_undegraded":"2024-09-16T10:36:09.319329+0000","last_fullsized":"2024-09-16T10:36:09.319329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24522,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320895+0000","last_change":"2024-09-16T10:33:36.977304+0000","last_active":"2024-09-16T10:36:09.320895+0000","last_peered":"2024-09-16T10:36:09.320895+0000","last_clean":"2024-09-16T10:36:09.320895+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:36:09.320895+0000","last_undegraded":"2024-09-16T10:36:09.320895+0000","last_fullsized":"2024-09-16T10:36:09.320895+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20138,"num_write_kb":40274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4171,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320943+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:36:09.320943+0000","last_peered":"2024-09-16T10:36:09.320943+0000","last_clean":"2024-09-16T10:36:09.320943+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:36:09.320943+0000","last_undegraded":"2024-09-16T10:36:09.320943+0000","last_fullsized":"2024-09-16T10:36:09.320943+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":89,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321058+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:36:09.321058+0000","last_peered":"2024-09-16T10:36:09.321058+0000","last_clean":"2024-09-16T10:36:09.321058+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:36:09.321058+0000","last_undegraded":"2024-09-16T10:36:09.321058+0000","last_fullsized":"2024-09-16T10:36:09.321058+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21159,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320799+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:36:09.320799+0000","last_peered":"2024-09-16T10:36:09.320799+0000","last_clean":"2024-09-16T10:36:09.320799+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:36:09.320799+0000","last_undegraded":"2024-09-16T10:36:09.320799+0000","last_fullsized":"2024-09-16T10:36:09.320799+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":5796,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":244373,"num_write_kb":443942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38426,"num_bytes_recovered":22503907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019045,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347172,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024668,"statfs":{"total":96636764160,"available":96281260032,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.246},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.32600000000000001}]}]},{"osd":1,"up_from":40,"seq":171798691989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505516,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866324,"statfs":{"total":96636764160,"available":96119115776,"internally_reserved":0,"allocated":557056,"data_stored":185777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.218},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.24399999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.27900000000000003}]}]},{"osd":0,"up_from":24,"seq":103079215272,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455400,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916440,"statfs":{"total":96636764160,"available":96170434560,"internally_reserved":0,"allocated":552960,"data_stored":185295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.433}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:18.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T10:36:18.763 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:18.763 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":66,"fsid":"3f765c65-edd5-4e6d-930a-33ce25cc666f","created":"2024-09-16T10:21:51.212707+0000","modified":"2024-09-16T10:36:10.316961+0000","last_up_change":"2024-09-16T10:23:54.356705+0000","last_in_change":"2024-09-16T10:23:48.304391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-16T10:22:00.707491+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":13,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unfounddel_pool","create_time":"2024-09-16T10:22:12.124014+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"346f1b3d-875c-427b-9fb3-a906c6ea62f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":20,"up_from":24,"up_thru":64,"down_at":21,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":2470025375}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6805","nonce":2470025375}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6807","nonce":2470025375}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":2470025375}]},"public_addr":"172.21.15.50:6804/2470025375","cluster_addr":"172.21.15.50:6805/2470025375","heartbeat_back_addr":"172.21.15.50:6807/2470025375","heartbeat_front_addr":"172.21.15.50:6806/2470025375","state":["exists","up"]},{"osd":1,"uuid":"c991e2ab-7bd4-4499-bd8f-a2f7c5581928","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":25,"up_from":40,"up_thru":65,"down_at":26,"lost_at":26,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":832073999}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6809","nonce":832073999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6811","nonce":832073999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":832073999}]},"public_addr":"172.21.15.50:6808/832073999","cluster_addr":"172.21.15.50:6809/832073999","heartbeat_back_addr":"172.21.15.50:6811/832073999","heartbeat_front_addr":"172.21.15.50:6810/832073999","state":["exists","up"]},{"osd":2,"uuid":"af14e6f0-c6ad-4545-ae6c-69130d4ec765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":17,"up_from":30,"up_thru":55,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":1896614959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6801","nonce":1896614959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6803","nonce":1896614959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":1896614959}]},"public_addr":"172.21.15.50:6800/1896614959","cluster_addr":"172.21.15.50:6801/1896614959","heartbeat_back_addr":"172.21.15.50:6803/1896614959","heartbeat_front_addr":"172.21.15.50:6802/1896614959","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.244822+0000","dead_epoch":21},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.213317+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T10:21:58.228331+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T10:36:19.777 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T10:36:19.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T10:36:19.914 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:36:19.914 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T10:36:19.914 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:36:19.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T10:36:20.210 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T10:36:20.223 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T10:36:20.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T10:36:20.348 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:36:20.348 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T10:36:20.348 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:36:20.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T10:36:20.641 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T10:36:20.654 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T10:36:20.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T10:36:20.778 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-09-16T10:36:20.778 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T10:36:20.778 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-09-16T10:36:20.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T10:36:21.069 INFO:teuthology.orchestra.run.smithi050.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T10:36:21.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:21.367 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:21.367 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:21.379 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-16T10:36:20.790700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3883,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203663,"num_write_kb":362527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30060,"num_bytes_recovered":17175030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1308084,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281807436,"statfs":{"total":289910292480,"available":288570814464,"internally_reserved":0,"allocated":1658880,"data_stored":556347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4830,"num_objects":-10,"num_object_clones":0,"num_object_copies":-20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-102626,"num_write_kb":-205228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-19700,"num_bytes_recovered":-22519808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.471769"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28956,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.350311+0000","last_change":"2024-09-16T10:29:03.999373+0000","last_active":"2024-09-16T10:36:09.350311+0000","last_peered":"2024-09-16T10:36:09.350311+0000","last_clean":"2024-09-16T10:36:09.350311+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:09.350311+0000","last_undegraded":"2024-09-16T10:36:09.350311+0000","last_fullsized":"2024-09-16T10:36:09.350311+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:45.088829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20645,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319359+0000","last_change":"2024-09-16T10:23:46.659541+0000","last_active":"2024-09-16T10:36:09.319359+0000","last_peered":"2024-09-16T10:36:09.319359+0000","last_clean":"2024-09-16T10:36:09.319359+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:09.319359+0000","last_undegraded":"2024-09-16T10:36:09.319359+0000","last_fullsized":"2024-09-16T10:36:09.319359+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3842","last_scrub_stamp":"2024-09-16T10:23:46.659404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:46.659404+0000","objects_scrubbed":3781,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:35:43.678673+0000","scrub_duration":1.5658865550000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20948,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:10.321528+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:36:10.321528+0000","last_peered":"2024-09-16T10:36:10.321528+0000","last_clean":"2024-09-16T10:36:10.321528+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:36:10.321528+0000","last_undegraded":"2024-09-16T10:36:10.321528+0000","last_fullsized":"2024-09-16T10:36:10.321528+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:28:34.206722+0000","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20544,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319239+0000","last_change":"2024-09-16T10:23:02.014955+0000","last_active":"2024-09-16T10:36:09.319239+0000","last_peered":"2024-09-16T10:36:09.319239+0000","last_clean":"2024-09-16T10:36:09.319239+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:09.319239+0000","last_undegraded":"2024-09-16T10:36:09.319239+0000","last_fullsized":"2024-09-16T10:36:09.319239+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:08:52.742091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20688,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.328449+0000","last_change":"2024-09-16T10:23:40.842061+0000","last_active":"2024-09-16T10:36:09.328449+0000","last_peered":"2024-09-16T10:36:09.328449+0000","last_clean":"2024-09-16T10:36:09.328449+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:36:09.328449+0000","last_undegraded":"2024-09-16T10:36:09.328449+0000","last_fullsized":"2024-09-16T10:36:09.328449+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3405","last_scrub_stamp":"2024-09-16T10:23:40.841845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:40.841845+0000","objects_scrubbed":3351,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:54:57.253449+0000","scrub_duration":1.3901078199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20726,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320596+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:36:09.320596+0000","last_peered":"2024-09-16T10:36:09.320596+0000","last_clean":"2024-09-16T10:36:09.320596+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:36:09.320596+0000","last_undegraded":"2024-09-16T10:36:09.320596+0000","last_fullsized":"2024-09-16T10:36:09.320596+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:17:16.018480+0000","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21339,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:10.405243+0000","last_change":"2024-09-16T10:36:10.404790+0000","last_active":"2024-09-16T10:36:10.405243+0000","last_peered":"2024-09-16T10:36:10.405243+0000","last_clean":"2024-09-16T10:36:10.405243+0000","last_became_active":"2024-09-16T10:36:10.324803+0000","last_became_peered":"2024-09-16T10:36:10.324803+0000","last_unstale":"2024-09-16T10:36:10.405243+0000","last_undegraded":"2024-09-16T10:36:10.405243+0000","last_fullsized":"2024-09-16T10:36:10.405243+0000","mapping_epoch":65,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:18.250767+0000","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20591,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.326828+0000","last_change":"2024-09-16T10:23:54.430222+0000","last_active":"2024-09-16T10:36:09.326828+0000","last_peered":"2024-09-16T10:36:09.326828+0000","last_clean":"2024-09-16T10:36:09.326828+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:36:09.326828+0000","last_undegraded":"2024-09-16T10:36:09.326828+0000","last_fullsized":"2024-09-16T10:36:09.326828+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"40'4243","last_scrub_stamp":"2024-09-16T10:23:54.430084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:54.430084+0000","objects_scrubbed":4195,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:13:39.759343+0000","scrub_duration":1.958036997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":81,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320376+0000","last_change":"2024-09-16T10:23:56.374107+0000","last_active":"2024-09-16T10:36:09.320376+0000","last_peered":"2024-09-16T10:36:09.320376+0000","last_clean":"2024-09-16T10:36:09.320376+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:36:09.320376+0000","last_undegraded":"2024-09-16T10:36:09.320376+0000","last_fullsized":"2024-09-16T10:36:09.320376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:39:15.411213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":29002,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.817525+0000","last_change":"2024-09-16T10:36:12.817640+0000","last_active":"2024-09-16T10:36:12.817525+0000","last_peered":"2024-09-16T10:36:12.817525+0000","last_clean":"2024-09-16T10:36:12.817525+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:36:12.817525+0000","last_undegraded":"2024-09-16T10:36:12.817525+0000","last_fullsized":"2024-09-16T10:36:12.817525+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20461","last_scrub_stamp":"2024-09-16T10:36:12.817500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.817500+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:34.937908+0000","scrub_duration":0.010570972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319055+0000","last_change":"2024-09-16T10:23:02.013960+0000","last_active":"2024-09-16T10:36:09.319055+0000","last_peered":"2024-09-16T10:36:09.319055+0000","last_clean":"2024-09-16T10:36:09.319055+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:09.319055+0000","last_undegraded":"2024-09-16T10:36:09.319055+0000","last_fullsized":"2024-09-16T10:36:09.319055+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:28:27.875337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30039,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:12.660270+0000","last_change":"2024-09-16T10:36:12.660360+0000","last_active":"2024-09-16T10:36:12.660270+0000","last_peered":"2024-09-16T10:36:12.660270+0000","last_clean":"2024-09-16T10:36:12.660270+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:36:12.660270+0000","last_undegraded":"2024-09-16T10:36:12.660270+0000","last_fullsized":"2024-09-16T10:36:12.660270+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'21040","last_scrub_stamp":"2024-09-16T10:36:12.660255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.660255+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:59:37.144016+0000","scrub_duration":0.0082777990000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"13'2","reported_seq":114,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.324719+0000","last_change":"2024-09-16T10:23:56.896838+0000","last_active":"2024-09-16T10:36:09.324719+0000","last_peered":"2024-09-16T10:36:09.324719+0000","last_clean":"2024-09-16T10:36:09.324719+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:36:09.324719+0000","last_undegraded":"2024-09-16T10:36:09.324719+0000","last_fullsized":"2024-09-16T10:36:09.324719+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:20.957215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29147,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:11.186312+0000","last_change":"2024-09-16T10:36:11.186430+0000","last_active":"2024-09-16T10:36:11.186312+0000","last_peered":"2024-09-16T10:36:11.186312+0000","last_clean":"2024-09-16T10:36:11.186312+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:11.186312+0000","last_undegraded":"2024-09-16T10:36:11.186312+0000","last_fullsized":"2024-09-16T10:36:11.186312+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:11.186295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:11.186295+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:30:59.156525+0000","scrub_duration":0.010167803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"11'1","reported_seq":94,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320702+0000","last_change":"2024-09-16T10:23:56.435112+0000","last_active":"2024-09-16T10:36:09.320702+0000","last_peered":"2024-09-16T10:36:09.320702+0000","last_clean":"2024-09-16T10:36:09.320702+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:36:09.320702+0000","last_undegraded":"2024-09-16T10:36:09.320702+0000","last_fullsized":"2024-09-16T10:36:09.320702+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:19:23.847227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29113,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:13.226852+0000","last_change":"2024-09-16T10:36:13.226993+0000","last_active":"2024-09-16T10:36:13.226852+0000","last_peered":"2024-09-16T10:36:13.226852+0000","last_clean":"2024-09-16T10:36:13.226852+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:13.226852+0000","last_undegraded":"2024-09-16T10:36:13.226852+0000","last_fullsized":"2024-09-16T10:36:13.226852+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20576","last_scrub_stamp":"2024-09-16T10:36:13.226815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:13.226815+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:09:09.027108+0000","scrub_duration":0.010571746999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20301,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321071+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:36:09.321071+0000","last_peered":"2024-09-16T10:36:09.321071+0000","last_clean":"2024-09-16T10:36:09.321071+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:36:09.321071+0000","last_undegraded":"2024-09-16T10:36:09.321071+0000","last_fullsized":"2024-09-16T10:36:09.321071+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320855+0000","last_change":"2024-09-16T10:23:56.549863+0000","last_active":"2024-09-16T10:36:09.320855+0000","last_peered":"2024-09-16T10:36:09.320855+0000","last_clean":"2024-09-16T10:36:09.320855+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:36:09.320855+0000","last_undegraded":"2024-09-16T10:36:09.320855+0000","last_fullsized":"2024-09-16T10:36:09.320855+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:24:52.514334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21214,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320982+0000","last_change":"2024-09-16T10:34:52.221024+0000","last_active":"2024-09-16T10:36:09.320982+0000","last_peered":"2024-09-16T10:36:09.320982+0000","last_clean":"2024-09-16T10:36:09.320982+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:36:09.320982+0000","last_undegraded":"2024-09-16T10:36:09.320982+0000","last_fullsized":"2024-09-16T10:36:09.320982+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":2199,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1449,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20911,"num_write_kb":41816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4343,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":83,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.319329+0000","last_change":"2024-09-16T10:23:02.013918+0000","last_active":"2024-09-16T10:36:09.319329+0000","last_peered":"2024-09-16T10:36:09.319329+0000","last_clean":"2024-09-16T10:36:09.319329+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:09.319329+0000","last_undegraded":"2024-09-16T10:36:09.319329+0000","last_fullsized":"2024-09-16T10:36:09.319329+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:46:06.694784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24534,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:17.784605+0000","last_change":"2024-09-16T10:36:17.784696+0000","last_active":"2024-09-16T10:36:17.784605+0000","last_peered":"2024-09-16T10:36:17.784605+0000","last_clean":"2024-09-16T10:36:17.784605+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:36:17.784605+0000","last_undegraded":"2024-09-16T10:36:17.784605+0000","last_fullsized":"2024-09-16T10:36:17.784605+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20140","last_scrub_stamp":"2024-09-16T10:36:17.784592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:17.784592+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:29:11.501545+0000","scrub_duration":0.01070285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":80,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320943+0000","last_change":"2024-09-16T10:23:56.549308+0000","last_active":"2024-09-16T10:36:09.320943+0000","last_peered":"2024-09-16T10:36:09.320943+0000","last_clean":"2024-09-16T10:36:09.320943+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:36:09.320943+0000","last_undegraded":"2024-09-16T10:36:09.320943+0000","last_fullsized":"2024-09-16T10:36:09.320943+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:27:16.207625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":89,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.321058+0000","last_change":"2024-09-16T10:23:56.369471+0000","last_active":"2024-09-16T10:36:09.321058+0000","last_peered":"2024-09-16T10:36:09.321058+0000","last_clean":"2024-09-16T10:36:09.321058+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:36:09.321058+0000","last_undegraded":"2024-09-16T10:36:09.321058+0000","last_fullsized":"2024-09-16T10:36:09.321058+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:01.610622+0000","last_clean_scrub_stamp":"2024-09-16T10:22:01.610622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:46:58.100624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21159,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-16T10:36:09.320799+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:36:09.320799+0000","last_peered":"2024-09-16T10:36:09.320799+0000","last_clean":"2024-09-16T10:36:09.320799+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:36:09.320799+0000","last_undegraded":"2024-09-16T10:36:09.320799+0000","last_fullsized":"2024-09-16T10:36:09.320799+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3864,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":203661,"num_write_kb":362525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30059,"num_bytes_recovered":17175011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019046,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347168,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024672,"statfs":{"total":96636764160,"available":96281264128,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.34799999999999998}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.33000000000000002}]}]},{"osd":1,"up_from":40,"seq":171798691990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505516,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866324,"statfs":{"total":96636764160,"available":96119115776,"internally_reserved":0,"allocated":557056,"data_stored":185777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20200000000000001,"5min":2.1070000000000002,"15min":3.0379999999999998},"min":{"1min":0.16500000000000001,"5min":0.156,"15min":0.156},"max":{"1min":0.37,"5min":12.798,"15min":17.675000000000001},"last":0.21299999999999999},{"interface":"front","average":{"1min":0.23799999999999999,"5min":2.2890000000000001,"15min":2.629},"min":{"1min":0.20100000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.39900000000000002,"5min":15.151,"15min":16.850999999999999},"last":0.248}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.251,"5min":1.8740000000000001,"15min":2.3540000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.40300000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.28699999999999998,"5min":1.748,"15min":2.5649999999999999},"min":{"1min":0.247,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47099999999999997,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.22500000000000001}]}]},{"osd":0,"up_from":24,"seq":103079215273,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455400,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916440,"statfs":{"total":96636764160,"available":96170434560,"internally_reserved":0,"allocated":552960,"data_stored":185295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.20399999999999999,"5min":1.8120000000000001,"15min":2.4369999999999998},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.27600000000000002,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.23100000000000001,"5min":2.141,"15min":2.1539999999999999},"min":{"1min":0.17899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.36599999999999999,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.26200000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:35:21 2024","interfaces":[{"interface":"back","average":{"1min":0.24399999999999999,"5min":1.9339999999999999,"15min":1.8919999999999999},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.16800000000000001},"max":{"1min":0.33200000000000002,"5min":15.534000000000001,"15min":18.093},"last":0.315},{"interface":"front","average":{"1min":0.26200000000000001,"5min":1.748,"15min":1.853},"min":{"1min":0.19700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":0.40699999999999997,"5min":11.863,"15min":13.752000000000001},"last":0.28899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:21.384 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-09-16T10:22:12.997533+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.384 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-09-16T10:23:46.659404+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=23, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.384 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-09-16T10:23:47.680381+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=23, tm_sec=47, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.385 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-09-16T10:22:12.997533+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.385 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-09-16T10:23:40.841845+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=23, tm_sec=40, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.385 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-09-16T10:23:41.819370+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=23, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.385 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-16T10:23:43.952628+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=23, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.385 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-09-16T10:23:54.430084+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=23, tm_sec=54, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.385 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T10:22:01.610622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.385 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T10:36:12.817500+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.385 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-16T10:22:01.610622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.385 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T10:36:12.660255+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.386 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-16T10:22:01.610622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.386 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T10:36:11.186295+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.386 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-16T10:22:01.610622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.386 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T10:36:13.226815+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=13, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.386 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T10:22:12.997533+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.386 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-16T10:22:01.610622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.386 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T10:22:12.997533+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.386 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-16T10:22:01.610622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.386 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T10:36:17.784592+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.387 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-16T10:22:01.610622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.387 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-16T10:22:01.610622+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=1, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.387 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T10:22:12.997533+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:21.387 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T10:36:41.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:36:41.686 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:36:41.686 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:36:41.700 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-16T10:36:40.795004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2434,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182752,"num_write_kb":320711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25717,"num_bytes_recovered":17175030,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160827,"ondisk_log_size":160827,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1308076,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281807444,"statfs":{"total":289910292480,"available":288570822656,"internally_reserved":0,"allocated":1658880,"data_stored":556347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002622"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28968,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:27.592739+0000","last_change":"2024-09-16T10:36:27.592893+0000","last_active":"2024-09-16T10:36:27.592739+0000","last_peered":"2024-09-16T10:36:27.592739+0000","last_clean":"2024-09-16T10:36:27.592739+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:27.592739+0000","last_undegraded":"2024-09-16T10:36:27.592739+0000","last_fullsized":"2024-09-16T10:36:27.592739+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20468","last_scrub_stamp":"2024-09-16T10:36:27.592717+0000","last_deep_scrub":"48'20468","last_deep_scrub_stamp":"2024-09-16T10:36:27.592717+0000","last_clean_scrub_stamp":"2024-09-16T10:36:27.592717+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:44.024728+0000","scrub_duration":0.011921997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20657,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:30.159278+0000","last_change":"2024-09-16T10:36:30.159394+0000","last_active":"2024-09-16T10:36:30.159278+0000","last_peered":"2024-09-16T10:36:30.159278+0000","last_clean":"2024-09-16T10:36:30.159278+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:30.159278+0000","last_undegraded":"2024-09-16T10:36:30.159278+0000","last_fullsized":"2024-09-16T10:36:30.159278+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20462","last_scrub_stamp":"2024-09-16T10:36:30.159254+0000","last_deep_scrub":"48'20462","last_deep_scrub_stamp":"2024-09-16T10:36:30.159254+0000","last_clean_scrub_stamp":"2024-09-16T10:36:30.159254+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:31.876273+0000","scrub_duration":0.013664887000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20949,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:18.458272+0000","last_change":"2024-09-16T10:34:02.253461+0000","last_active":"2024-09-16T10:36:18.458272+0000","last_peered":"2024-09-16T10:36:18.458272+0000","last_clean":"2024-09-16T10:36:18.458272+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:36:18.458272+0000","last_undegraded":"2024-09-16T10:36:18.458272+0000","last_fullsized":"2024-09-16T10:36:18.458272+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'4014","last_scrub_stamp":"2024-09-16T10:23:47.680381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:47.680381+0000","objects_scrubbed":3939,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for deep scrub","scrub_duration":1.621791945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20557,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:36.085979+0000","last_change":"2024-09-16T10:36:36.086122+0000","last_active":"2024-09-16T10:36:36.085979+0000","last_peered":"2024-09-16T10:36:36.085979+0000","last_clean":"2024-09-16T10:36:36.085979+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:36.085979+0000","last_undegraded":"2024-09-16T10:36:36.085979+0000","last_fullsized":"2024-09-16T10:36:36.085979+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20462","last_scrub_stamp":"2024-09-16T10:36:36.085953+0000","last_deep_scrub":"48'20462","last_deep_scrub_stamp":"2024-09-16T10:36:36.085953+0000","last_clean_scrub_stamp":"2024-09-16T10:36:36.085953+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:49:25.173219+0000","scrub_duration":0.011643202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20700,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:26.601145+0000","last_change":"2024-09-16T10:36:26.601316+0000","last_active":"2024-09-16T10:36:26.601145+0000","last_peered":"2024-09-16T10:36:26.601145+0000","last_clean":"2024-09-16T10:36:26.601145+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:36:26.601145+0000","last_undegraded":"2024-09-16T10:36:26.601145+0000","last_fullsized":"2024-09-16T10:36:26.601145+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20505","last_scrub_stamp":"2024-09-16T10:36:26.601118+0000","last_deep_scrub":"48'20505","last_deep_scrub_stamp":"2024-09-16T10:36:26.601118+0000","last_clean_scrub_stamp":"2024-09-16T10:36:26.601118+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:21.213545+0000","scrub_duration":0.011907403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20728,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:20.640410+0000","last_change":"2024-09-16T10:34:18.284717+0000","last_active":"2024-09-16T10:36:20.640410+0000","last_peered":"2024-09-16T10:36:20.640410+0000","last_clean":"2024-09-16T10:36:20.640410+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:36:20.640410+0000","last_undegraded":"2024-09-16T10:36:20.640410+0000","last_fullsized":"2024-09-16T10:36:20.640410+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3438","last_scrub_stamp":"2024-09-16T10:23:41.819370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:41.819370+0000","objects_scrubbed":3370,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for deep scrub","scrub_duration":1.3540002499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21340,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:13.457869+0000","last_change":"2024-09-16T10:36:10.404790+0000","last_active":"2024-09-16T10:36:13.457869+0000","last_peered":"2024-09-16T10:36:13.457869+0000","last_clean":"2024-09-16T10:36:13.457869+0000","last_became_active":"2024-09-16T10:36:10.324803+0000","last_became_peered":"2024-09-16T10:36:10.324803+0000","last_unstale":"2024-09-16T10:36:13.457869+0000","last_undegraded":"2024-09-16T10:36:13.457869+0000","last_fullsized":"2024-09-16T10:36:13.457869+0000","mapping_epoch":65,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for deep scrub","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20603,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:25.598453+0000","last_change":"2024-09-16T10:36:25.599099+0000","last_active":"2024-09-16T10:36:25.598453+0000","last_peered":"2024-09-16T10:36:25.598453+0000","last_clean":"2024-09-16T10:36:25.598453+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:36:25.598453+0000","last_undegraded":"2024-09-16T10:36:25.598453+0000","last_fullsized":"2024-09-16T10:36:25.598453+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20390","last_scrub_stamp":"2024-09-16T10:36:25.598429+0000","last_deep_scrub":"48'20390","last_deep_scrub_stamp":"2024-09-16T10:36:25.598429+0000","last_clean_scrub_stamp":"2024-09-16T10:36:25.598429+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:56:42.248280+0000","scrub_duration":0.01328883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":93,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:21.754457+0000","last_change":"2024-09-16T10:36:21.754550+0000","last_active":"2024-09-16T10:36:21.754457+0000","last_peered":"2024-09-16T10:36:21.754457+0000","last_clean":"2024-09-16T10:36:21.754457+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:36:21.754457+0000","last_undegraded":"2024-09-16T10:36:21.754457+0000","last_fullsized":"2024-09-16T10:36:21.754457+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:21.754437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:21.754437+0000","last_clean_scrub_stamp":"2024-09-16T10:36:21.754437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:53.049486+0000","scrub_duration":0.0014528500000000001,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":29004,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:20.640287+0000","last_change":"2024-09-16T10:36:12.817640+0000","last_active":"2024-09-16T10:36:20.640287+0000","last_peered":"2024-09-16T10:36:20.640287+0000","last_clean":"2024-09-16T10:36:20.640287+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:36:20.640287+0000","last_undegraded":"2024-09-16T10:36:20.640287+0000","last_fullsized":"2024-09-16T10:36:20.640287+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20461","last_scrub_stamp":"2024-09-16T10:36:12.817500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:12.817500+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.010570972,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":95,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:23.104711+0000","last_change":"2024-09-16T10:36:23.104850+0000","last_active":"2024-09-16T10:36:23.104711+0000","last_peered":"2024-09-16T10:36:23.104711+0000","last_clean":"2024-09-16T10:36:23.104711+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:23.104711+0000","last_undegraded":"2024-09-16T10:36:23.104711+0000","last_fullsized":"2024-09-16T10:36:23.104711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:23.104689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:23.104689+0000","last_clean_scrub_stamp":"2024-09-16T10:36:23.104689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:11:16.453684+0000","scrub_duration":0.0015296260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30051,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:22.574110+0000","last_change":"2024-09-16T10:36:22.574220+0000","last_active":"2024-09-16T10:36:22.574110+0000","last_peered":"2024-09-16T10:36:22.574110+0000","last_clean":"2024-09-16T10:36:22.574110+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:36:22.574110+0000","last_undegraded":"2024-09-16T10:36:22.574110+0000","last_fullsized":"2024-09-16T10:36:22.574110+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'21040","last_scrub_stamp":"2024-09-16T10:36:22.574085+0000","last_deep_scrub":"48'21040","last_deep_scrub_stamp":"2024-09-16T10:36:22.574085+0000","last_clean_scrub_stamp":"2024-09-16T10:36:22.574085+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:41:07.518035+0000","scrub_duration":0.012370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"66'3","reported_seq":127,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:24.553161+0000","last_change":"2024-09-16T10:36:24.553260+0000","last_active":"2024-09-16T10:36:24.553161+0000","last_peered":"2024-09-16T10:36:24.553161+0000","last_clean":"2024-09-16T10:36:24.553161+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:36:24.553161+0000","last_undegraded":"2024-09-16T10:36:24.553161+0000","last_fullsized":"2024-09-16T10:36:24.553161+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'3","last_scrub_stamp":"2024-09-16T10:36:24.553137+0000","last_deep_scrub":"66'3","last_deep_scrub_stamp":"2024-09-16T10:36:24.553137+0000","last_clean_scrub_stamp":"2024-09-16T10:36:24.553137+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:09:02.621610+0000","scrub_duration":0.0030244619999999999,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29148,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:11.186430+0000","last_change":"2024-09-16T10:36:11.186430+0000","last_active":"2024-09-16T10:36:11.186430+0000","last_peered":"2024-09-16T10:36:11.186430+0000","last_clean":"2024-09-16T10:36:11.186430+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:11.186430+0000","last_undegraded":"2024-09-16T10:36:11.186430+0000","last_fullsized":"2024-09-16T10:36:11.186430+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:11.186295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:11.186295+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.010167803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"66'2","reported_seq":108,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:27.777416+0000","last_change":"2024-09-16T10:36:27.777416+0000","last_active":"2024-09-16T10:36:27.777416+0000","last_peered":"2024-09-16T10:36:27.777416+0000","last_clean":"2024-09-16T10:36:27.777416+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:36:27.777416+0000","last_undegraded":"2024-09-16T10:36:27.777416+0000","last_fullsized":"2024-09-16T10:36:27.777416+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-16T10:36:27.777295+0000","last_deep_scrub":"66'2","last_deep_scrub_stamp":"2024-09-16T10:36:27.777295+0000","last_clean_scrub_stamp":"2024-09-16T10:36:27.777295+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:39.906763+0000","scrub_duration":0.002714008,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29114,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:13.226993+0000","last_change":"2024-09-16T10:36:13.226993+0000","last_active":"2024-09-16T10:36:13.226993+0000","last_peered":"2024-09-16T10:36:13.226993+0000","last_clean":"2024-09-16T10:36:13.226993+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:13.226993+0000","last_undegraded":"2024-09-16T10:36:13.226993+0000","last_fullsized":"2024-09-16T10:36:13.226993+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20576","last_scrub_stamp":"2024-09-16T10:36:13.226815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:13.226815+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.010571746999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20302,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:10.321268+0000","last_change":"2024-09-16T10:35:53.377384+0000","last_active":"2024-09-16T10:36:10.321268+0000","last_peered":"2024-09-16T10:36:10.321268+0000","last_clean":"2024-09-16T10:36:10.321268+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:36:10.321268+0000","last_undegraded":"2024-09-16T10:36:10.321268+0000","last_fullsized":"2024-09-16T10:36:10.321268+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20033,"num_write_kb":40065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":93,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:26.825748+0000","last_change":"2024-09-16T10:36:26.825748+0000","last_active":"2024-09-16T10:36:26.825748+0000","last_peered":"2024-09-16T10:36:26.825748+0000","last_clean":"2024-09-16T10:36:26.825748+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:36:26.825748+0000","last_undegraded":"2024-09-16T10:36:26.825748+0000","last_fullsized":"2024-09-16T10:36:26.825748+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:26.825614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:26.825614+0000","last_clean_scrub_stamp":"2024-09-16T10:36:26.825614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:13.087372+0000","scrub_duration":0.0027595470000000002,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21227,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:18.752002+0000","last_change":"2024-09-16T10:36:18.752002+0000","last_active":"2024-09-16T10:36:18.752002+0000","last_peered":"2024-09-16T10:36:18.752002+0000","last_clean":"2024-09-16T10:36:18.752002+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:36:18.752002+0000","last_undegraded":"2024-09-16T10:36:18.752002+0000","last_fullsized":"2024-09-16T10:36:18.752002+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20914","last_scrub_stamp":"2024-09-16T10:36:18.751903+0000","last_deep_scrub":"48'20914","last_deep_scrub_stamp":"2024-09-16T10:36:18.751903+0000","last_clean_scrub_stamp":"2024-09-16T10:36:18.751903+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.0091957210000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":95,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:34.130199+0000","last_change":"2024-09-16T10:36:34.130521+0000","last_active":"2024-09-16T10:36:34.130199+0000","last_peered":"2024-09-16T10:36:34.130199+0000","last_clean":"2024-09-16T10:36:34.130199+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:34.130199+0000","last_undegraded":"2024-09-16T10:36:34.130199+0000","last_fullsized":"2024-09-16T10:36:34.130199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:34.130176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:34.130176+0000","last_clean_scrub_stamp":"2024-09-16T10:36:34.130176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:15.595794+0000","scrub_duration":0.0016726289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24535,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:17.784696+0000","last_change":"2024-09-16T10:36:17.784696+0000","last_active":"2024-09-16T10:36:17.784696+0000","last_peered":"2024-09-16T10:36:17.784696+0000","last_clean":"2024-09-16T10:36:17.784696+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:36:17.784696+0000","last_undegraded":"2024-09-16T10:36:17.784696+0000","last_fullsized":"2024-09-16T10:36:17.784696+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20140","last_scrub_stamp":"2024-09-16T10:36:17.784592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:36:17.784592+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for deep scrub","scrub_duration":0.01070285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":93,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:28.801098+0000","last_change":"2024-09-16T10:36:28.801098+0000","last_active":"2024-09-16T10:36:28.801098+0000","last_peered":"2024-09-16T10:36:28.801098+0000","last_clean":"2024-09-16T10:36:28.801098+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:36:28.801098+0000","last_undegraded":"2024-09-16T10:36:28.801098+0000","last_fullsized":"2024-09-16T10:36:28.801098+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:28.800958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:28.800958+0000","last_clean_scrub_stamp":"2024-09-16T10:36:28.800958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:09.569886+0000","scrub_duration":0.001647702,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":101,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:24.742853+0000","last_change":"2024-09-16T10:36:24.742948+0000","last_active":"2024-09-16T10:36:24.742853+0000","last_peered":"2024-09-16T10:36:24.742853+0000","last_clean":"2024-09-16T10:36:24.742853+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:36:24.742853+0000","last_undegraded":"2024-09-16T10:36:24.742853+0000","last_fullsized":"2024-09-16T10:36:24.742853+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:24.742833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:24.742833+0000","last_clean_scrub_stamp":"2024-09-16T10:36:24.742833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:21.769923+0000","scrub_duration":0.0015081770000000001,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"48'20896","reported_seq":21160,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:10.321565+0000","last_change":"2024-09-16T10:35:13.238656+0000","last_active":"2024-09-16T10:36:10.321565+0000","last_peered":"2024-09-16T10:36:10.321565+0000","last_clean":"2024-09-16T10:36:10.321565+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:36:10.321565+0000","last_undegraded":"2024-09-16T10:36:10.321565+0000","last_fullsized":"2024-09-16T10:36:10.321565+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:22:12.997533+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1932,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20893,"num_write_kb":41775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4403,"num_bytes_recovered":4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":2415,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182750,"num_write_kb":320709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25716,"num_bytes_recovered":17175011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160822,"ondisk_log_size":160822,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019050,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347168,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024672,"statfs":{"total":96636764160,"available":96281264128,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.318}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.36299999999999999}]}]},{"osd":1,"up_from":40,"seq":171798691994,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505512,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866328,"statfs":{"total":96636764160,"available":96119119872,"internally_reserved":0,"allocated":557056,"data_stored":185777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.19700000000000001,"5min":1.5860000000000001,"15min":2.8109999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.27300000000000002,"5min":12.798,"15min":17.675000000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.23400000000000001,"5min":1.8220000000000001,"15min":2.484},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.318,"5min":15.151,"15min":16.850999999999999},"last":0.316}]},{"osd":2,"last update":"Mon Sep 16 10:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.24099999999999999,"5min":1.587,"15min":2.2160000000000002},"min":{"1min":0.19400000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.33900000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.27100000000000002,"5min":1.5009999999999999,"15min":2.391},"min":{"1min":0.219,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.34999999999999998,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.33400000000000002}]}]},{"osd":0,"up_from":24,"seq":103079215277,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455396,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916444,"statfs":{"total":96636764160,"available":96170438656,"internally_reserved":0,"allocated":552960,"data_stored":185295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.215,"5min":1.335,"15min":2.2639999999999998},"min":{"1min":0.157,"5min":0.157,"15min":0.157},"max":{"1min":0.40699999999999997,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.23100000000000001,"5min":1.466,"15min":2.0339999999999998},"min":{"1min":0.17399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.433,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.24099999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.26800000000000002,"5min":1.427,"15min":1.8700000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.16800000000000001},"max":{"1min":0.56200000000000006,"5min":15.534000000000001,"15min":18.093},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.27300000000000002,"5min":1.2749999999999999,"15min":1.7929999999999999},"min":{"1min":0.21199999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.53200000000000003,"5min":11.863,"15min":13.752000000000001},"last":0.28699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:36:41.702 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-09-16T10:23:47.680381+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=23, tm_sec=47, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:41.702 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-09-16T10:23:41.819370+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=23, tm_sec=41, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:41.703 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-16T10:23:43.952628+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=23, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:41.703 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T10:36:12.817500+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:41.703 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T10:36:11.186295+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=11, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:41.703 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T10:36:13.226815+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=13, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:41.703 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T10:22:12.997533+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:41.703 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T10:36:18.751903+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:41.703 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T10:36:17.784592+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:41.704 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T10:22:12.997533+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=22, tm_sec=12, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:36:41.704 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T10:37:01.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:37:02.003 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:37:02.003 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:37:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2024-09-16T10:37:00.799214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141826,"num_write_kb":238871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17153,"num_bytes_recovered":17170451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160828,"ondisk_log_size":160828,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1308076,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281807444,"statfs":{"total":289910292480,"available":288570822656,"internally_reserved":0,"allocated":1658880,"data_stored":556347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1932,"num_objects":-4,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-40926,"num_write_kb":-81840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8564,"num_bytes_recovered":-4579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002609"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28968,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:27.592739+0000","last_change":"2024-09-16T10:36:27.592893+0000","last_active":"2024-09-16T10:36:27.592739+0000","last_peered":"2024-09-16T10:36:27.592739+0000","last_clean":"2024-09-16T10:36:27.592739+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:27.592739+0000","last_undegraded":"2024-09-16T10:36:27.592739+0000","last_fullsized":"2024-09-16T10:36:27.592739+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20468","last_scrub_stamp":"2024-09-16T10:36:27.592717+0000","last_deep_scrub":"48'20468","last_deep_scrub_stamp":"2024-09-16T10:36:27.592717+0000","last_clean_scrub_stamp":"2024-09-16T10:36:27.592717+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:44.024728+0000","scrub_duration":0.011921997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20657,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:30.159278+0000","last_change":"2024-09-16T10:36:30.159394+0000","last_active":"2024-09-16T10:36:30.159278+0000","last_peered":"2024-09-16T10:36:30.159278+0000","last_clean":"2024-09-16T10:36:30.159278+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:30.159278+0000","last_undegraded":"2024-09-16T10:36:30.159278+0000","last_fullsized":"2024-09-16T10:36:30.159278+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20462","last_scrub_stamp":"2024-09-16T10:36:30.159254+0000","last_deep_scrub":"48'20462","last_deep_scrub_stamp":"2024-09-16T10:36:30.159254+0000","last_clean_scrub_stamp":"2024-09-16T10:36:30.159254+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:31.876273+0000","scrub_duration":0.013664887000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20960,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:48.694232+0000","last_change":"2024-09-16T10:36:48.694373+0000","last_active":"2024-09-16T10:36:48.694232+0000","last_peered":"2024-09-16T10:36:48.694232+0000","last_clean":"2024-09-16T10:36:48.694232+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:36:48.694232+0000","last_undegraded":"2024-09-16T10:36:48.694232+0000","last_fullsized":"2024-09-16T10:36:48.694232+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20664","last_scrub_stamp":"2024-09-16T10:36:48.694209+0000","last_deep_scrub":"48'20664","last_deep_scrub_stamp":"2024-09-16T10:36:48.694209+0000","last_clean_scrub_stamp":"2024-09-16T10:36:48.694209+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:13:40.564100+0000","scrub_duration":0.0098038299999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20557,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:36.085979+0000","last_change":"2024-09-16T10:36:36.086122+0000","last_active":"2024-09-16T10:36:36.085979+0000","last_peered":"2024-09-16T10:36:36.085979+0000","last_clean":"2024-09-16T10:36:36.085979+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:36.085979+0000","last_undegraded":"2024-09-16T10:36:36.085979+0000","last_fullsized":"2024-09-16T10:36:36.085979+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20462","last_scrub_stamp":"2024-09-16T10:36:36.085953+0000","last_deep_scrub":"48'20462","last_deep_scrub_stamp":"2024-09-16T10:36:36.085953+0000","last_clean_scrub_stamp":"2024-09-16T10:36:36.085953+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:49:25.173219+0000","scrub_duration":0.011643202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20700,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:26.601145+0000","last_change":"2024-09-16T10:36:26.601316+0000","last_active":"2024-09-16T10:36:26.601145+0000","last_peered":"2024-09-16T10:36:26.601145+0000","last_clean":"2024-09-16T10:36:26.601145+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:36:26.601145+0000","last_undegraded":"2024-09-16T10:36:26.601145+0000","last_fullsized":"2024-09-16T10:36:26.601145+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20505","last_scrub_stamp":"2024-09-16T10:36:26.601118+0000","last_deep_scrub":"48'20505","last_deep_scrub_stamp":"2024-09-16T10:36:26.601118+0000","last_clean_scrub_stamp":"2024-09-16T10:36:26.601118+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:21.213545+0000","scrub_duration":0.011907403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20739,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:49.676994+0000","last_change":"2024-09-16T10:36:49.677110+0000","last_active":"2024-09-16T10:36:49.676994+0000","last_peered":"2024-09-16T10:36:49.676994+0000","last_clean":"2024-09-16T10:36:49.676994+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:36:49.676994+0000","last_undegraded":"2024-09-16T10:36:49.676994+0000","last_fullsized":"2024-09-16T10:36:49.676994+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20371","last_scrub_stamp":"2024-09-16T10:36:49.676970+0000","last_deep_scrub":"48'20371","last_deep_scrub_stamp":"2024-09-16T10:36:49.676970+0000","last_clean_scrub_stamp":"2024-09-16T10:36:49.676970+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:24:09.853188+0000","scrub_duration":0.0099085990000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21340,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:13.457869+0000","last_change":"2024-09-16T10:36:10.404790+0000","last_active":"2024-09-16T10:36:13.457869+0000","last_peered":"2024-09-16T10:36:13.457869+0000","last_clean":"2024-09-16T10:36:13.457869+0000","last_became_active":"2024-09-16T10:36:10.324803+0000","last_became_peered":"2024-09-16T10:36:10.324803+0000","last_unstale":"2024-09-16T10:36:13.457869+0000","last_undegraded":"2024-09-16T10:36:13.457869+0000","last_fullsized":"2024-09-16T10:36:13.457869+0000","mapping_epoch":65,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'3824","last_scrub_stamp":"2024-09-16T10:23:43.952628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:22:12.997533+0000","last_clean_scrub_stamp":"2024-09-16T10:23:43.952628+0000","objects_scrubbed":3752,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for deep scrub","scrub_duration":1.536242753,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20604,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:25.599099+0000","last_change":"2024-09-16T10:36:25.599099+0000","last_active":"2024-09-16T10:36:25.599099+0000","last_peered":"2024-09-16T10:36:25.599099+0000","last_clean":"2024-09-16T10:36:25.599099+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:36:25.599099+0000","last_undegraded":"2024-09-16T10:36:25.599099+0000","last_fullsized":"2024-09-16T10:36:25.599099+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20390","last_scrub_stamp":"2024-09-16T10:36:25.598429+0000","last_deep_scrub":"48'20390","last_deep_scrub_stamp":"2024-09-16T10:36:25.598429+0000","last_clean_scrub_stamp":"2024-09-16T10:36:25.598429+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:56:42.248280+0000","scrub_duration":0.01328883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":93,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:21.754457+0000","last_change":"2024-09-16T10:36:21.754550+0000","last_active":"2024-09-16T10:36:21.754457+0000","last_peered":"2024-09-16T10:36:21.754457+0000","last_clean":"2024-09-16T10:36:21.754457+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:36:21.754457+0000","last_undegraded":"2024-09-16T10:36:21.754457+0000","last_fullsized":"2024-09-16T10:36:21.754457+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:21.754437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:21.754437+0000","last_clean_scrub_stamp":"2024-09-16T10:36:21.754437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:53.049486+0000","scrub_duration":0.0014528500000000001,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":29015,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:41.795119+0000","last_change":"2024-09-16T10:36:41.795203+0000","last_active":"2024-09-16T10:36:41.795119+0000","last_peered":"2024-09-16T10:36:41.795119+0000","last_clean":"2024-09-16T10:36:41.795119+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:36:41.795119+0000","last_undegraded":"2024-09-16T10:36:41.795119+0000","last_fullsized":"2024-09-16T10:36:41.795119+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20461","last_scrub_stamp":"2024-09-16T10:36:41.795101+0000","last_deep_scrub":"48'20461","last_deep_scrub_stamp":"2024-09-16T10:36:41.795101+0000","last_clean_scrub_stamp":"2024-09-16T10:36:41.795101+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:40:17.880798+0000","scrub_duration":0.0096894609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":95,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:23.104711+0000","last_change":"2024-09-16T10:36:23.104850+0000","last_active":"2024-09-16T10:36:23.104711+0000","last_peered":"2024-09-16T10:36:23.104711+0000","last_clean":"2024-09-16T10:36:23.104711+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:23.104711+0000","last_undegraded":"2024-09-16T10:36:23.104711+0000","last_fullsized":"2024-09-16T10:36:23.104711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:23.104689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:23.104689+0000","last_clean_scrub_stamp":"2024-09-16T10:36:23.104689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:11:16.453684+0000","scrub_duration":0.0015296260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30051,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:22.574110+0000","last_change":"2024-09-16T10:36:22.574220+0000","last_active":"2024-09-16T10:36:22.574110+0000","last_peered":"2024-09-16T10:36:22.574110+0000","last_clean":"2024-09-16T10:36:22.574110+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:36:22.574110+0000","last_undegraded":"2024-09-16T10:36:22.574110+0000","last_fullsized":"2024-09-16T10:36:22.574110+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'21040","last_scrub_stamp":"2024-09-16T10:36:22.574085+0000","last_deep_scrub":"48'21040","last_deep_scrub_stamp":"2024-09-16T10:36:22.574085+0000","last_clean_scrub_stamp":"2024-09-16T10:36:22.574085+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:41:07.518035+0000","scrub_duration":0.012370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"66'3","reported_seq":127,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:24.553161+0000","last_change":"2024-09-16T10:36:24.553260+0000","last_active":"2024-09-16T10:36:24.553161+0000","last_peered":"2024-09-16T10:36:24.553161+0000","last_clean":"2024-09-16T10:36:24.553161+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:36:24.553161+0000","last_undegraded":"2024-09-16T10:36:24.553161+0000","last_fullsized":"2024-09-16T10:36:24.553161+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'3","last_scrub_stamp":"2024-09-16T10:36:24.553137+0000","last_deep_scrub":"66'3","last_deep_scrub_stamp":"2024-09-16T10:36:24.553137+0000","last_clean_scrub_stamp":"2024-09-16T10:36:24.553137+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:09:02.621610+0000","scrub_duration":0.0030244619999999999,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29159,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:41.010626+0000","last_change":"2024-09-16T10:36:41.010733+0000","last_active":"2024-09-16T10:36:41.010626+0000","last_peered":"2024-09-16T10:36:41.010626+0000","last_clean":"2024-09-16T10:36:41.010626+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:41.010626+0000","last_undegraded":"2024-09-16T10:36:41.010626+0000","last_fullsized":"2024-09-16T10:36:41.010626+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:41.010603+0000","last_deep_scrub":"48'20419","last_deep_scrub_stamp":"2024-09-16T10:36:41.010603+0000","last_clean_scrub_stamp":"2024-09-16T10:36:41.010603+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:05.816005+0000","scrub_duration":0.010658094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"66'2","reported_seq":108,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:27.777416+0000","last_change":"2024-09-16T10:36:27.777416+0000","last_active":"2024-09-16T10:36:27.777416+0000","last_peered":"2024-09-16T10:36:27.777416+0000","last_clean":"2024-09-16T10:36:27.777416+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:36:27.777416+0000","last_undegraded":"2024-09-16T10:36:27.777416+0000","last_fullsized":"2024-09-16T10:36:27.777416+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-16T10:36:27.777295+0000","last_deep_scrub":"66'2","last_deep_scrub_stamp":"2024-09-16T10:36:27.777295+0000","last_clean_scrub_stamp":"2024-09-16T10:36:27.777295+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:39.906763+0000","scrub_duration":0.002714008,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29126,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:42.967256+0000","last_change":"2024-09-16T10:36:42.967256+0000","last_active":"2024-09-16T10:36:42.967256+0000","last_peered":"2024-09-16T10:36:42.967256+0000","last_clean":"2024-09-16T10:36:42.967256+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:42.967256+0000","last_undegraded":"2024-09-16T10:36:42.967256+0000","last_fullsized":"2024-09-16T10:36:42.967256+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20576","last_scrub_stamp":"2024-09-16T10:36:42.967118+0000","last_deep_scrub":"48'20576","last_deep_scrub_stamp":"2024-09-16T10:36:42.967118+0000","last_clean_scrub_stamp":"2024-09-16T10:36:42.967118+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:47.576458+0000","scrub_duration":0.010541129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20314,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:56.744674+0000","last_change":"2024-09-16T10:36:56.744801+0000","last_active":"2024-09-16T10:36:56.744674+0000","last_peered":"2024-09-16T10:36:56.744674+0000","last_clean":"2024-09-16T10:36:56.744674+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:36:56.744674+0000","last_undegraded":"2024-09-16T10:36:56.744674+0000","last_fullsized":"2024-09-16T10:36:56.744674+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20034","last_scrub_stamp":"2024-09-16T10:36:56.744656+0000","last_deep_scrub":"48'20034","last_deep_scrub_stamp":"2024-09-16T10:36:56.744656+0000","last_clean_scrub_stamp":"2024-09-16T10:36:56.744656+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:56.822643+0000","scrub_duration":0.0097261210000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":93,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:26.825748+0000","last_change":"2024-09-16T10:36:26.825748+0000","last_active":"2024-09-16T10:36:26.825748+0000","last_peered":"2024-09-16T10:36:26.825748+0000","last_clean":"2024-09-16T10:36:26.825748+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:36:26.825748+0000","last_undegraded":"2024-09-16T10:36:26.825748+0000","last_fullsized":"2024-09-16T10:36:26.825748+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:26.825614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:26.825614+0000","last_clean_scrub_stamp":"2024-09-16T10:36:26.825614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:13.087372+0000","scrub_duration":0.0027595470000000002,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21238,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:50.725175+0000","last_change":"2024-09-16T10:36:50.725295+0000","last_active":"2024-09-16T10:36:50.725175+0000","last_peered":"2024-09-16T10:36:50.725175+0000","last_clean":"2024-09-16T10:36:50.725175+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:36:50.725175+0000","last_undegraded":"2024-09-16T10:36:50.725175+0000","last_fullsized":"2024-09-16T10:36:50.725175+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20914","last_scrub_stamp":"2024-09-16T10:36:50.725154+0000","last_deep_scrub":"48'20914","last_deep_scrub_stamp":"2024-09-16T10:36:50.725154+0000","last_clean_scrub_stamp":"2024-09-16T10:36:50.725154+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:16.905443+0000","scrub_duration":0.011013719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":95,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:34.130199+0000","last_change":"2024-09-16T10:36:34.130521+0000","last_active":"2024-09-16T10:36:34.130199+0000","last_peered":"2024-09-16T10:36:34.130199+0000","last_clean":"2024-09-16T10:36:34.130199+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:34.130199+0000","last_undegraded":"2024-09-16T10:36:34.130199+0000","last_fullsized":"2024-09-16T10:36:34.130199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:34.130176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:34.130176+0000","last_clean_scrub_stamp":"2024-09-16T10:36:34.130176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:15.595794+0000","scrub_duration":0.0016726289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24546,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:47.687082+0000","last_change":"2024-09-16T10:36:47.687201+0000","last_active":"2024-09-16T10:36:47.687082+0000","last_peered":"2024-09-16T10:36:47.687082+0000","last_clean":"2024-09-16T10:36:47.687082+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:36:47.687082+0000","last_undegraded":"2024-09-16T10:36:47.687082+0000","last_fullsized":"2024-09-16T10:36:47.687082+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20140","last_scrub_stamp":"2024-09-16T10:36:47.687058+0000","last_deep_scrub":"48'20140","last_deep_scrub_stamp":"2024-09-16T10:36:47.687058+0000","last_clean_scrub_stamp":"2024-09-16T10:36:47.687058+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:27.302153+0000","scrub_duration":0.0099877669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":93,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:28.801098+0000","last_change":"2024-09-16T10:36:28.801098+0000","last_active":"2024-09-16T10:36:28.801098+0000","last_peered":"2024-09-16T10:36:28.801098+0000","last_clean":"2024-09-16T10:36:28.801098+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:36:28.801098+0000","last_undegraded":"2024-09-16T10:36:28.801098+0000","last_fullsized":"2024-09-16T10:36:28.801098+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:28.800958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:28.800958+0000","last_clean_scrub_stamp":"2024-09-16T10:36:28.800958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:09.569886+0000","scrub_duration":0.001647702,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":102,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:24.742948+0000","last_change":"2024-09-16T10:36:24.742948+0000","last_active":"2024-09-16T10:36:24.742948+0000","last_peered":"2024-09-16T10:36:24.742948+0000","last_clean":"2024-09-16T10:36:24.742948+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:36:24.742948+0000","last_undegraded":"2024-09-16T10:36:24.742948+0000","last_fullsized":"2024-09-16T10:36:24.742948+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:24.742833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:24.742833+0000","last_clean_scrub_stamp":"2024-09-16T10:36:24.742833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:21.769923+0000","scrub_duration":0.0015081770000000001,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"66'20897","reported_seq":21173,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:52.717145+0000","last_change":"2024-09-16T10:36:52.717267+0000","last_active":"2024-09-16T10:36:52.717145+0000","last_peered":"2024-09-16T10:36:52.717145+0000","last_clean":"2024-09-16T10:36:52.717145+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:36:52.717145+0000","last_undegraded":"2024-09-16T10:36:52.717145+0000","last_fullsized":"2024-09-16T10:36:52.717145+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'20897","last_scrub_stamp":"2024-09-16T10:36:52.717126+0000","last_deep_scrub":"66'20897","last_deep_scrub_stamp":"2024-09-16T10:36:52.717126+0000","last_clean_scrub_stamp":"2024-09-16T10:36:52.717126+0000","objects_scrubbed":1,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:07:59.941047+0000","scrub_duration":0.017388154999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141824,"num_write_kb":238869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17152,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019054,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347168,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024672,"statfs":{"total":96636764160,"available":96281264128,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.22,"5min":1.3979999999999999,"15min":1.5369999999999999},"min":{"1min":0.17899999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.34999999999999998,"5min":13.170999999999999,"15min":13.74},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.23699999999999999,"5min":1.153,"15min":1.649},"min":{"1min":0.20599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.26900000000000002,"5min":12.265000000000001,"15min":13.917},"last":0.33700000000000002}]},{"osd":1,"last update":"Mon Sep 16 10:36:12 2024","interfaces":[{"interface":"back","average":{"1min":0.25700000000000001,"5min":1.5700000000000001,"15min":2.4990000000000001},"min":{"1min":0.20699999999999999,"5min":0.189,"15min":0.189},"max":{"1min":0.35799999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.26300000000000001,"5min":1.593,"15min":2.4460000000000002},"min":{"1min":0.186,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.376,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.371}]}]},{"osd":1,"up_from":40,"seq":171798691998,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505512,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866328,"statfs":{"total":96636764160,"available":96119119872,"internally_reserved":0,"allocated":557056,"data_stored":185777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.19700000000000001,"5min":1.5860000000000001,"15min":2.8109999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.27300000000000002,"5min":12.798,"15min":17.675000000000001},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.23400000000000001,"5min":1.8220000000000001,"15min":2.484},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.318,"5min":15.151,"15min":16.850999999999999},"last":0.41599999999999998}]},{"osd":2,"last update":"Mon Sep 16 10:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.24099999999999999,"5min":1.587,"15min":2.2160000000000002},"min":{"1min":0.19400000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.33900000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.27100000000000002,"5min":1.5009999999999999,"15min":2.391},"min":{"1min":0.219,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.34999999999999998,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.438}]}]},{"osd":0,"up_from":24,"seq":103079215281,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455396,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916444,"statfs":{"total":96636764160,"available":96170438656,"internally_reserved":0,"allocated":552960,"data_stored":185295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.215,"5min":1.335,"15min":2.2639999999999998},"min":{"1min":0.157,"5min":0.157,"15min":0.157},"max":{"1min":0.40699999999999997,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.23100000000000001,"5min":1.466,"15min":2.0339999999999998},"min":{"1min":0.17399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.433,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.26800000000000002,"5min":1.427,"15min":1.8700000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.16800000000000001},"max":{"1min":0.56200000000000006,"5min":15.534000000000001,"15min":18.093},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.27300000000000002,"5min":1.2749999999999999,"15min":1.7929999999999999},"min":{"1min":0.21199999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.53200000000000003,"5min":11.863,"15min":13.752000000000001},"last":0.38300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:37:02.018 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-09-16T10:23:43.952628+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=23, tm_sec=43, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=10, tm_min=36, tm_sec=18, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T10:37:02.019 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T10:37:22.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T10:37:22.317 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-09-16T10:37:22.317 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-09-16T10:37:22.329 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-16T10:37:20.803640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":502,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141826,"num_write_kb":238871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17153,"num_bytes_recovered":17170451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160828,"ondisk_log_size":160828,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1308076,"kb_used_data":1620,"kb_used_omap":12,"kb_used_meta":1306419,"kb_avail":281807444,"statfs":{"total":289910292480,"available":288570822656,"internally_reserved":0,"allocated":1658880,"data_stored":556347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13310,"internal_metadata":1337773058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002374"},"pg_stats":[{"pgid":"2.f","version":"48'20468","reported_seq":28968,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:27.592739+0000","last_change":"2024-09-16T10:36:27.592893+0000","last_active":"2024-09-16T10:36:27.592739+0000","last_peered":"2024-09-16T10:36:27.592739+0000","last_clean":"2024-09-16T10:36:27.592739+0000","last_became_active":"2024-09-16T10:29:03.975301+0000","last_became_peered":"2024-09-16T10:29:03.975301+0000","last_unstale":"2024-09-16T10:36:27.592739+0000","last_undegraded":"2024-09-16T10:36:27.592739+0000","last_fullsized":"2024-09-16T10:36:27.592739+0000","mapping_epoch":45,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20468","last_scrub_stamp":"2024-09-16T10:36:27.592717+0000","last_deep_scrub":"48'20468","last_deep_scrub_stamp":"2024-09-16T10:36:27.592717+0000","last_clean_scrub_stamp":"2024-09-16T10:36:27.592717+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:36:44.024728+0000","scrub_duration":0.011921997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20468,"num_write_kb":40936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4192,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"48'20462","reported_seq":20658,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:30.159394+0000","last_change":"2024-09-16T10:36:30.159394+0000","last_active":"2024-09-16T10:36:30.159394+0000","last_peered":"2024-09-16T10:36:30.159394+0000","last_clean":"2024-09-16T10:36:30.159394+0000","last_became_active":"2024-09-16T10:23:02.013939+0000","last_became_peered":"2024-09-16T10:23:02.013939+0000","last_unstale":"2024-09-16T10:36:30.159394+0000","last_undegraded":"2024-09-16T10:36:30.159394+0000","last_fullsized":"2024-09-16T10:36:30.159394+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20462","last_scrub_stamp":"2024-09-16T10:36:30.159254+0000","last_deep_scrub":"48'20462","last_deep_scrub_stamp":"2024-09-16T10:36:30.159254+0000","last_clean_scrub_stamp":"2024-09-16T10:36:30.159254+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:59:31.876273+0000","scrub_duration":0.013664887000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16675,"num_write_kb":25788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"48'20664","reported_seq":20960,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:48.694232+0000","last_change":"2024-09-16T10:36:48.694373+0000","last_active":"2024-09-16T10:36:48.694232+0000","last_peered":"2024-09-16T10:36:48.694232+0000","last_clean":"2024-09-16T10:36:48.694232+0000","last_became_active":"2024-09-16T10:34:02.110857+0000","last_became_peered":"2024-09-16T10:34:02.110857+0000","last_unstale":"2024-09-16T10:36:48.694232+0000","last_undegraded":"2024-09-16T10:36:48.694232+0000","last_fullsized":"2024-09-16T10:36:48.694232+0000","mapping_epoch":51,"log_start":"44'10600","ondisk_log_start":"44'10600","created":15,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20664","last_scrub_stamp":"2024-09-16T10:36:48.694209+0000","last_deep_scrub":"48'20664","last_deep_scrub_stamp":"2024-09-16T10:36:48.694209+0000","last_clean_scrub_stamp":"2024-09-16T10:36:48.694209+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T22:13:40.564100+0000","scrub_duration":0.0098038299999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16721,"num_write_kb":25561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4336,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"48'20462","reported_seq":20557,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:36.085979+0000","last_change":"2024-09-16T10:36:36.086122+0000","last_active":"2024-09-16T10:36:36.085979+0000","last_peered":"2024-09-16T10:36:36.085979+0000","last_clean":"2024-09-16T10:36:36.085979+0000","last_became_active":"2024-09-16T10:23:02.014675+0000","last_became_peered":"2024-09-16T10:23:02.014675+0000","last_unstale":"2024-09-16T10:36:36.085979+0000","last_undegraded":"2024-09-16T10:36:36.085979+0000","last_fullsized":"2024-09-16T10:36:36.085979+0000","mapping_epoch":34,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20462","last_scrub_stamp":"2024-09-16T10:36:36.085953+0000","last_deep_scrub":"48'20462","last_deep_scrub_stamp":"2024-09-16T10:36:36.085953+0000","last_clean_scrub_stamp":"2024-09-16T10:36:36.085953+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:49:25.173219+0000","scrub_duration":0.011643202,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20462,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.b","version":"48'20505","reported_seq":20700,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:26.601145+0000","last_change":"2024-09-16T10:36:26.601316+0000","last_active":"2024-09-16T10:36:26.601145+0000","last_peered":"2024-09-16T10:36:26.601145+0000","last_clean":"2024-09-16T10:36:26.601145+0000","last_became_active":"2024-09-16T10:23:01.016074+0000","last_became_peered":"2024-09-16T10:23:01.016074+0000","last_unstale":"2024-09-16T10:36:26.601145+0000","last_undegraded":"2024-09-16T10:36:26.601145+0000","last_fullsized":"2024-09-16T10:36:26.601145+0000","mapping_epoch":33,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20505","last_scrub_stamp":"2024-09-16T10:36:26.601118+0000","last_deep_scrub":"48'20505","last_deep_scrub_stamp":"2024-09-16T10:36:26.601118+0000","last_clean_scrub_stamp":"2024-09-16T10:36:26.601118+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:21.213545+0000","scrub_duration":0.011907403,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17147,"num_write_kb":27592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.a","version":"48'20371","reported_seq":20739,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:49.676994+0000","last_change":"2024-09-16T10:36:49.677110+0000","last_active":"2024-09-16T10:36:49.676994+0000","last_peered":"2024-09-16T10:36:49.676994+0000","last_clean":"2024-09-16T10:36:49.676994+0000","last_became_active":"2024-09-16T10:34:18.205595+0000","last_became_peered":"2024-09-16T10:34:18.205595+0000","last_unstale":"2024-09-16T10:36:49.676994+0000","last_undegraded":"2024-09-16T10:36:49.676994+0000","last_fullsized":"2024-09-16T10:36:49.676994+0000","mapping_epoch":53,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20371","last_scrub_stamp":"2024-09-16T10:36:49.676970+0000","last_deep_scrub":"48'20371","last_deep_scrub_stamp":"2024-09-16T10:36:49.676970+0000","last_clean_scrub_stamp":"2024-09-16T10:36:49.676970+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:24:09.853188+0000","scrub_duration":0.0099085990000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16996,"num_write_kb":27252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"48'20920","reported_seq":21351,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:58.727590+0000","last_change":"2024-09-16T10:36:58.727704+0000","last_active":"2024-09-16T10:36:58.727590+0000","last_peered":"2024-09-16T10:36:58.727590+0000","last_clean":"2024-09-16T10:36:58.727590+0000","last_became_active":"2024-09-16T10:36:10.324803+0000","last_became_peered":"2024-09-16T10:36:10.324803+0000","last_unstale":"2024-09-16T10:36:58.727590+0000","last_undegraded":"2024-09-16T10:36:58.727590+0000","last_fullsized":"2024-09-16T10:36:58.727590+0000","mapping_epoch":65,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20920","last_scrub_stamp":"2024-09-16T10:36:58.727569+0000","last_deep_scrub":"48'20920","last_deep_scrub_stamp":"2024-09-16T10:36:58.727569+0000","last_clean_scrub_stamp":"2024-09-16T10:36:58.727569+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:22:54.137974+0000","scrub_duration":0.010808353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17162,"num_write_kb":26820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4405,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"48'20390","reported_seq":20604,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:25.599099+0000","last_change":"2024-09-16T10:36:25.599099+0000","last_active":"2024-09-16T10:36:25.599099+0000","last_peered":"2024-09-16T10:36:25.599099+0000","last_clean":"2024-09-16T10:36:25.599099+0000","last_became_active":"2024-09-16T10:23:01.015829+0000","last_became_peered":"2024-09-16T10:23:01.015829+0000","last_unstale":"2024-09-16T10:36:25.599099+0000","last_undegraded":"2024-09-16T10:36:25.599099+0000","last_fullsized":"2024-09-16T10:36:25.599099+0000","mapping_epoch":33,"log_start":"44'10300","ondisk_log_start":"44'10300","created":15,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20390","last_scrub_stamp":"2024-09-16T10:36:25.598429+0000","last_deep_scrub":"48'20390","last_deep_scrub_stamp":"2024-09-16T10:36:25.598429+0000","last_clean_scrub_stamp":"2024-09-16T10:36:25.598429+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:56:42.248280+0000","scrub_duration":0.01328883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16193,"num_write_kb":23996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":93,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:21.754457+0000","last_change":"2024-09-16T10:36:21.754550+0000","last_active":"2024-09-16T10:36:21.754457+0000","last_peered":"2024-09-16T10:36:21.754457+0000","last_clean":"2024-09-16T10:36:21.754457+0000","last_became_active":"2024-09-16T10:23:56.373730+0000","last_became_peered":"2024-09-16T10:23:56.373730+0000","last_unstale":"2024-09-16T10:36:21.754457+0000","last_undegraded":"2024-09-16T10:36:21.754457+0000","last_fullsized":"2024-09-16T10:36:21.754457+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:21.754437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:21.754437+0000","last_clean_scrub_stamp":"2024-09-16T10:36:21.754437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:37:53.049486+0000","scrub_duration":0.0014528500000000001,"objects_trimmed":0,"snaptrim_duration":0.00089507699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"48'20461","reported_seq":29015,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:41.795119+0000","last_change":"2024-09-16T10:36:41.795203+0000","last_active":"2024-09-16T10:36:41.795119+0000","last_peered":"2024-09-16T10:36:41.795119+0000","last_clean":"2024-09-16T10:36:41.795119+0000","last_became_active":"2024-09-16T10:26:01.952689+0000","last_became_peered":"2024-09-16T10:26:01.952689+0000","last_unstale":"2024-09-16T10:36:41.795119+0000","last_undegraded":"2024-09-16T10:36:41.795119+0000","last_fullsized":"2024-09-16T10:36:41.795119+0000","mapping_epoch":43,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20461","last_scrub_stamp":"2024-09-16T10:36:41.795101+0000","last_deep_scrub":"48'20461","last_deep_scrub_stamp":"2024-09-16T10:36:41.795101+0000","last_clean_scrub_stamp":"2024-09-16T10:36:41.795101+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:40:17.880798+0000","scrub_duration":0.0096894609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":95,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:23.104711+0000","last_change":"2024-09-16T10:36:23.104850+0000","last_active":"2024-09-16T10:36:23.104711+0000","last_peered":"2024-09-16T10:36:23.104711+0000","last_clean":"2024-09-16T10:36:23.104711+0000","last_became_active":"2024-09-16T10:23:02.013667+0000","last_became_peered":"2024-09-16T10:23:02.013667+0000","last_unstale":"2024-09-16T10:36:23.104711+0000","last_undegraded":"2024-09-16T10:36:23.104711+0000","last_fullsized":"2024-09-16T10:36:23.104711+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:23.104689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:23.104689+0000","last_clean_scrub_stamp":"2024-09-16T10:36:23.104689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:11:16.453684+0000","scrub_duration":0.0015296260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"48'21040","reported_seq":30052,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:22.574220+0000","last_change":"2024-09-16T10:36:22.574220+0000","last_active":"2024-09-16T10:36:22.574220+0000","last_peered":"2024-09-16T10:36:22.574220+0000","last_clean":"2024-09-16T10:36:22.574220+0000","last_became_active":"2024-09-16T10:35:36.253728+0000","last_became_peered":"2024-09-16T10:35:36.253728+0000","last_unstale":"2024-09-16T10:36:22.574220+0000","last_undegraded":"2024-09-16T10:36:22.574220+0000","last_fullsized":"2024-09-16T10:36:22.574220+0000","mapping_epoch":61,"log_start":"44'11000","ondisk_log_start":"44'11000","created":15,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'21040","last_scrub_stamp":"2024-09-16T10:36:22.574085+0000","last_deep_scrub":"48'21040","last_deep_scrub_stamp":"2024-09-16T10:36:22.574085+0000","last_clean_scrub_stamp":"2024-09-16T10:36:22.574085+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:41:07.518035+0000","scrub_duration":0.012370549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"66'3","reported_seq":128,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:24.553260+0000","last_change":"2024-09-16T10:36:24.553260+0000","last_active":"2024-09-16T10:36:24.553260+0000","last_peered":"2024-09-16T10:36:24.553260+0000","last_clean":"2024-09-16T10:36:24.553260+0000","last_became_active":"2024-09-16T10:23:56.896302+0000","last_became_peered":"2024-09-16T10:23:56.896302+0000","last_unstale":"2024-09-16T10:36:24.553260+0000","last_undegraded":"2024-09-16T10:36:24.553260+0000","last_fullsized":"2024-09-16T10:36:24.553260+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'3","last_scrub_stamp":"2024-09-16T10:36:24.553137+0000","last_deep_scrub":"66'3","last_deep_scrub_stamp":"2024-09-16T10:36:24.553137+0000","last_clean_scrub_stamp":"2024-09-16T10:36:24.553137+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:09:02.621610+0000","scrub_duration":0.0030244619999999999,"objects_trimmed":0,"snaptrim_duration":0.0021320919999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"48'20419","reported_seq":29159,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:41.010626+0000","last_change":"2024-09-16T10:36:41.010733+0000","last_active":"2024-09-16T10:36:41.010626+0000","last_peered":"2024-09-16T10:36:41.010626+0000","last_clean":"2024-09-16T10:36:41.010626+0000","last_became_active":"2024-09-16T10:34:35.234178+0000","last_became_peered":"2024-09-16T10:34:35.234178+0000","last_unstale":"2024-09-16T10:36:41.010626+0000","last_undegraded":"2024-09-16T10:36:41.010626+0000","last_fullsized":"2024-09-16T10:36:41.010626+0000","mapping_epoch":55,"log_start":"44'10400","ondisk_log_start":"44'10400","created":15,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20419","last_scrub_stamp":"2024-09-16T10:36:41.010603+0000","last_deep_scrub":"48'20419","last_deep_scrub_stamp":"2024-09-16T10:36:41.010603+0000","last_clean_scrub_stamp":"2024-09-16T10:36:41.010603+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:05.816005+0000","scrub_duration":0.010658094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"66'2","reported_seq":108,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:27.777416+0000","last_change":"2024-09-16T10:36:27.777416+0000","last_active":"2024-09-16T10:36:27.777416+0000","last_peered":"2024-09-16T10:36:27.777416+0000","last_clean":"2024-09-16T10:36:27.777416+0000","last_became_active":"2024-09-16T10:23:56.434736+0000","last_became_peered":"2024-09-16T10:23:56.434736+0000","last_unstale":"2024-09-16T10:36:27.777416+0000","last_undegraded":"2024-09-16T10:36:27.777416+0000","last_fullsized":"2024-09-16T10:36:27.777416+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-16T10:36:27.777295+0000","last_deep_scrub":"66'2","last_deep_scrub_stamp":"2024-09-16T10:36:27.777295+0000","last_clean_scrub_stamp":"2024-09-16T10:36:27.777295+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:21:39.906763+0000","scrub_duration":0.002714008,"objects_trimmed":0,"snaptrim_duration":0.000462664,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"48'20576","reported_seq":29126,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:42.967256+0000","last_change":"2024-09-16T10:36:42.967256+0000","last_active":"2024-09-16T10:36:42.967256+0000","last_peered":"2024-09-16T10:36:42.967256+0000","last_clean":"2024-09-16T10:36:42.967256+0000","last_became_active":"2024-09-16T10:31:29.304076+0000","last_became_peered":"2024-09-16T10:31:29.304076+0000","last_unstale":"2024-09-16T10:36:42.967256+0000","last_undegraded":"2024-09-16T10:36:42.967256+0000","last_fullsized":"2024-09-16T10:36:42.967256+0000","mapping_epoch":47,"log_start":"44'10500","ondisk_log_start":"44'10500","created":15,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20576","last_scrub_stamp":"2024-09-16T10:36:42.967118+0000","last_deep_scrub":"48'20576","last_deep_scrub_stamp":"2024-09-16T10:36:42.967118+0000","last_clean_scrub_stamp":"2024-09-16T10:36:42.967118+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:30:47.576458+0000","scrub_duration":0.010541129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"48'20034","reported_seq":20314,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:56.744674+0000","last_change":"2024-09-16T10:36:56.744801+0000","last_active":"2024-09-16T10:36:56.744674+0000","last_peered":"2024-09-16T10:36:56.744674+0000","last_clean":"2024-09-16T10:36:56.744674+0000","last_became_active":"2024-09-16T10:35:53.294244+0000","last_became_peered":"2024-09-16T10:35:53.294244+0000","last_unstale":"2024-09-16T10:36:56.744674+0000","last_undegraded":"2024-09-16T10:36:56.744674+0000","last_fullsized":"2024-09-16T10:36:56.744674+0000","mapping_epoch":63,"log_start":"44'10000","ondisk_log_start":"44'10000","created":15,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20034","last_scrub_stamp":"2024-09-16T10:36:56.744656+0000","last_deep_scrub":"48'20034","last_deep_scrub_stamp":"2024-09-16T10:36:56.744656+0000","last_clean_scrub_stamp":"2024-09-16T10:36:56.744656+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:57:56.822643+0000","scrub_duration":0.0097261210000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":93,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:26.825748+0000","last_change":"2024-09-16T10:36:26.825748+0000","last_active":"2024-09-16T10:36:26.825748+0000","last_peered":"2024-09-16T10:36:26.825748+0000","last_clean":"2024-09-16T10:36:26.825748+0000","last_became_active":"2024-09-16T10:23:56.549570+0000","last_became_peered":"2024-09-16T10:23:56.549570+0000","last_unstale":"2024-09-16T10:36:26.825748+0000","last_undegraded":"2024-09-16T10:36:26.825748+0000","last_fullsized":"2024-09-16T10:36:26.825748+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:26.825614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:26.825614+0000","last_clean_scrub_stamp":"2024-09-16T10:36:26.825614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:35:13.087372+0000","scrub_duration":0.0027595470000000002,"objects_trimmed":0,"snaptrim_duration":0.00042772699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"48'20914","reported_seq":21239,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:50.725295+0000","last_change":"2024-09-16T10:36:50.725295+0000","last_active":"2024-09-16T10:36:50.725295+0000","last_peered":"2024-09-16T10:36:50.725295+0000","last_clean":"2024-09-16T10:36:50.725295+0000","last_became_active":"2024-09-16T10:34:52.119897+0000","last_became_peered":"2024-09-16T10:34:52.119897+0000","last_unstale":"2024-09-16T10:36:50.725295+0000","last_undegraded":"2024-09-16T10:36:50.725295+0000","last_fullsized":"2024-09-16T10:36:50.725295+0000","mapping_epoch":57,"log_start":"44'10900","ondisk_log_start":"44'10900","created":15,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20914","last_scrub_stamp":"2024-09-16T10:36:50.725154+0000","last_deep_scrub":"48'20914","last_deep_scrub_stamp":"2024-09-16T10:36:50.725154+0000","last_clean_scrub_stamp":"2024-09-16T10:36:50.725154+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:38:16.905443+0000","scrub_duration":0.011013719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":95,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:34.130199+0000","last_change":"2024-09-16T10:36:34.130521+0000","last_active":"2024-09-16T10:36:34.130199+0000","last_peered":"2024-09-16T10:36:34.130199+0000","last_clean":"2024-09-16T10:36:34.130199+0000","last_became_active":"2024-09-16T10:23:02.013555+0000","last_became_peered":"2024-09-16T10:23:02.013555+0000","last_unstale":"2024-09-16T10:36:34.130199+0000","last_undegraded":"2024-09-16T10:36:34.130199+0000","last_fullsized":"2024-09-16T10:36:34.130199+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:34.130176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:34.130176+0000","last_clean_scrub_stamp":"2024-09-16T10:36:34.130176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:15:15.595794+0000","scrub_duration":0.0016726289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"48'20140","reported_seq":24546,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:47.687082+0000","last_change":"2024-09-16T10:36:47.687201+0000","last_active":"2024-09-16T10:36:47.687082+0000","last_peered":"2024-09-16T10:36:47.687082+0000","last_clean":"2024-09-16T10:36:47.687082+0000","last_became_active":"2024-09-16T10:33:36.905176+0000","last_became_peered":"2024-09-16T10:33:36.905176+0000","last_unstale":"2024-09-16T10:36:47.687082+0000","last_undegraded":"2024-09-16T10:36:47.687082+0000","last_fullsized":"2024-09-16T10:36:47.687082+0000","mapping_epoch":49,"log_start":"44'10100","ondisk_log_start":"44'10100","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"48'20140","last_scrub_stamp":"2024-09-16T10:36:47.687058+0000","last_deep_scrub":"48'20140","last_deep_scrub_stamp":"2024-09-16T10:36:47.687058+0000","last_clean_scrub_stamp":"2024-09-16T10:36:47.687058+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:08:27.302153+0000","scrub_duration":0.0099877669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":93,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:28.801098+0000","last_change":"2024-09-16T10:36:28.801098+0000","last_active":"2024-09-16T10:36:28.801098+0000","last_peered":"2024-09-16T10:36:28.801098+0000","last_clean":"2024-09-16T10:36:28.801098+0000","last_became_active":"2024-09-16T10:23:56.548975+0000","last_became_peered":"2024-09-16T10:23:56.548975+0000","last_unstale":"2024-09-16T10:36:28.801098+0000","last_undegraded":"2024-09-16T10:36:28.801098+0000","last_fullsized":"2024-09-16T10:36:28.801098+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:28.800958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:28.800958+0000","last_clean_scrub_stamp":"2024-09-16T10:36:28.800958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:09.569886+0000","scrub_duration":0.001647702,"objects_trimmed":0,"snaptrim_duration":0.00045944900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":102,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:24.742948+0000","last_change":"2024-09-16T10:36:24.742948+0000","last_active":"2024-09-16T10:36:24.742948+0000","last_peered":"2024-09-16T10:36:24.742948+0000","last_clean":"2024-09-16T10:36:24.742948+0000","last_became_active":"2024-09-16T10:23:56.369103+0000","last_became_peered":"2024-09-16T10:23:56.369103+0000","last_unstale":"2024-09-16T10:36:24.742948+0000","last_undegraded":"2024-09-16T10:36:24.742948+0000","last_fullsized":"2024-09-16T10:36:24.742948+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T10:36:24.742833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T10:36:24.742833+0000","last_clean_scrub_stamp":"2024-09-16T10:36:24.742833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:21.769923+0000","scrub_duration":0.0015081770000000001,"objects_trimmed":0,"snaptrim_duration":0.00092611699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"66'20897","reported_seq":21174,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-16T10:36:52.717267+0000","last_change":"2024-09-16T10:36:52.717267+0000","last_active":"2024-09-16T10:36:52.717267+0000","last_peered":"2024-09-16T10:36:52.717267+0000","last_clean":"2024-09-16T10:36:52.717267+0000","last_became_active":"2024-09-16T10:35:13.158587+0000","last_became_peered":"2024-09-16T10:35:13.158587+0000","last_unstale":"2024-09-16T10:36:52.717267+0000","last_undegraded":"2024-09-16T10:36:52.717267+0000","last_fullsized":"2024-09-16T10:36:52.717267+0000","mapping_epoch":59,"log_start":"44'10800","ondisk_log_start":"44'10800","created":15,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'20897","last_scrub_stamp":"2024-09-16T10:36:52.717126+0000","last_deep_scrub":"66'20897","last_deep_scrub_stamp":"2024-09-16T10:36:52.717126+0000","last_clean_scrub_stamp":"2024-09-16T10:36:52.717126+0000","objects_scrubbed":1,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:07:59.941047+0000","scrub_duration":0.017388154999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":483,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141824,"num_write_kb":238869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17152,"num_bytes_recovered":17170432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160823,"ondisk_log_size":160823,"up":32,"acting":32,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":30,"seq":128849019058,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":347168,"kb_used_data":536,"kb_used_omap":4,"kb_used_meta":346619,"kb_avail":94024672,"statfs":{"total":96636764160,"available":96281264128,"internally_reserved":0,"allocated":548864,"data_stored":185275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":354938544},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.88700000000000001,"15min":1.502},"min":{"1min":0.22500000000000001,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.42999999999999999,"5min":11.718999999999999,"15min":13.74},"last":0.254},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.84499999999999997,"15min":1.611},"min":{"1min":0.23599999999999999,"5min":0.188,"15min":0.17899999999999999},"max":{"1min":0.50800000000000001,"5min":12.265000000000001,"15min":13.917},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Sep 16 10:37:13 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":1.177,"15min":2.3290000000000002},"min":{"1min":0.222,"5min":0.189,"15min":0.189},"max":{"1min":0.48199999999999998,"5min":19.518999999999998,"15min":19.518999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.35999999999999999,"5min":1.083,"15min":2.3199999999999998},"min":{"1min":0.26700000000000002,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.47799999999999998,"5min":11.766999999999999,"15min":17.914000000000001},"last":0.308}]}]},{"osd":1,"up_from":40,"seq":171798692002,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":505512,"kb_used_data":544,"kb_used_omap":4,"kb_used_meta":504955,"kb_avail":93866328,"statfs":{"total":96636764160,"available":96119119872,"internally_reserved":0,"allocated":557056,"data_stored":185777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4438,"internal_metadata":517074602},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 10:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.19700000000000001,"5min":1.5860000000000001,"15min":2.8109999999999999},"min":{"1min":0.14899999999999999,"5min":0.14899999999999999,"15min":0.14899999999999999},"max":{"1min":0.27300000000000002,"5min":12.798,"15min":17.675000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.23400000000000001,"5min":1.8220000000000001,"15min":2.484},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.318,"5min":15.151,"15min":16.850999999999999},"last":0.42699999999999999}]},{"osd":2,"last update":"Mon Sep 16 10:36:22 2024","interfaces":[{"interface":"back","average":{"1min":0.24099999999999999,"5min":1.587,"15min":2.2160000000000002},"min":{"1min":0.19400000000000001,"5min":0.189,"15min":0.189},"max":{"1min":0.33900000000000002,"5min":14.089,"15min":18.638999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.27100000000000002,"5min":1.5009999999999999,"15min":2.391},"min":{"1min":0.219,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.34999999999999998,"5min":10.656000000000001,"15min":11.412000000000001},"last":0.48999999999999999}]}]},{"osd":0,"up_from":24,"seq":103079215285,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455396,"kb_used_data":540,"kb_used_omap":4,"kb_used_meta":454843,"kb_avail":93916444,"statfs":{"total":96636764160,"available":96170438656,"internally_reserved":0,"allocated":552960,"data_stored":185295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4440,"internal_metadata":465759912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 10:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.215,"5min":1.335,"15min":2.2639999999999998},"min":{"1min":0.157,"5min":0.157,"15min":0.157},"max":{"1min":0.40699999999999997,"5min":16.806000000000001,"15min":16.806000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.23100000000000001,"5min":1.466,"15min":2.0339999999999998},"min":{"1min":0.17399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.433,"5min":16.760999999999999,"15min":16.760999999999999},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Sep 16 10:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.26800000000000002,"5min":1.427,"15min":1.8700000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.16800000000000001},"max":{"1min":0.56200000000000006,"5min":15.534000000000001,"15min":18.093},"last":0.376},{"interface":"front","average":{"1min":0.27300000000000002,"5min":1.2749999999999999,"15min":1.7929999999999999},"min":{"1min":0.21199999999999999,"5min":0.186,"15min":0.186},"max":{"1min":0.53200000000000003,"5min":11.863,"15min":13.752000000000001},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T10:37:22.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T10:37:22.630 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T10:37:22.630 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T10:37:22.630 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T10:37:22.630 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:37:27.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-16T10:37:28.732 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T10:37:28.733 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T10:37:28.733 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:37:28.808 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T10:37:28.808 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T10:37:28.808 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:37:32.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-16T10:37:32.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-16T10:37:34.910 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T10:37:34.910 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T10:37:34.910 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T10:37:34.910 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:37:34.942 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T10:37:34.942 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T10:37:34.942 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T10:37:34.942 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:37:34.998 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T10:37:34.998 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T10:37:34.998 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:37:35.052 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T10:37:35.053 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T10:37:35.053 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T10:37:35.107 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T10:37:35.108 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T10:37:35.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-09-16T10:37:35.209 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-16T10:37:35.210 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T10:37:35.799 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-16T10:37:35.799 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T10:37:35.921 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi050.front.sepia.ceph.com 2024-09-16T10:37:35.921 DEBUG:teuthology.orchestra.run.smithi050:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T10:37:35.998 INFO:tasks.ceph:Archiving mon data... 2024-09-16T10:37:35.999 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906637/data/mon.a.tgz 2024-09-16T10:37:36.000 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-09-16T10:37:36.019 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.z7W2xYJssX 2024-09-16T10:37:36.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.z7W2xYJssX 2024-09-16T10:37:36.179 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.z7W2xYJssX 2024-09-16T10:37:36.277 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.z7W2xYJssX is 283KB 2024-09-16T10:37:36.373 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.z7W2xYJssX 2024-09-16T10:37:36.389 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906637/data/mon.b.tgz 2024-09-16T10:37:36.389 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-09-16T10:37:36.444 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.0gkAkLGEBs 2024-09-16T10:37:36.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.0gkAkLGEBs 2024-09-16T10:37:36.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.0gkAkLGEBs 2024-09-16T10:37:36.647 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.0gkAkLGEBs is 285KB 2024-09-16T10:37:36.742 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.0gkAkLGEBs 2024-09-16T10:37:36.758 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906637/data/mon.c.tgz 2024-09-16T10:37:36.758 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-09-16T10:37:36.813 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.xKH3H13v4i 2024-09-16T10:37:36.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.xKH3H13v4i 2024-09-16T10:37:36.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.xKH3H13v4i 2024-09-16T10:37:37.013 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.xKH3H13v4i is 285KB 2024-09-16T10:37:37.080 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.xKH3H13v4i 2024-09-16T10:37:37.096 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T10:37:37.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T10:37:37.161 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T10:37:37.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T10:37:37.377 INFO:teuthology.orchestra.run.smithi050.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T10:37:37.382 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T10:37:37.384 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906637/remote/smithi050/crash 2024-09-16T10:37:37.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T10:37:37.410 INFO:tasks.ceph:Compressing logs... 2024-09-16T10:37:37.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T10:37:37.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-16T10:45:29.763 INFO:tasks.ceph:Archiving logs... 2024-09-16T10:45:29.765 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906637/remote/smithi050/log 2024-09-16T10:45:29.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T10:46:32.274 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T10:46:32.288 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T10:46:32.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T10:46:32.386 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T10:46:32.399 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T10:46:32.399 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:46:32.418 INFO:teuthology.orchestra.run.smithi050.stderr:bash: line 1: ntpq: command not found 2024-09-16T10:46:32.463 INFO:teuthology.orchestra.run.smithi050.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T10:46:32.464 INFO:teuthology.orchestra.run.smithi050.stdout:=============================================================================== 2024-09-16T10:46:32.464 INFO:teuthology.orchestra.run.smithi050.stdout:^* hv01.front.sepia.ceph.com 3 6 377 9 -603us[ -603us] +/- 63ms 2024-09-16T10:46:32.464 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 +1284us[+1284us] +/- 68ms 2024-09-16T10:46:32.464 INFO:teuthology.orchestra.run.smithi050.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 -749us[ -749us] +/- 88ms 2024-09-16T10:46:32.464 INFO:teuthology.orchestra.run.smithi050.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T10:46:32.465 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T10:46:32.479 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T10:46:32.480 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T10:46:32.490 DEBUG:teuthology.orchestra.run.smithi050:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T10:46:32.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-16T10:46:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:46:32.660 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T10:46:32.670 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T10:46:32.685 INFO:teuthology.task.internal:Duration was 1865.092475 seconds 2024-09-16T10:46:32.686 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T10:46:32.695 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T10:46:32.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T10:46:32.772 INFO:teuthology.orchestra.run.smithi050.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T10:46:32.857 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T10:46:32.857 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-09-16T10:46:32.858 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T10:46:32.879 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T10:46:32.879 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T10:46:32.947 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T10:46:32.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T10:46:33.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T10:46:33.135 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T10:46:33.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T10:46:33.174 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T10:46:33.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:46:33.255 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-09-16T10:46:33.303 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:46:33.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:46:33.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T10:46:33.370 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T10:46:33.371 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906637/remote/smithi050 2024-09-16T10:46:33.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T10:46:33.434 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T10:46:33.434 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T10:46:33.484 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T10:46:33.493 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T10:46:33.494 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T10:46:33.507 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T10:46:33.507 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T10:46:33.539 INFO:teuthology.orchestra.run.smithi050.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 10:46 /home/ubuntu/cephtest 2024-09-16T10:46:33.541 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T10:46:33.554 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T10:46:33.583 INFO:teuthology.run:Summary data: description: rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}} duration: 1865.0924746990204 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T10:46:33.583 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T10:46:33.681 INFO:teuthology.run:pass